My brother and I were in that sort of situation a few years ago and went to a funeral of an uncle, intending to scoot off straight afterwards. Two people afterwards came up to us and were genuinely warm and friendly, asking us back for the wake. One of them was our cousin, the Uncle's daughter...
Funny things, families.
If it were me, I would probably go. If your younger brother was going to be totally dismissive of your possible attendance, he may not have told you in the first place.
If you go and people blank you or are actually hostile, you can leave and know you're better off out...