I think Angelacawein’s suggestion is good. Invite one or two of his family members for a quiet get-together. It doesn’t have to be a meal, it could be a hike, a drive, a board game, or anything you know they would enjoy. Maybe you don’t need to tell them about your conditions, you could just say you don’t feel good in large groups.
If they try to reassure you that you needn’t worry and everything will be fine, just keep repeating, “no really, I can’t.” Practice saying this many times before you talk to them.
If they live nearby, you could show up at their event for a short period, maybe half an hour. (This is assuming that you can handle being there for that long.) Or make a nice food contribution and drop it off before the party.