I would take him on separate trips without your daughter the same as you are taking trips with her alone. You may find trips with just he and you equally as pleasant, you may find out things about him and why he acts the way he does when your’re all on vacay together, it will give you the chance to understand him better and I would bet money he is a different kid when not with his sister. The competition for attention and fighting between my two when we go places can be brutal. I’m not saying he won’t be defiant and there’s no way to know for sure. I could be totally wrong. But I know my two boys act very differently when I take them somewhere together than when I’m doing things with them one on one. It’s possible after a trip or two alone with him he will also see the difference in his behavior and may be able to modify enough for you all to try and do one together down the road. Now that mine are getting older (10&13) I’ve offered to take them on separate trips of their choice on their respective birthdays. Just my two cents. Hang in there!