Only the two of you can find the answer that is right for you. However, I can say that people with ADHD can be easily mentally/emotionally exhausted. Every time he wants to check out, it seems it is prefaced by a somewhat stressful event (yes, vacations can be stressful and/or overwhelming for some people). Often, instead of taking the space and time they need to refresh and move past the mental/emotional exhaustion, they lash out at those around them. He may have learned that this is a possibility and retreats within himself and from others in order to prevent that.
Maybe you can discuss these situations with him and come to an agreement where you stay together but you give each other lots of space. Almost as if he’s getting a meditative retreat but without pulling away completely. Couples counseling could help as well (as long as the clinician is well-versed in adult ADHD).
ADDitude Community Moderator, Author & Trainer on Parenting ADHD, Mom to teen w/ ADHD, LDs, and autism