I’m not a medical professional, but as a mom of a son with ADHD plus taking ADHD meds myself, here are my thoughts:
1) This might be the wrong med for you. How many have you tried and how did those make you feel, especially compared to this one? My son’s psychiatrist talks a lot about how slightly different the different meds are, but the right one interacts really differently with your personal chemistry and feels “right” to you where the other ones wouldn’t. He would probably ask questions to determine whether you should try a different med.
2) If it’s the right med, it might be too high a dose. The right dose is supposed to help without having intolerable side effects. It sounds like on your empty days you are having too many side effects. I work full time and find that my meds help me immensely on busy days at work (help me actually have productive days), but they also help me on my days off. While on my meds, I can even take a nap on a day off if I am tired (sick or was up too late). I have heard other people describe that as well. People without ADHD can’t sleep on a stimulant, but ADHD people can!
My meds don’t feel like “speed”, they make me feel normal instead of unfocused / going in 10 different directions. Even if I do have a few different things happening at once, I feel that and have to rein it in, like a normal person. I am not manic or thriving off having that many things happening.
Assuming you have tried other meds, this would be my first guess on what is happening – too high a dose.
3) Would it be possible / reasonable to have different doses on different days? I truly don’t know if that is recommended, although kids sometimes don’t take meds on weekends and only take them on school days, so I would guess it might be a possibility. IF you know ahead of time which days are busy vs. empty, maybe this is something you could discuss with your doctor?
4) Another possibility – ADHD is often comorbid with other diagnoses, and living with undiagnosed ADHD as a child may have caused you to have a problem with anxiety. Could you perhaps need anxiety medication in addition to the ADHD? The irritability and panic are what made me think of that. But those could also be side-effects of the wrong / too much medication. Just something to think about. My son and I both have anxiety in addition to our ADHD. Rather than increasing his ADHD med, his doctor added on the anxiety med and that was the right choice for him.