If your son is only there on weekends I would let it be. But if it is also affecting school, which may become more problematic as he gets older, then I would start to take action.
If you can get an unbiased third party like family doctor or social worker to speak to him about his concerns and try to see if there is some common ground I would start there. If your ex can hear someone other than you about this it may help you get to a place where you can agree on what to do for your son.
However, my personal experience was that in my case this was not effective. If that happens I would weigh the distress your son may experience being unmedicated at school versus the distress you will experience if you end up trying to gain control through court. FYI going through family court is time-consuming and exhausting, and does create conflict. However in my case gaining control was 100% necessary and has changed my son’s life. He went from being suspended multiple times to winning the principal’s award in grade 6. If your child can manage ok without meds I would not advise going through this. But if his school experience is seriously effected and your ex will not offer alternative solutions such as being willing to homeschool or provide therapy, you may eventually have to consider this. If he doesn’t believe your son even has ADHD you may end up having to get a court order than he give your son the medication or grant you the ability to give it to him regardless of his wishes. The reality is that many doctors will NOT PRESCRIBE OR TREAT if both parents are not on board and he could potentially block you from getting the medication. I would start documenting communications to have something to prove he is resisting should it come down to court. If you can avoid it, much better for everyone. But I do not agree with the notion it should be avoided forever. I wish I had taken action sooner to be honest.