I stopped taking adderall for that very reason! I have anxiety anyway but as the adderall left my system I would get anxious over the most insignificant things. I was put on an extended release and the only thing that got extended was the anxiety!
Talk to your provider.