We had a similar issue, although my son did not qualify for gifted. He started school very early and was never held back, even though he was young, due to his academic achievements. As he ended his elementary school career, it just became more and more obvious that he was not with his appropriate peer group, being a full year younger than many of his classmates.
We talked with him a lot, about being the smallest and the youngest, and about how he would feel repeating a grade. He made the decision that he would want to repeat the grade, but that he wanted to do it at the end of 5th grade, when everyone would be leaving the school anyway. The traditional public schools would not allow us to retain him, due to the fact that he was on grade level, so we began looking for an alternative educational setting. There are many options, including charters and homeschools, but we ended up going to a small private school that understood our concerns and agreed with our plan.
The decision we made was by far the best for our son. He knows that he was not ‘held back’ due to being ‘dumb’ or underperforming, he knows that he was involved in the decision to repeat the grade, basically to fix an error that we made when we allowed him to start school early in the first place. As a school employee myself, I have seen the detriment to retention and in general I am against retention as a academic intervention, but for my son (and maybe for yours), he will not struggle academically or feel inferior in that area; for him the issue really was social development.
I wish you good luck. This parenting thing is hard!