I took over a very poor team as a project. First year no Scholarships, I took the advice of my mentor and cut two out of the three players he said to cut. We had agreed with the small amount of residual money, it would be impossible to recruit three(in was hard to recruit two, as my pitiful Freshman atest to). (Roanoke College-Naismith)

At mid term, that third poor player is in scholastic peril. AD has warned me I need to get his grades up. Here is the question.

What happens if I let him flunk out? Will he leave the program, or will I still end up cutting him after the season, and take the prestige hit?

I don't think they ever "flunk out" and leave the team. They just become inelligible to play if their gpa goes below 2.0.

If his GPA drops below 2.0 at the end of the term, he will become ineligible plus you will take a reputation hit. He will not leave because of it.