I believe it will effect your rep if you try to move up in schools. However, if you plan on staying a while, I wouldn't worry too much if it's Ayers you're speaking of, his Ath/Spd isn't all that great for D-1.
Even if he leaves, you still get an open schollie for next season, but as mentioned it will hurt you a bit if you try to move up to a high prestige school. Plus you have 5 other freshman.