There is no downside for the player's development -- playing time positively impacts the speed of category growth. It has no effect on the maximum theoretical growth within any category.
As mrslam34 pointed out, the downside is that most freshmen suck. But if your team is going to be terrible no matter who you start, then you should always start your freshmen to max out their categories and IQ.