Until you get up to D1 the effect is really negligible as far as its play on head-to-head recruiting battles. I don't look at anything in D2 or D3 other than likely bank accounts of the people I'd be battling against and distances involved. (obviously there's a bit of an issue if you're D3 and going up against D2, in that case the difference in divisions plays a small role, but a D3 can compete and win against a D2 under the correct set of circumstances)
Where prestige WILL help you at D2 and D3 is that you will either see kids available to sign immediately that someone with a lower prestige might have to either pull down or wait to drop down in order to get. Players will also fall to you quicker with a higher prestige than if you have a lower one.