Every school has a prestige grade that they started with (Duke is an A+, crappy DI schools are D's, etc.) A team's current prestige grade is determined not only by what they've done the last handful of seasons, but also takes into account the baseline prestige (as well as conference strength being factored in). So if Duke and Prairie View had identical results over the last four seasons, you'd expect Duke to still have a higher prestige.
Once recruiting starts, baseline prestige in and of itself doesn't matter. But it does impact recruiting indirectly in that overall prestige obviously effects recruiting, and baseline prestige effects your overall prestige grade.
Sort of a similar concept for jobs ... baseline doesn't really impact the jobs process once it starts. But in helping determine a team's prestige, that obviously impacts what teams will qualify for that opening. (There may also be a slight baseline effect during the process, in that you may be qualifed for -- for example -- a B-level mid-major team, but not qualified for a B-level team with a high baseline prestige. That doesn't always happen, but it's possible.)
(Note: Baseline prestige only exists at DI.)