Another consideration is the extra money for recruiting from the postseason. That's the advantage the Big 6 C prestige school has over the mid-major C prestige school. (There may also be a conference aspect to recruiting as you suggested, I'm not sure.)
Also, something that trumps the rest is baseline prestige. The higher the baseline prestige of the school, the easier it is to improve the prestige. So you don't have to do as much to bootstrap a Big 6 school as most other schools. Most non-Big 6 schools in DI and a D baseline prestige which makes it difficult to get to the B range and difficult to stay there.
It's certainly not impossible to build up a Big 6 school in even a killer conference. Take a look at BC in Naismith. The ACC has been an absolute terror in Naismith, but Stull was able to take a bottomed out BC squad and rebuild them.