I think there are a couple issues:
1) Sometimes options are limited. Vegas was the A's at one point, I think, then it was the Jays, then Mets. It's a hitters park, so I think some organizations don't like their pitchers there. It seems to be the last resort for teams that don't have an affiliate yet.
2) Corporate relationships. I don't know the details, but the O's were with Rochester for years before a falling out. Norfolk is much closer and works out well, but from what I understand, their relationship with the Red Wings just went south for some reason.
One thing I've always wondered about is if a team could potentially start a new affiliate somewhere and have ownership of the team, or if that's forbidden by MLB.