I don't think 96 out of 360ish tells the whole story.
While all these schools are in the same division organizationally, the FBS/FCS divide absolutely exists in basketball, it's just not legislated.
Let's be realistic - the teams that legitimately start the year competing at the top plane of D-I hoops are as follows: The BCS conference teams, the top half of a handful of mid-major conferences, and a handful of teams at the top of other mid-major conferences. That's maybe 100-120 teams that are actually relevant to the championship picture in some way. If you remove from 96 the 8 teams that aren't really relevant but get in with auto-bids from low major conferences anyhow, you've got fully 75% of the teams that are really upper division getting in.
Also, for the pro leagues you mentioned, with the exception of the NFL which is a different beast altogether due to different practicalities, your playoffs are best of 5, best of 7, so your lower seeds have to go out and outplay the higher seeds over the course of a longer series. In NCAA hoops, all it takes is one bad night for a Kansas or a Syracuse to flush an entire season down the drain. I think those teams have earned the gimmie first round games they get now, and wouldn't get under a 96 team tournament, because that 16-17 seed they get in their first game would legitimately be one of the top 64 or so teams in the country, as opposed to what they get now.
In summary - if your primary interest is in creating a better TV product, more upsets, etc, then more is absolutely better. If you're interested in a championship format that maintains a higher chance of your actual best teams winning a championship, you hold the line.