I would suggest the biggest single factor was the loss to Brown. That's usually way it goes for cusp teams, one "bad loss" can be a real barrier to at-large, per PR.
From what I've seen (and I don't mean to suggest that I've studied it, this is just conjecture based on my general pattern recognition skills), the system makes a determination of a "good" or "bad" loss based mostly, if not entirely, on the quality of the opponent. IOW, losing by 20 to a good team - like a half-dozen of your non-conference losses - won't hurt you near as much as *any* loss to a bad team, IE Brown.
If you had not lost to Brown, you probably made the tourney. There are lots of factors at play, involving a lot of other teams, and I don't think any of the developers/site managers have ever come out and gotten very explicit about the formula, except to imply that it was supposed to be based on the real life process (unclear if it's ever been updated, though, kind of doubtful). But I'm pretty confident in saying that loss hurt you more than anything else.