I would not double team anyone who wasn't the primary scorer, ever. Agreed that it depends where their scoring comes from, but I'd expect that the kind of team you described got most of its scoring from the very good interior players, in which case I'd be playing a minus defense.
EDIT: Looks like you played a -3, which is what I would've opted for, and defended them quite well but lost the game with a rough offensive effort.