Obviously kicking him off would have made a psychological difference, but how many people would dump everything they had into that one player at that moment? Probably not many.
Even then, who knows if it would have stopped him from coming after the guy?
Throwing everything at the guy was your only shot in retrospect, but it seems to me like you were just unlucky. With those other battles, there's no way to know how much he had left. I think a lot of people would have done the exact same thing as you, regardless what they say here.