But we can. If we accept that the ability to locate and change speeds are probably the most important skills a pitcher can possess, we can actually WATCH an occassional game and evaluate a pitcher's ability to get batters out without throwing it past them.
We can distinguish a pitcher's role in BABIP? Without watching every pitch of every game, how could we do that? We know that even hall of fame pitchers with amazing control aren't able to control their BABIP. So how do you distinguish what the pitcher did and what the defense did?