For premium position defenders, I would still generally require above-average offensive production for the position even from a superstar defender. I also think there's a huge gap between SS and C/3B/2B/CF. I don't care how good a 3B you are, you're not a HOFer with a #6-7 hole bat at that position. If you're a SS, then possibly.
I also think catchers should get a little extra credit for having lower total numbers (HR, RBI, etc) because they should only be expected to make 120ish starts a year.
For non-premium defenders, I don't care how many GGs you win, it still doesn't count as anything more than a little extra credit if it's a close case, in my opinion.