Defense absolutely matters. You're not going to see 83-79 games very often, however. Remember, this is a fantasy situation and most people really supe up their offenses. So you have to look at it in perspective: how is your defense doing relative to the league? You can see this by going to team rankings, select opposing stats, and then sort by fg%. Typically people who complain that defense doesn't do anything (or doesn't do much) have the idea in their heads that they won't give up a lot of points or will give up a really low fg%, but that is almost never going to happen in a normal league. When you look relative to the league, however, you usually see you're doing well. If not, really look at other teams and examine yours. Three lockdown defenders does not a great defense make in this sim. Great defensive teams (depending on league type) typically have above average (60+) defenders for most of their rotation and elite (80+) defenders for a majority of that (who can also cover for the weak defenders; for example, a 90 d and a 60 d who are both 100% at PF/C). It's easy to think you should have a great defense when you look at your SL and see 100, 80, 60, 90, 100, but if they (typically) only play 60% of the game. When the guys replacing them are like 40, 50, 50, 40, 60 ... your overall numbers will drop off significantly.
Also, when it comes to defensive scores, remember there are many other factors when it comes to points allowed: If you're getting killed on the boards or turn the ball over a ton, even if you are limiting your opponent to 40% fg (you won't over a season in a normal league), you're still going to allow a ton of extra points. Same with fouls. If you're sending your opponent to the line a ton, it doesn't matter if you have ALL 100 defenders, that defense doesn't to jack against a free throw.
So, yeah... defense matters a ton. You just have to adjust your expectations of it, compare your results to other teams, then compare your team makeup to others and figure out what you want.