The # stats are not used in the at-bat calculations. They are calculated in part by using historical averages from all seasons. They can be a useful reference for knowing how a player's performance will be affected on average against competition from a wide range of historical seasons, but in the individual at-bat calculations, only raw AVG and AVG+ are used on the hitter's side, and raw OAV and OAV+ on the pitcher's side. While the hitter's season (+ stats) and pitcher's season (+ stats) are calculated against each other, this excludes all the other seasons which are weighted equally by the # stats. So # can be misleading. If a pitcher has "lower" normalized stats (or higher, in the case of batters), that does not automatically mean a permanent boost in that player's performance.
Over a sample size that small, bad luck is more often the culprit, but even over longer periods of time, # stats and whether they go "up" or "down" for a player are not the standard by which performance should be judged.