Shorten the pitching staff. You've probably got at least two pitchers you don't want pitching critical innings. Use their roster spots for, say, a pinch hitter and a baserunning/defense extra.
If you've got enough innings in them, you really only need a three man rotation in the playoffs. And schedule your rotation around the off days. When I won my WS, I used a 2 1/2 man rotation. The 1/2 man was a rookie with a nice home split, I started him only in home games. I only needed two spot starts from other pitchers, they were both losses.