As has been said, shorten the rotation to your top 3. You may need a 4th to start 1 game in a 7 game series if you get that far. Your 4th/5th starter can be moved to the bullpen.
In addition, you can adjust your playoff roster by removing your worst bullpen guy and adding a useful position player (i.e. defensive replacement, power off the bench, pinch runner, 3rd catcher if you want to DH a C, etc). You won't need the extra bullpen guy since you are moving to a 3/4 man rotation and you never know what might help in a tight playoff series. I've won multiple playoff games before when a pinch runner that spent the season in AAA comes in off the bench and steals a base that leads to the winning run.