Yes. If he was declining his part of the mutual option he'd already be listed as a free agent.

Not applicable for him probably, but when exercisiing options, always look to see if you can declne it and then sign the guy as a free agent for enough of a discount that with his buyout and new salary, you actually save money. This is normally applicable with an aging veteran who was signed to a big contract and is now losing skills but may still have use as a bench player or bullpen pitcher.

I tried to decline a $10M option on a good, older player one time-  he would not negotiate so he could go to free agency.
You don't get comp picks that way so it's a bit of a risky strategy.  If they guy is good, accept the option then try to negotiate an extension at the end of the season.

If you decline a mutual option and then resign the player, does the buyout money get used towards the new salary, or is his compensation buyout+new salary?

Buyout plus new salary.