What is the process when a team claims a player on non-revokable waivers, but doesn't have enough in player payroll? Does the claim fail and the player goes to the next team that claimed him? Or is it a broken transaction, where no other team can have the player?

From the Help Manual:

In order to be awarded the claim for a player prior to the transaction deadline or after the deadline when a player is not set to be automatically pulled back, the franchise must be able to take on his remaining salary for the season as well as the remainder of the contract. Otherwise, the next franchise in the waiver rank will be eligible for the claim.

Must have missed it, thanks.

Note that the claim itself is blocked after the deadline, so there shouldn't be a problem there.