ike - rate of improvement and rate of decline are strongly tied to the amount of room left for growth. so your second paragraph there is correct. the further you are from the cap, the faster the decline at 0.
now, to answer your original question - its impossible to say, because "low" is a range, from 0-5. in the 5 case, you would probably lose maybe 2 points over a season. so if you have a better use for those minutes - like being able to gain 3 in another, more important category, its a no-brainer. now, most likely, in that low range, you are in the 0-3 part. why? because you improve much more quickly at 5 and 4 than 2 and 1, so maybe 80% or so of the time spent in "low" potential is in the 0-3 range where you are pretty safe at 0 minutes (may still see a 1 point drop).
so what i typically do, assuming i have decent uses for the other minutes, assuming this is a season i am doing anything in the first place, then i would set the low potentials at 0, especially if they aren't extremely important. then, after a handful of games, you can re-evaluate. but you are pretty safe to set and forget in the normal case, which is, even if you expected to lose say an entire 1.5 points (more than you should expect, on average), you had a better use for the minutes anyway.