I did not play college basketball, but I've played my entire life including organized high school ball. I'm pretty sure that I could spent 10 hours a day practicing my ballhandling and I'll never be as good as Chris Paul. There are physical limitations. For ball handling, hand/eye coordination, size and strength of hands, body balance, etc. will determine your ultimate potential.
I would also argue that any improvement is still part of your potential, so there's no reason not to have a hard cap.