If you get polyurethane caulk on your hands, you won't be able to remove it with simple soap and water. Instead, your hands will be brown and you will be forced to use slightly more abrasive cleaning methods to get the job done. The best idea is to wear gloves during a project so you prevent this from happening ahead of time, but assuming you can't turn back the clock and do that now, here are a few ways to remove the poly caulk.
Step 1 - Soak Your Hand
You want to soak the caulk on your hands so that it comes off, similar to the way you would get cooked food off a pot or pan. Use your cotton swab and removal solvent of choice to put a light but efficient layer of remover on top of your hand. The less you can use, the better off you will be because those chemicals aren't ideal to keep on your skin long. Leave your hands to set for five minutes.
Step 2 - Scrub and Rinse
Use the rough washcloth you have to scrub off the caulk you just loosened up. Then rinse off your hands along the way. Repeat this until all the polyurethane caulk has been removed.