I hear ya Sean.
I also own a FM modulator (years old) and had a hard time plugging it in (or anything for that matter). The ground springs are set too tight. In some cases, I unscrew the top of the plug (+ tip) which should give access to the spring and compress it for less tention. In other cases, like the modulator that doesn't come apart that easily (or I'm not willing to make junk out of it

), I just apply a small dab of electrical silicone grease (faucet silicone will work also) to the ground springs and that should fix the problem. I keep the plug end of the modulator in a small sandwich bag when unplugged so I don't get silicone all over the interior of my car.
Hope this helps