Just have it on a seperate key ring.

I have mine on a neck lanyard, and wear it around my neck 75% of the time. The only other thing on it is my gym access card. This way when I go to the gym, I only carry the car key, the access card, and I don't need a pocket for it.