I would check with my insurace company before pulling off any side mirrors as well. They may have their own opinion, law or not. Personally if you did manage to get into an accident and there was an injury lawsuit, I could see the insurance company lawyers saying... "its not our problem, the insured party removed a rearview mirror and therefore is at fault, we won't protect him from liability."

Is it likely... probably not, but lawyers will be lawyers and insurance companies will do the same. I can say that if someone clipped my front quarter panel during a lane change after removing a mirror I'd certainly sue the hell out of them--and I doubt I'd be the only one of that opinion.