I spent some time searching the PriusChat forums about the security system. Rightly or wrongly I'm assuming the Prius RS3200 is going to work pretty much the same as the one put into the TCH. While there was a bit of confusion even among Prius owners how the thing works, here's what I think was being said:

The basic security system is already present in the vehicles ECU, and just needs to be activated by the dealer using their programming device (scan tool). Only if you get the glass breakage sensor does hardware actually need to be installed.

The system has two arming modes, active and passive. Switching between them requires the dealer to make a configuration change. Your system sounds like it's working in active mode and you can't get the passive mode to work because, well, it's in active mode. The dealer needs to switch the mode for you. It's either one or the other apparently.

You mention expecting to have a motion detector within the car to sense if someone is reaching thru an open window. The only mention of this I've found is if someone reached in an open window and opens the door from the inside does the alarm go off. I'm curious if you actually have any docs that say there is a motion detector with the TCH RS3200?

Anyway, hope this might help a bit if what I've found does indeed apply to the TCH. Keep us posted what you end up doing...