1) You can't limit access to just the passenger.
2) I am sure you could use all the functions and still maintain complete control of the car but the vast majority of drivers can't. Heck, most can talk on a cell phone and drive at the same time.

Toyota did the right thing.

Suggestion: Plan your trip ahead, set up some quick destination buttons, speeddial phone entries and your favorite radio stations ahead of time. Those work while in motion.

J