I don't really see the point of the button- it's only good in a few situations. It won't work in any of the following situations:

-It won't work if your speed is 25mph or greater.
-It won't work if the traction battery is low.
-It won't work if the engine is warming up.
-It won't work if the defogger is on.
-It won't work if the gas pedal is depressed firmly.
-It won't work if the hybrid system temp. is too high / too low.

(Source: Toyota Iguide http://www.toyotaiguide.com/iGuide_HighHy.htm)

So the only real circumstances that the EV button would actually work would be in parking lots or if you are driving very slowly- and the Camry Hybrid will most likely be cruising in electric-only mode anyway.

So in the original poster's scenario where they would use it while driving down the half-mile street near your house in the morning- the EV button wouldn't work because the gas engine would still be warming up.