Toyota says that EV mode (with the switch) won't generally improve mileage. Don't forget that all power must eventually come from the ICE and that since the ICE must be warmed up anyway that it might as well be driving the wheels. In other words, don't sweat this and enjoy the drive!