I've been in more rural areas and the location indicator has drifted off the road. BUT it sounds like yours was not detecting turns etc.
The Camry's GPS has a "dead reckoning" ability. A pizeo-electric (tuning fork like) directional sensor, and it also is aware of distance traveled (it counts wheel revoltions). Using these two inputs it can compute a darn good ground track even without a GPS signal. It's most used in indoor garages or forested areas. It also works when you first turn on the car until it aquires the necessary satellites.
If your's is not doing a reasonable job of "dead reckoning" combined with an area of poor map coverage, that might be your issue.
J