Did mine yesterday, pretty much a breeze thanks to Bitter's tips above regarding the Acc outlet. I also did the sliding armrest from the XLE, which was a little tougher due to the baby seat in the center back which I didn't want to uninstall, so it made for a tighter working space.

I was thinking if you found a totaled XLE in a junkyard, you could probably take the entire center console and put it in the hybrid, getting the wood grain paneling in the process. MIght be a big expensive if you do it with new OEM parts, though.