I've read a lot of material about A/C in cars, and talked to a number of different dealership service managers. The general consensus is that A/C, in a traditional engine belt driven configuration, hybrid or not, does have an impact on fuel economy. But unlike 25 or more years ago, that impact is less due to more efficient compressor design, lower volume of refrigerant charges, less friction in the A/C compressors and other systematic design improvements. This is just good engineering design and progress. So, if you never turn on your A/C, then the compressor has no opportunity to engage and load down your engine and as such you will ideally run with optimum efficiency.

But the other comment made in print and in person is that if you choose to not use your A/C you will loose it eventually. In an A/C system, refrigerant and oil are circulated throughout the entire system at the same time. The oil lubricates the compressor, but it also keeps rubber seals fresh and prevents them from cracking. If you don't use your A/C the oil doesn't get circulated through the system and as such, over time, seals will tend to shrink, crack and leak. The refrigerant then leaves the system, and your A/C will not work at all. This is a bad situation even though the current refrigerants used do not contain CFCs, it still isn't all that advisable to be letting it out into the atmosphere.

To keep an A/C system running properly and lubricated. The system should be run for at least 10 minutes straight monthly, including winter. Since most cars use the A/C compressor as a dehumidifier in the winter when in defrost mode, that usually provides plenty of compressor use without having to actually turn on the A/C system. The same can be said of most auto climate systems. If they are left in full auto to their own desires at a reasonable cabin temp setting, the a/c compressor will run more than enough to keep the system lubricated.

It isn't worth a couple MPG to let my A/C go down the tubes.

It has been said:
Hybrid drivers come in 3 flavors, greenie, techie and cheapie. Pick any 2.
2005 Prius, Melinium Silver over gray, package 5 (AI)