This started happening to my 2007 Camry Hybrid too recently. The car has only 42K miles and I started hearing the more frequent noise coming off the ABS pump, specially when the car is idle and most recently I hear a different noise when I fully press the brake pedal. I took to my dealer last week and was told I need to replace the "Actuator Assy, BR" part "44050-30300". They quoted me $2,629 but told me the would give me a 25% discount and quoted $560 for labor. I know I can get it online for much less so I'm not even worried about their quote.
This does not affect the brakes function but can eventually fail and the ABS won't work. I do agree this seems like a premature failure not typical of a Toyota and I even read they redesigned this pump in newer models as they knew about the failure so I'm thinking it is at least worth writing Toyota a letter and see what they come back with.
I read in another place recommendations to be buy it used and even DIY install it. I'll weigh my options when it actually fails as I'm not ready to spend $2k to repair it.