I don't expect it would be that much. The weight difference is probably going to be your biggest difference, and is likely similar to arrying an additional passenger, and the actual average wattage for listening to music will be far under the peak capacity of the speakers/amps, meaning the electric load will be less significant than driving with the lights on.

So, compare mileage with a passenger, or weight in the car, and the lights on during the daytime over some known route. The effect of the speakers should be less than what you observe in that scenario.