That is definitely not normal. All I can suggest is blipping the throttle a bit more on your downshift. After all, you should never downshift without using the throttle to rev-match engine speed in the first place.