Problem with voltage dropping from battery to charger plug

[GT350Mike]

I'm going into my second year of owning a 2005 Club Car and run into my first problem that I can't seem to figure out. I bought and installed new batteries last week and it was reading 54v (so far, so good) but when I check the voltage at the charger plug/receptacle, it only reads 41v. The wire running from the positive side to the plug looks normal and so does the connector. This is where it doesn't make sense to me.....I removed the receptacle from the cart and cleaned all three connectors and after re-installing the receptacle, it read 54v. I knew a dirty connector could not make full contact with the pin and would cause it to heat up because of the resistance, but didn't know I could loose voltage.

Anyway, after getting the 54v I put the charger on it overnight and after a full day of use the following day, I plugged in the charger and the charger never kicked in. I checked the voltage and got 41v again. With all that being said, does it sound like I just need to replace the connectors or could I have other problems?

BTW, to rule out any potential problems with my Power Driver II 22110 charger, I went ahead and replaced the relay, rectifier, and circuit breaker.