Let's start with some basics. If something electrical is getting warm it's passing a lot of current. Possibly more than it should. Your gearbox sounds pretty far from standard, personally I think it's most likely the crappy thin wiring but even if you rewire with 16awg it's still going to get warm. If anything it'll get worse because the fatter wiring will pass more current. The wires might not get as warm but the motor still will. Ultimately you're running a high torque motor with high speed gears in an enclosure with little to no ventilation. It's going to get warm it's just basic physics.