The pot needs to rotate 160 degrees each direction so I put a 3.5 to 1 gear ratio on it.

I needed a 4 to 1 ratio but the pinion would have been to small to fit the pot so I just have to move the joystick a bit further which is not a problem.
I made some changes on the back side of the controller and was able to eliminate these parts-
This change also gave my better angles on the return springs and now they go back to neutral consistently-