You are removing the other stuff to get higher ungoverned RPM. The ungoverned RPM can make the governor gear assembly come apart similar to what it can do to a flywheel and connecting rod.
Best thing to do is to remove the gear. And while the engine is open put a billet rod (and cam) in it. Then finish it off with an aftermarket flywheel.