Is there a way to get behind there without removing it? I don't have a torque wrench.
There is not usually a real good way to get behind there without pulling the flywheel. It depends on the type of flywheel. It's in your best interest to pull it, and get the e-clip off of it. I can almost guarantee you it's stuck on the magnet inside. If you don't remove it, you could cause damage to your magneto.
Depending on what engine you have (you haven't said) you can loosen the flywheel nut to the end of the threads. Then, place a piece of wood on the nut, and whack it hard with a hammer. Just hard enough to pop the flywheel, but not too hard. That's the old-fashioned way of popping the flywheel off the crank, and usually works pretty well. When I was a kid, that's the ONLY way we ever removed a flywheel, and we never damaged a single one.
MAKE SURE YOU REMOVE THE SPARKPLUG WIRE FIRST!