We might be able to help out but will need a lot more information since there are many things that can cause this problem.
The first thing to check is if the balance can rotate freely without ANY resistance.
After replacing the upper and lower hole jewels and balance staff does the balance spin freely with the hairspring and pallet lever removed?
With the hairspring in place on the balance does the balance vibrate freely with the lever removed or does it come to an abrupt stop? Does changing positions affect the motion when doing these tests?
I’m not sure what you mean when you say the roller jewel keeps coming out of the fork. Can you explain that a bit more?
Answers to these questions will help narrow down the options and pinpoint the problem.