If you have already removed some of that material has it not increased the end-shake on the barrel arbor? Did you check the barrel to make sure the arbor holes were not worn? If you do have a worn arbor hole on the barrel it will allow the barrel to lean over slightly which could have caused that wear in the first place.
If I am teaching you to suck eggs then please tell me to keep quiet 
Paul.
