Property dispute (taking the fence down)

To make a long story short: This property has been in my family for 40 years. We aquired a new "neighbor" to the south last year. We greeted the "neighbors" at first and then after a few months the troubles started. Basically their is a dispute over where the property lines are or rather who owns what. A little explanation. Neither property had been surveyed as to our knowledge. The property to the south supposedly had been done at one time, but the survery was never registered. This occured in the late 80's with a different owner. At that time we were told that the survery showed that their garage was a foot onto our property. My parents signed a quick deed so that the new owner's northern property line would abut our southern property line, which would be the center point between the two garages. Now, after a dispute with the new neighbor we have our property surveyed and find that we never owned the 3 feet that was quick deeded to the former owner. The survey shows the boundary of our SE point which matches the plat. But our deed shows that we own another 4 feet to the south of this point. His deed shows his southern most line is 6 feet from what our deed shows. The surveyor placed a temporary stake on the SE point from the plat until he could find the discrepancy. Since that time the new neighbor has placed a fence in the area that is under question as to ownership. I have just contacted a title company to find the answers to this. I have also had a consultation with two different attorneys and been told different things. Regarding the neighbors fence: If indeed we own the property on which it is constructed one attorney told me that I could indeed remove the fence as long as I didn't damage the fence. The other told me that the only way I could remove the fence is to go to court, which would take many thousands of dollars and maybe a year. Who is correct? Also, the title company informed me that the plat is the master and not my deed. Even though my property tax statement says 'Compare this statement with your deed...'. So which is correct? The deed upon which my property taxes are based or the plat?

If the land is yours and there is no question you can take the fence down. try to be reasonable and give notice in writing.