Can we take back what we paid for?

I live in the state of Tennessee.
Early this year, we began boarding our horses at a facility represented by the operator to be wholly owned by him, and he just starting over after losing a leg in a horse training accident. After some consideration and inquiries as to character, we decided to trade this operator some physical improvements in exchange for free board of our horses and a percentage of his overall boarding income. We began with handshake agreement (we know... how clever!) and began construction of a barn, purchased a round pen and other necessities...and then were informed that this operator did NOT in fact own the land but was instead leasing, and had amassed a serious amount of past-due obligations on basics such as utilities, transportation, and related expenses. We again were very clever, and bailed him out so that he could continue operating and we would have some hope of recovering our initial investment. All was going along nicely, until the man's wife decided she needed to be pampered 24/7, and he quit investing even his sweat in this project. Then we learned that these people were continuing to represent to customers that they owned the land, and that we were simply hired hands of a sort (which is of no import to us at all if it helps them sleep). But, needless to say, we got a bit more clever and stopped everything after meeting our goal of the barn completion, advertising and the like, and drew up a very informal document simply outlining what we paid on their behalf and what the expenses for the physical improvements were, repayment expectations and investment return, and asked them to sign. They did, we did, as the document was basically just an outline of what we had already verbally agreed to. However, this infuriated the princess bride, who has gone to great lengths to make the whole situation unbearable. We have decided to move our horses, and have secured alternate accomodations.
With the background out of the way, here is the question: Can we take the physical property we put in place with us when we leave, or are we required to leave it? We have received absolutely no compensation, with the exception of 1 month "free" board for each of 3 horses (we have 8), so feel that we need not leave anything behind. There are no "permanent" structures in that no foundations were poured, nothing was plumbed or wired, etc. We simply do not want a legal issue with the actual owner of this land, nor the tenant.
If I tear down the new barn, gaher up my round pen, buckets, truck tires, etc. etc. > am I in any trouble?
Thank you in advance for any assistance you can offer.

[judg4]

You can take items that are not fixtures yes. Involve the police if you feel any heated arguments may break out etc. You may have to go to court to get an order if he puts up some kind of argument or says it is a business dispute.

I thank you for your help!! This is a nightmare, and all I want is my stuff and to leave. You've been quite helpful.

let us all here know what happens if you can...