Family violating rights

I have been dating a man with mild dementia and severe language aphasia for over a year. He currently lives in an assisted living facility. His ex-wife, who is his poa, had me investigated by a state agency protecting the rights of residents of long term care, and they found that I was not in any way financially exploiting or abusing him.

He has not been declared incompetent. Jim (name has been changed) asked me to marry him and I advised him that we would have to see an attorney to find out if he could legally be married. After months of consulting attorneys on prenups, changing poas, etc., we were certain we had everything in order and had planned a small wedding. Our marriage license appeared in the local newspaper and his ex-wife apparently contacted his mother and brother who live in another state. They called Jim to ask if we were getting married and he said no because he did not want them to know because of their initial reaction to our relationship.

The day before our wedding, which was to be attended by two health care workers who have worked with Jim and were ready to witness that he is competent, a member of Jim's family called the Justice of the Peace claiming that they had a medical affidavit that Jim is incompetent and the Justice of the Peace cancelled our wedding without seeing any affidavit. In fact, the Justice of the Peace could not even tell me the name of the family member who called or what their relationship is to Jim. I also received a call from Jim's brother threatening me that if I married Jim that he would find some way to have it annulled. He stated that Jim had told them that we were not getting married so he concluded that I was planning everything on my own and he didn't even know about it. (He applied for a marriage license, signed a prenup, shopped for wedding rings, etc, but didn't know he was getting married?). I have been told that if Jim had been declared incompetent, the 2 affidavits required would have to be on file with the facility where he lives and that he would not be allowed to take his own medication. The fact that he is taking his own meds and is allowed to come and go from the facility without supervision leads me to believe that there is no such affidavit.

What legal recourse do I now have? I feel that the family's interference amounts to emotional abuse. They rarely contact Jim or visit him unless there is some crisis. They tried to stop me from visiting him at the assisted living facility, but someone, either another resident or a staff member reported the executive director to the state for violating the state's residents' rights law which allows residents to associate with whomever they want without restriction.

You can still get married; they would have to get a court to overturn it. And that is not so easy to do.