Relocation Nightmare

I will apologize in advance if I am being redundant...I did search the forums but was unable to locate a similar case. It's a very long story but I'll try to keep it brief...
My ex was very abusive and has a history of psychiatric issues. When we divorced he was incarcerated on a felony conviction and I was granted sole legal custody as well as primary physical placement. He was granted visitation at 'reasonable times upon reasonable notice'. With him being incarcerated though he has not seen the children.
He has written to our children one time in the past year. My address was sealed from court records due to the violent threats he issued but his mother offered to serve as proxy in communication with our children (even she refused to give him my address.)
Last month, he pled guilty to 2nd degree sexual assault of a minor and is awaiting sentencing...despite the charge the actual allegation is that he raped a 14 year old blood relative. I have sent notice to both he and the court of my intention to relocate to the UK. He is now contesting me and his mother is definitely on his side of this situation - she was never witness to the abuse, threats, violence, etc. and assumes I am an embittered ex spouse.
I'm asking to move because I'm engaged to a wonderful man (a British citizen) who lives overseas but due to his career he is precluded from relocating here. My ex's family sees our children once every 3 months for a few hours. I have told his family I would be happy to bring the children back in summer for visits and that I would make full use of email, webcams, even teleconferencing should that be an option. That does not appease my ex's mother. Financially, emotionally, psychologically, I've really struggled on my own. My parents are both deceased and I only have a sister in the area who fully supports my move. Due to my abusive marriage, I wasn't 'allowed' to have friends. I have no support system here...my ex's family I see only a few times a year.
My ex told me he never wanted the children and has threatened to 'take off' once he is freed from prison to avoid paying any child support (currently CS is only held open). But, he is attempting to use them as a pawn in order to control me and keep me here.
I really need some advice on where to turn next though. He wrote a letter to the court that should I be allowed to move he believes I would never allow his mother to see the children again - I have emails to his mother to prove just the opposite. I think I might just need a friendly voice to tell me I've got a great case to present to the courts. I live in Wisconsin, by the way.

Thanks so much for your time, consideration, and advice.

[tbyte]

I think you have a good case.
Make sure you take things one step at a time and stay within the boundaries of the law and your divorce agreement.

[PapaJohn]

I agree with Tbyte. I think the court would agree as well considering the guys history and current situation.

[Littleacres]

Do you have a child custody agreement in place at the moment. If so what does it say about where you must live, does it say that you can't move out of state or out of country. Does it say that you have have primary custody/primary residence? what is the visitation agreement in the child custody agreement? Does your state have grandparent rights. Can the grandmother of the child fight for visitation? I don't see any problem with your relocation, as long as it doesn't say that you can't move where you want. If you move to UK, from what you say,the only problem that I see is that if their is a legal visitiation in place for him to see the child, you may have to fly the child back to the staes for those visitation. As long as he pays child support, you must at least grant him supervised visitation, he may also be able to fight that the transport of your daughter back and forth from country to country be your expense, seeing as how you are moving the child out of country. If he chooses to opt out of paying his child support, then at that point he has legally given up, the right to decide or choose, when, where and how long he may have access to the child, you can never totally take all visitatin away from the father, unfortunatley even if he is abusive, at most he muast at least have supervised visitation. The only reason the mother is now on his side is because she probably does not wish to lose acces to her grand daughter

First of all...thank you for the answers received...it was a little intimidating posting my problem in the first place and I really appreciate the kind response. To Littleacres..my divorce papers are very specific - if I move more than 149 miles away I have to notify their father. I did notify him and he objected so I must now file a motion with family court requesting the judge allow my relocation.
My ex doesn't pay child support and is incarcerated. He will be going to prison in July for raping his niece - he's already pled guilty. I have sole legal custody and primary physical placement but he does have visitation at 'reasonable times upon reasonable notice'. Obviously with him in jail, he hasn't been visiting...he didn't even send a card on their birthdays. He has no relationship with them at all anymore.
He requested an order in January to force me to take the kids to visit him in prison but the judge denied him.
There are grandparent rights in Wisconsin...I've never not allowed her to see them in spite of not having an order. I have no issue with her seeing my children - but I know she feels the kids should see their dad, despite the circumstances, and she would prefer anything to me moving....even if staying here leaves them in poverty with an overworked and overstressed Mom.