Please help!!!!!

My childs father who is a good father at best of times, has been leaving my girls at the drop of a hat and leaving them with his mother anytime he HAS to do something which is effectin them and im left to pick up the pieces so therefore said he was only seeing them at weekends rather than as well as during the week as this is upsetting for my girls and for me. But last week his mum told me my kids would be staying with them if i stoped him seeing them as much as he does. Then basically told me the same . Which is making me feel like they are going to try and take my kids off me. Im a good mother to my kids and dont believe any one would take my kids off me but if they are trying this what am i to expect? What would be their first move? Can they do anything without me knowing?

They will not be able to do that, dont worry. That is an idle threat.