Can my ex use my good friend against me?

I work at a Salvation Army in CA and one of the guys there and i are into eachother but have not taken it any further then going to the movies twice and texting and talking at work. hes only seen my two kids once while i was at stater brothers and we ran into eachother. but because of me "associating myself with those kind of people" as he puts it. hes threatening to try to take the kids from me. theres is nothing he can do right cause the guy is not around the kids right? and being that the guy is in the rehab and is clean and is doing better than my ex who him and his wife still get high and do drugs when they dont have my two children i dont know where he sees any room to talk.

[AFFA]

I wish to inform you that in matters relating to child custody it is the best interest of the child which is taken into consideration. In this you are having the custody of the children. Now, if your ex tries to take the custody of the kids from you then he will have to file a case for modification of custody in court. Further, in this the person has to show that why you should not have the custody and why he should have the custody from the point of view of child's best interest. As regards you meeting someone that will affect the child custody only when it can be shown to court that the meeting of you with that person has an adverse effect on kids.

AFF