Borrowed money from work and want to leave

My boss (owner) gave me a loan for financial reasons. The agreements that was made was that if I left or got fired then it would be paid in full. At this time, I did not know he would be a micro-manager, severe type A personality, demeaning, belittle and so forth. Too make a long story short and to leave out other factors (to keep this short), I have been offered a position on a government job starting out $2.50 more an hour, retirement benefits, and just wondered that as long as I continue to pay this loan back, what can he legally do to me?

I know that I signed and what we agreed on but I am worried as to what he will do and I am really scared. I cannot get a loan due to I am in the middle of paying some credit debt.

He could try to sue you, but if you are making payments it is not likely he would really do that given the cost and effort.