Well, he moved out. I would change the locks now.

I would also pack up ALL of his clothes (in garbage bags), call him and tell him that you are putting the rest of his clothes out on the street on garbage night. Keep them in a pile separate from the garbage cans and let him know that either he is picking up his things or the garbage collector is.

Is your address on his license? Is there anything that would prove that he has ever lived there? If not, tell him that if he breaks the door down, you will call the police and have him arrested. Don't let his threats intimidate you into doing what he wants to do, letting him walk all over you.