I'm no expert and I'm just going to give my 2 cents. He's got a problem and knows it. He knows that in order to see his kids (yours) he would have to face his problem. Avoiding you avoids his problem and keeps the statis quoe. I would ask for supervised visitation. Maybe some how you should let him know that if he resolves his drinking and drugs he will get more visitation.
Your x has a disease and like any disease you wouldn't let your kids around someone that was to ill do cope with the kids or put them at risk. You do not have to tell them what he has but that he is sick and they will have to wait until he gets better.
You need to get court oders in place to protect the kids as soon as possible or you could yourself get in trouble for witholding visits.