Your problem is that you've not set yourself clear business boundaries. The fault doesn't lay with the customer - it's firmly in your corner. You need to get clear with yourself about when you will answer the phone or your emails, what you will do about scope creep (remember the person who pushes the boundaries of the contract?). Make sure you are clear about what you will and won't do for the money that is being paid by the client.
Having said that, the next thing is to work out your levels of flexibility and how you will handle being flexible. "Hey, good to hear from you. I don't normally answer the phone at this time of night. Is it something urgent or can I call you back first thing in the morning?" Sometimes your clients DO need you urgently, but if they're simply in the habit of pushing boundaries, you want to nip that in bud with them as early as you can in your relationship.