Can my ex take our kids out of the state of Georgia without telling me?

We have joint custody with me having physical custody. He suppose to get them 6 weeks every summer however this is the 1st time he has gotten them in four years the kids said they leaving but when I ask where daddy says its none of your business and he refuses to communicate with me. What rights do I have?