Outdoor plant ant infestation in my soil! :( or :)?

So i have some outdoorplants and one of them is well into flowering and i had some sort of animal or something dig up a bunch of the dirt from the pot as they are lolipoped so the pot it is in is easilly accessible so i dug a hole in the ground put the pot in so it fit nice and snug so it couldnt fall over and put some of the excess big dirt chunks i dug from the ground and put it on the top off my soil so nothing would dig in my medium and fuck the roots, but now i have a damn ant colony living in there! I have no idea what to do, whether or not theyre kind of helping i dont know because i would think that its giving extra oxygen to the roots but at the same time im thinking having an ant colony in my medium could be damaging also, if you know what i should do please let me know it would be greatly appreciated!