Yes you can. As a site administrator, you can delete Interest Groups you do not want. Click on the Groups icon on the Settings page, and go to the Interest Groups tab. Next to the name of each Interest Group, when you hover on the section, you will find Edit and Delete actions. Click on Delete to get rid of the Interest Groups.