Chief George Turner announced the plan on Monday. Police officials say the new unit will be operating within 60 days.
Turner says the new unit will consist of 50 officers who will rank just below the department's SWAT team.
The Red Dog unit has been involved in several controversies, including a raid on a bar that led patrons to sue the department. The Atlanta City Council agreed last month to pay $1.2 million to 19 patrons to settle the case.
Turner denies that recent issues with the Red Dog unit led to the changes, saying he started creating the new agency some time ago.