U.S. authorities will look at the department's use of force, including deadly force, among other issues, U.S. Attorney General Loretta Lynch said at a news briefing.
"Our goal in this investigation ... is not to focus on individuals but to improve systems," the United States' top law enforcement official said.
She said federal officials would be investigating "constitutional violations" in one of the nation's largest police departments.
Lynch's announcement came after almost two weeks of protests in Chicago following the release of a 2014 police squad car dashboard video showing police officer Jason Van Dyke emptying his gun into 17-year-old Laquan McDonald, shooting him 16 times. Van Dyke, who is white, was charged late last month with first-degree murder.
High-profile killings of black men at the hands of mainly white police officers in U.S. cities have prompted a national debate and protests about the use of excessive force by police.
Two hours after Lynch's briefing, Cook County State's Attorney Anita Alvarez gave a detailed presentation to reporters explaining why she would not seek charges in another 2014 police shooting death of a black man.
Alvarez, who has been under fire for her handling of the McDonald case, said that Ronald Johnson III, 25, was holding a gun and fleeing arrest when he was shot by Officer George Hernandez on Oct. 12, 2014.
Prosecutors on Monday showed police car dashboard video to reporters and played audiotapes of police radio communications and 911 emergency calls.
Warning: Below video is graphic
An attorney for Johnson's family, Michael Oppenheimer, has said that the video showed Johnson did not have a gun. Oppenheimer was not immediately available for comment.
Prosecutors showed reporters an image with a red-colored circle around Johnson's hand, saying that forensic experts clarified images of a weapon.
The video showed Hernandez firing at Johnson as he runs into a park. Alvarez said that Johnson had been asked repeatedly by multiple officers to drop his weapon, and that a 9mm semiautomatic pistol was found with Johnson after he was shot.
Alvarez said officers were at the scene due to an earlier shooting. The car in which Johnson was a passenger had been shot at, and rather than calling police, the driver and passengers had returned to the scene.
Another officer had tried to arrest Johnson, but he had pulled away, knocking the officer over, before running off, Alvarez said.
Alvarez said the weapon recovered from Johnson was linked to a 2013 unsolved shooting.
“Based upon an objective review of the evidence and the law, we have determined that the prosecution could not establish beyond a reasonable doubt that the actions of Officer Hernandez were not reasonable and permissible," Alvarez said.
Responding to questions about why Johnson was shot in the back, Alvarez said Johnson could have turned and fired at Hernandez or other officers.
Alvarez said that in both the McDonald and Johnson shooting videos, there was no audio, which she found "frustrating."
The federal investigation of Chicago police follows other high-profile investigations of departments in Ferguson, Missouri and in Cleveland. Baltimore police are also under federal scrutiny.
In Ferguson, a St. Louis suburb which drew national attention following the fatal shooting of an unarmed black teen by a white police officer, federal authorities found racially biased abuses in both the police department and municipal court.
Last December, the Justice Department found that the Cleveland police systematically engaged in excessive use of force.
Chicago Mayor Rahm Emanuel, who had initially disagreed with calls for a federal civil rights investigation, said on Monday that Lynch would have the city's "complete cooperation."
Emanuel, who has faced heavy criticism over the McDonald shooting, has ousted his handpicked police superintendent and replaced the head of the city's Independent Police Review Authority, which reviews police misconduct allegations.