The judge dismissed the jury in the involuntary manslaughter trial of Officer William Porter - the first of six officers to be tried in Gray's death - after 16 hours of deliberations during which it was unable to reach a verdict on any of the charges against the policeman.
"I do declare a mistrial," Baltimore City Circuit Court Judge Barry Williams said.
The judge said an administrative judge would schedule a new trial as early as Thursday.
Within minutes of his ruling, a phalanx of uniformed officers surrounded the courthouse. They pushed back a group of a few dozen protesters and arrested at least two people, one of whom appeared to swing at an officer. Large numbers of police have been stationed around Baltimore over the past few days.
Porter, 26, was the first of six officers to be tried in Gray's death from a broken neck suffered while he was transported in the back of a police van.
The panel of five men and seven women had said on Tuesday that it was deadlocked, but Williams had told the jurors to keep trying to reach a verdict.
Porter, who like Gray is black, was charged for having put Gray in the back of the van without seat-belting him and with being too slow to pass on his request for medical assistance.
His attorneys had argued that he may have been unaware of department policy mandating that detainees be seat-belted, which was put into place shortly before Gray's arrest.
Gray's death triggered protests, rioting and arson in the majority-black city of 620,000 people, and intensified a U.S. debate on police treatment of minorities. It followed the police killings of black men in cities including Ferguson, Missouri, and New York, which sparked the Black Lives Matter movement.
CALLS FOR CALM
Mayor Stephanie Rawlings-Blake urged people to remain calm about the ruling. "We must respect the outcome of the judicial process," she said in a statement.
Officials in Baltimore had come under heavy criticism for a restrained initial response to April's riots, which some observers contended allowed arson and looting to spiral out of control.
"I urge everyone to remember that collectively, our reaction needs to be one of respect for our neighborhoods," Rawlings-Blake said. "In the case of any disturbance in the city we are prepared to respond."
Porter had been charged with involuntary manslaughter, second-degree assault, reckless endangerment and misconduct in office. The charges against the other officers range from second-degree murder for the van's driver, to misconduct.
Warren Brown, a Baltimore defense lawyer who was in the courtroom, said of the decision, "I am not surprised at all. I think you will have the same scenario with the other trials."
He said he wanted to see if the jury broke down on racial lines. Seven of the jurors are black and five are white.
Gray was arrested after fleeing from police. He was put in a transport van, shackled and handcuffed, but was not secured by a seat belt, in violation of department policy. He died a week later.
Porter testified Gray told him he needed medical aid. Porter told the van's driver and a supervisor that Gray had asked for aid but none was summoned, according to testimony.
The defense argued that Porter did not believe Gray was seriously injured until the van's final stop. His lawyers have said that Porter acted as any reasonable officer would have.
"In some ways, a hung jury might be better than an acquittal," Deray McKesson, a prominent U.S. civil rights activist, said on Twitter.
One legal expert said he was surprised to see a mistrial declared on just the third day of deliberations.
"I thought the judge would never declare a mistrial absent a fistfight until the jury had been deliberating for six or seven days," said Jim Cohen, a professor at Fordham Law School in New York. "They chose the wrong defendant to try first."