The ruling, which took many by surprise, found that the killer had been subjected to strip searches, had been woken up hourly by guards for long periods and that the authorities had done little to alleviate the impact of his isolation.
A survivor of Breivik's shooting spree said the decision "feels a bit like being punched in the gut" and a newspaper editorial denounced the decision as "wrong". A law professor said it was likely to be appealed.
Breivik killed eight people in a bomb attack in Oslo in July 2011 before attacking a youth meeting of the Labour Party on an island to the northwest of the capital, killing 69 people.
He took Norwegian authorities to court in March, accusing them of exposing him to inhuman, degrading treatment or punishment in breach of the European Convention on Human Rights.
Breivik protested his isolation from other inmates and from outsiders who are not professionals.
"The prohibition of inhuman and degrading treatment represents a fundamental value in a democratic society. This applies no matter what - also in the treatment of terrorists and killers," judge Helen Andenaes Sekulic said in her ruling.
The verdict said the Norwegian state had broken Article 3 of the convention, pointing to the fact that Breivik is spending 22 to 23 hours a day alone in his cell.
"It's a completely locked world with very little human contact," it said, adding there had been no attempt to ease the security "even though Breivik has behaved in an exemplary manner during his time in prison".
His isolation is "an inhuman treatment" in the meaning of the European convention, it said, noting that all his visits, except for his mother who died in 2013, are from professionals, and only through a glass wall.
The wall must be seen as a "completely exaggerated security measure," said the verdict.
The ruling, however, said the Norwegian state had not violated Breivik's right to a private and family life. It said strict censorship of his letters was "in line with the law".
In March, the case raised dismay, and some laughter, among Norwegians taken aback by Breivik's complaints of cold coffee and microwaved meals he said were "worse than waterboarding".
Breivik's lawyer said prison authorities must ease the isolation of his client.
"He must first and foremost be allowed to be in contact with other people," Oeystein Storrvik told reporters after the verdict. He declined to say what Breivik's reaction was to the ruling.
Lawyers representing the state said they would consider whether to appeal. "We are surprised by the verdict," said Marius Emberland, one of the two lawyers representing the state.
The justice minister, Anders Anundsen, whose ministry was being sued by Breivik, did not say whether the verdict would be appealed.
A survivor of the shooting at Utoeya island said the decision was shocking. "I was surprised, and then angry and upset. To be honest, it was like being punched in the gut that the perpetrator won such a public victory," Eskil Pedersen told state broadcaster NRK.
The tabloid VG said in an editorial published online on Wednesday the decision was "wrong" as it trivialized the problem of torture in prison.
Kjetil Larsen, a professor of law at Oslo University, said the verdict was surprising and likely to be challenged.
He said the court felt the security considerations seem to have taken too much over. "I think it is very probably that they (lawyers for the state) will appeal. I think it's unlikely that they will simply accept this," he told Reuters.
"I don't think personally the prison conditions (are too strict) in relation to what he did. He killed 77 people," Lisbeth Kristine Roeyneland, head of the support group for the victims of the attacks and their relatives, told NRK.
She added she was "surprised" and "a little disappointed" the state had not won on all points. "But I am very happy that the state has won on Article 8 and that he cannot contact other far-right extremists and spread his message."
She was referring to the article in the convention about prisoners having the right to a private life, a family life and correspondence. Breivik wants to exchange letters with outsiders, including several far-right extremists.
Another Utoeya shooting survivor said the verdict was a sign that Norway has a working court system, respecting human rights even under extreme conditions.
"It also means we have to take the ruling seriously and evaluate how we treat prisoners, what abuses they may suffer, and how we avoid abuse," survivor Bjoern Ihler said on Twitter.
The state must pay Breivik's legal fees of some 331,000 Norwegian crowns ($40,732.45), the judge ruled.
Ahead of the verdict, lawyers for both parties said they would appeal if it did no go in their favor.
Breivik's lawyer said his client would not appeal the part of the verdict that ruled against him.