Clean-shaven and wearing a black suit, white shirt and golden tie, Breivik raised his right arm in a flat-handed Nazi-style salute on arrival at the court, slightly different from the outstretched arm and clenched fist he used in 2012.
His lawyer said Breivik considers himself a national socialist, or Nazi, and that the gesture was "the worst thing you can do in a courtroom". Breivik later suggested it was an old Norse gesture, he said.
Judge Helen Andenaes Sekulic was not pleased either way. She told Breivik not to repeat the salute when court proceedings resume on Wednesday.
Appearing in public for the first time since he was sentenced in 2012, Breivik is claiming inhuman treatment by Norway, where he is serving 21 years for killing eight people with a bomb in Oslo and gunning down 69 others on an island nearby, many of them teenagers.
He has had just one visitor with whom he had physical contact - his mother, who was allowed into prison and gave him a hug shortly before she died of cancer in 2013.
Breivik's lawyer, Oeystein Storrvik, accused Norway of violating a ban on "inhuman and degrading treatment" under the European Convention on Human Rights by keeping the 37-year-old isolated from other inmates in a special three-room cell.
"There is no tradition in Norway for this type of isolation," he told the special court that will meet until Friday in a gymnasium at Skien jail about 100 km (60 miles) south of Oslo.
Norway rejects the charges of inhuman treatment.
"Breivik is a very dangerous man," said Marius Emberland, the lawyer representing the state, defending Breivik's conditions.
He said Breivik had been given some opportunities for interaction with others, including meeting volunteers to play chess, but that he had declined.
Another prisoner tried to attack Breivik last year, getting to within earshot. When stopped by guards, the man shouted: "You are a killer, a child killer ... And I love my country," Emberland said.
Storrvik told Reuters he had advised Breivik against making the salute. "He (Breivik) says he is a national socialist," he said.
Oeystein Soerensen, a professor of history at Oslo University, said Breivik seemed to want to signal to like-minded fanatics "that he is now a full-blooded Nazi. He wasn't that in 2011."
In 2011, for instance, a rambling manifesto written by Breivik expressed sympathy for Israel, seeing it as an ally in his hostility to Muslims. And Breivik's previous clenched fist was "a sort of home-made fascist salute," he said.
Opinions are divided among the survivors and relatives of victims who have spoken out publicly. Some have said the lawsuit is a joke and do not want to be reminded of July 22, 2011, while one survivor said Breivik's human rights should be respected.
"Breivik made us inhuman as victims of his actions and we're in danger of falling into the same trap as him if we take away his human rights," survivor Bjoern Ihler told Reuters in Oslo, at a court where the case was televised.
Breivik killed eight people with a bomb in Oslo and gunned down 69 others on an island nearby, many of them teenagers. He is serving Norway's maximum sentence of 21 years, which can be extended.
Breivik will have a chance to speak on Wednesday. The single judge - there is no jury - will issue a ruling in coming weeks. Storrvik says he may eventually appeal to the European Court of Human Rights if Breivik loses.
Norway considered it too dangerous to hear the case in Oslo. The makeshift courtroom has walls lined with timber bars and a climbing frame as well as two basketball hoops.