At least 25 other residents were wounded in the attack at the Tsukui Yamayuri-En facility for mentally and physically disabled in Sagamihara town, about 25 miles (40 km) southwest of Tokyo.
"This is a very heart-wrenching and shocking incident in which many innocent people became victims," Chief Cabinet Secretary Yoshihide Suga told a regular news conference in Tokyo.
Prime Minister Shinzo Abe later told a gathering in Tokyo: "The lives of many innocent people were taken away and I am greatly shocked. We will make every effort to discover the facts and prevent a reoccurance."
The suspect was a 26-year-old former employee of the facility who gave himself up to police. The man, Satoshi Uematsu, said in letters he wrote in February that he could "obliterate 470 disabled people", Kyodo news agency reported.
He said he would kill 260 severely disabled people at two areas in the facility during a night shift, and would not hurt employees.
"My goal is a world in which the severely disabled can be euthanized, with their guardians’ consent, if they are unable to live at home and be active in society," Uematsu wrote in the two letters given to the speaker of the lower house of parliament, Kyodo reported.
Uematsu was committed to hospital after he expressed a "willingness to kill severely disabled people", an official in Sagamihara told Reuters. He was freed on March 2 after a doctor deemed he had improved, the official said.
Uematsu lived near the facility, and a neighbor described him as a polite, young man who always greeted him with a smile.
"It would be easier to understand if there had been a warning but there were no signs," said Akihiro Hasegawa, 73. "We didn't know the darkness of his heart."
The suspect apparently began changing about five months ago, said Yuji Kuroiwa, the governor of Kanagawa prefecture, where the facility is located.
"You could say there were warning signs, but it's difficult to say if this could have been prevented," he told reporters.
"This was not an impulsive crime ... He went in the dark of the night, opened one door at a time, and stabbed sleeping people one by one," Kuroiwa said. "I just can't believe the cruelty of this crime. We need to prevent this from ever happening again."
Staff at the facility called police at 2.30 a.m. local time (1730 GMT Monday) with reports of a man armed with a knife on the grounds, media reports said. The man wore a black T-shirt and trousers, the reports said.
The 3-hectare (7.6 acre) facility was established by the local government. Surrounded by tree-covered mountains and on the banks of the Sagami River, it cares for people with a wide range of disabilities.
The facility's website said the center had a maximum capacity of 160 people, including staff.
Japan has one of the lowest crime rates in the world and residents of Sagamihara said they were in shock. The last murder in the area was 10 years ago.
"This is a peaceful, quiet town so I never thought such an incident would happen here," said Oshikazu Shimo, one of many residents of the town who gathered near the facility.
Taxi driver Susumu Fujimura said of the attacker: "He said 'we should get rid of disabled people' but he's the worthless one."
"That kind of person can't defend themselves," Fujimura said, referring to the victims. "That's why so many died. It makes you weep to think of somebody just murdering them."
The dead ranged in age from 19 to 70 and included nine males and 10 females, Kyodo said.
Police had recovered a bag with several knives, at least one stained with blood, a Kanagawa prefecture official said.
At least 29 emergency squads responded to the attack, Kyodo reported, with those wounded taken to at least six hospitals in the western Tokyo area.
Such mass killings are extremely rare in Japan and typically involve stabbings. Japan has strict gun laws and possession of firearms by the public is rare.
Eight children were stabbed to death at their school in Osaka by a former janitor in 2001. Seven people died in 2008 when a man drove a truck into a crowd and began stabbing people in Tokyo's popular electronics and "anime" district of Akihabara.
A revision to Japan's Swords and Firearms Control Law was introduced in 2009 in the wake of that attack, banning the possession of double-edged knives and further tightening gun-ownership rules.
Members of a doomsday cult killed 12 people and made thousands ill in 1995 in simultaneous attacks with sarin nerve gas on five Tokyo rush-hour subway trains.