The outcome deprived the maverick right-wing prime minister of a clear-cut victory with which to challenge EU migrant quotas and the radical nationalist Jobbik party called on him to quit.
Jobbik said Sunday's referendum, in which the government said quotas were rejected by 98.3 percent of voters but only 40 percent of eligible Hungarians cast a valid vote, was a fiasco that offered a "trump card" to Brussels. At least 50 percent turnout in the plebiscite was required to legitimise the result.
Along with other ex-Communist countries in eastern Europe, Hungary opposes a policy that would require all EU states to take in some of the hundreds of thousands of mainly Muslim migrants seeking asylum in the bloc after arriving last year.
Orban told parliament the referendum had achieved its goal as more Hungarians, some 3.28 million, repudiated the migrant quotas than had voted for EU membership in a 2003 referendum.
He said 1 million more voters supported his government's migration policy now than voted for his Fidesz party in the 2014 parliamentary election, when he was returned to power.
"This decision, this support (from the people) obliges me to act on it," Orban told parliament as opposition lawmakers hissed and booed.
"Fidesz and the Christian Democratic Party believe the appropriate, honest and necessary step is to give legal weight to the will of the people ... Therefore, I am initiating an amendment to the constitution," he said, without elaborating.
Jobbik chairman Gabor Vona has repeatedly called for a constitutional amendment to protect Hungary from migrant quotas imposed by the European Commission.
"Since yesterday you (Orban) have become a failed politician ... You will not be taken seriously by Brussels bureaucrats," Vona said in parliament on Monday. "Brussels will ruthlessly exploit your irresponsibility and mistake."
Vona said EU authorities would zero in on the fact that the referendum failed legally, and this would weaken Hungary's bargaining position in EU deliberations on migrant policy.
The European Commission said it took note of Hungary's referendum and it was up to Budapest to deal with the outcome.
Vona also said the referendum result ended a era in which Orban's government could use the migration issue to distract attention from all of Hungary's problems under its watch, including serious corruption and emigration due to low wages.
Orban has led resistance to the stance taken by German Chancellor Angela Merkel, who has said EU states have an obligation to share the burden of taking in refugees.
He responded to the influx last year by sealing Hungary's southern borders with a razor-wire fence and deploying thousands of soldiers and police.
In power since 2010 and with his party still firmly ahead in opinion polls, Orban could still use the referendum to keep the issue of migration on the political agenda in the run-up to the 2018 election, some analysts say.
Many Hungarians were disappointed with Sunday's outcome.
"What is at stake is the future of the country. The people should have voted for the future of our children and yet many people were too lazy to go to vote. They should be ashamed of themselves," said Istvan Nemeth, a lawyer in Budapest.
Orban says deciding whether to accept migrants is a matter of national sovereignty. He says Hungary - with its Christian roots - does not want to take in Muslims in large numbers as they pose a security risk.
Human rights groups criticised Orban's government for stoking fears and xenophobia with an aggressive campaign ahead of the referendum.