His decision to quit after just two and a half years in office deals a blow to the European Union, already reeling from multiple crises and struggling to overcome anti-establishment forces that have battered the Western world this year.
The euro fell to 20-month lows against the dollar, with markets worried that instability in the euro zone's third largest economy could reignite a dormant financial crisis and deal a hammer blow to Italy's fragile banking sector.
Renzi's resignation could open the door to early elections next year and to the possibility of an anti-euro party, the opposition 5-Star Movement, gaining power in the heart of the single currency. 5-Star campaigned hard for a 'No' vote.
"I take full responsibility for the defeat," Renzi said in a televised address to the nation, saying he would hand in his formal resignation to President Sergio Mattarella on Monday.
Mattarella will have to embark on a round of consultations with party leaders before naming a new prime minister -- Italy's fifth in as many years -- who will be tasked with drawing up a new electoral law.
Early projections said Renzi managed to win little more than 40 percent of the vote on Sunday following months of bitter campaigning that pitted him against all major opposition parties, including the anti-system 5-Star Movement.
Italy's parties will now have to work together on the new electoral law, with the 5-Star urging a swift deal to open the way for elections in early 2017, a year ahead of schedule.
Opinion polls show Renzi's Democratic Party (PD) is neck-and-neck with the 5-Star, which has called for a referendum on Italy's membership of the euro currency.
Renzi, 41, took office in 2014 promising to shake up hidebound Italy and presenting himself as an anti-establishment "demolition man" determined to crash through a smothering bureaucracy and redraw the nation's creaking institutions.
Sunday's referendum, designed to hasten the legislative process by reducing the powers of the upper house Senate and regional authorities, was to have been his crowning achievement.
However, his reforms so far have made little impact, and the 5-Star Movement has claimed the anti-establishment banner, tapping into a populist mood that saw Britons vote to leave the European Union and Americans elect Donald Trump president.
In one relief for mainstream Europe, Austrian voters roundly rejected on Sunday a candidate vying to become the first freely elected far-right head of state in Europe since World War Two, choosing instead a Greens leader as president.
The biggest immediate loser from the 'No' triumph in Italy could be Italy's third-largest bank, Monte dei Paschi di Siena, which is bowed by bad loans and is looking to raise 5 billion euros ($5.3 billion) this month to stave off collapse.
Investors are likely to shun the operation if political chaos prevails, meaning a state intervention will be needed to save it. Several other lenders also need a cash injection to stay afloat, raising fears of a domino effect.
Economy Minister Pier Carlo Padoan, seen as a possible candidate to replace Renzi, sought to calm nervous markets on Friday, saying there was "no risk of a financial earthquake" if 'No' wins, though there may be "48 hours of turbulence".
A Nov. 25 Reuters poll suggested investors would expect to demand an extra 25 basis points in yield to hold Italian debt over its German equivalent if the reform is rejected, with the euro dipping 1.25 percent.
The risk to stability, meanwhile, is enough to have the European Central Bank preparing to step in if needed.