Thousands of devotees had packed into the Puttingal Devi temple, about 70 km from the state capital Thiruvananthapuram in the coastal district of Kollam, to watch the display that started at midnight and went on four hours.
The blaze started when a cracker fell onto a shed where the fireworks were stored, sparking a string of powerful explosions that blew the roof of the administrative block of the temple and caused another building to collapse, residents said.
"There were body parts on the floor and on the roof there was an arm," Anita Prakash, a resident said. "In the past, there's been fireworks but not on this scale."
Kerala is studded with temples managed by rich and powerful trusts that often flout local regulations. Each year temples hold fireworks displays, often competing to stage the most spectacular ones, with judges who decide the winners.
Kollam district magistrate A. Shainamol said people living in the area near the temple had complained about the danger of these fireworks in the past.
Prime Minister Narendra Modi flew to Kollam with a team of doctors to help state authorities cope with the large number of injured, moving swiftly to head off criticism of a lack of public safety.
"The fire at the temple in Kollam is heart-rending and shocking beyond words," he said in a Twitter post. "My thoughts are with families of the deceased and prayers with the injured."
Modi has faced public criticism in the past for failing to respond quickly to disasters such as the floods in Chennai late last year. Large parts of the metropolis were under water for days before government help arrived.
Earlier this month, a flyover under construction in the eastern city of Kolkata for years collapsed killing 27 people, prompting allegations that shoddy material was used and that the metal parts had corroded during the years of delay.
Television images from the devastated temple site showed people, some clutching children with burn injuries, being taken to hospitals. Others carried out charred bodies of victims.
Kerala's Home Minister Ramesh Chennithala told Reuters that 60 of the 100 dead had been identified while the number of people admitted to hospitals in Kollam and the state capital had risen to 383.
With Kerala in the midst of an election for a new state assembly, the fire quickly turned into a political issue as local leaders demanded an investigation.
In Kerala, Modi's Bharatiya Janata Party (BJP) trails its Congress party and communist rivals. The BJP has focused on building up grassroots strength in the state for decades.
Kerala's Chief Minister Oommen Chandy said the Kollam district administration had denied the temple authorities permission for the display and the government would investigate why they went ahead anyway.
"There was no permission to even store the fireworks," he said.
Witnesses said they had repeatedly warned the local administration about the danger of firework displays in the crowded neighborhood. Chennithala said a case for illegal possession of explosives had been registered against the temple.
The temple trust was not immediately available for comment.
The explosion from the fireworks was so strong that some parts of the temple roof caved in. Local media showed bulldozers trying to clear the area of the debris.
The Puttingal temple is one of the oldest in the state. It was built on the site of an ant hill where locals believe a goddess appeared centuries ago.