Rabbi Ovadia Yosef, spiritual leader of Shas, spoke out as Middle East talks are poised to begin in Washington.
The United States condemned the remarks as "deeply offensive".
Israeli PM Benjamin Netanyahu distanced himself from the comments with a statement saying that his government wanted peace with the Palestinians.
The attack on Mr Abbas, delivered in the rabbi's weekly sermon, also prompted chief Palestinian negotiator Saeb Erakat to condemn the remarks as "an incitement to genocide".
Mr Erakat urged the Israeli government "to do more about peace and stop spreading hatred", the AFP news agency reported.
Rabbi Yosef expressed the wish that "all the nasty people who hate Israel, like Abu Mazen (Abbas), vanish from our world".
He went on to say: "May God strike them down with the plague along with all the nasty Palestinians who persecute Israel."
The remarks come as Mr Netanyahu is due in Washington this week for direct peace talks with Mr Abbas.
US President Barack Obama hopes to bring the leaders together on Thursday for the first face-to-face discussions since December 2008, when the Palestinians broke off negotiations over Israel's offensive against the Gaza Strip.
The US response to Rabbi Yosef, a founder of the ultra-Orthodox Shas party, was swift.
In a statement, US State Department spokesman Philip Crowley said: "We regret and condemn the inflammatory statements by Rabbi Ovadia Yosef.
"These remarks are not only deeply offensive, but incitement such as this hurts the cause of peace."
Mr Netanyahu's office issued a statement saying that Rabbi Yosef's comments "do not reflect the views of Benjamin Netanyahu or of his government".
The 89-year-old former chief rabbi of Israel has been at the centre of controversy before, with comments about Arabs, secular Jews, liberals, women and gays.
In 2001, during a Palestinian uprising, he called for the annihilation of Arabs and said it was forbidden to be merciful to them.
He later said he was referring only to "terrorists" who attacked Israelis.Meanwhile, Mr Abbas, the president of the Palestinian Authority, said on Sunday that Israel's policy of settlement-building could undermine the new round of peace talks.
"Israel will be held accountable for the failure of the talks if settlement construction should continue," Mr Abbas said in a speech delivered in Jordan.
"The negotiations need to bring about serious action that will be able to bring liberation from the occupation and independence."
Mr Abbas said Palestinians understood Israel's need for security, but said that need should not be an "excuse to expand settlements and steal land".
In the statement responding to Rabbi Yosef's remarks, Mr Netanyahu's office said Israel intended to negotiate in good faith in Washington.
"Israel plans to take part in peace negotiations out of a desire to advance toward a peace agreement with the Palestinians that will end the conflict and ensure peace, security and good neighbourly relations between the two peoples," the statement said.
Analysts say expectations for the latest round of talks are low, with the settlement issue just one of a number of area of difference.