With 98% of votes counted, President Lula's former cabinet chief has 47% with Jose Serra trailing on 33%.
The two will contest a run-off vote in four weeks' time.
A strong showing by the Green Party candidate, Marina Silva, who polled 19%, may have cost Ms Rousseff a first-round win.
"We can confirm there will be a second round in the presidential elections," Ricardo Lewandowski, the president of the Supreme Electoral Tribunal, told reporters in Brasilia late on Sunday.
Workers Party candidate Dilma Rousseff is the favoured successor to President Luiz Inacio Lula da Silva, who has completed two terms, and cannot run for a third.
"We are warriors, and we are accustomed to challenges," she said in a speech in Brasilia after the result was announced. "We do well in second rounds."
Ms Rousseff was the front runner for much of the campaign and benefited from Lula's widespread popularity and the country's booming economy.
Many analysts believe a scandal involving her directly would be the only scenario under which she could lose a runoff.
"This is an electoral climate that favours the incumbent party," political analyst Luiz Piva told the Reuters news agency. "Brazilians are generally very happy with their government."
Centre-left candidate Jose Serra, the Social Democratic former governor of Sao Paulo state, had seen a boost in his support after corruption allegations surfaced involving a former aide of Ms Rousseff.
But Ms Silva seems to also have benefited. The third-place candidate's 19% share of the ballots was far higher than the 14% forecast for her.
"We defended a victorious idea and Brazil heard our cry," Ms Silva said in reaction to the result.
Brazil, one of the world's most populous democracies, was also choosing local and national representatives.
Voting is compulsory in Brazil, with results coming quickly, thanks to Brazil's electronic voting system.
Ms Rousseff, 62, served as Lula's chief of staff from 2005 until this year, and is a career civil servant. Her run for the presidency is her first attempt at elected office.
During the 1960s and 1970s she was involved in the armed struggle against Brazil's military rulers, and was jailed for three years.
The 68-year-old Mr Serra is hugely experienced, having served as Sao Paulo mayor, Sao Paulo state governor and health minister under President Fernando Henrique Cardoso Mr Lula's predecessor. He lost the presidential election in a run-off to Mr Lula in 2002.