Aijalon Mahli Gomes, 31, was sentenced in April to eight years' hard labour after being found guilty of illegally entering the country.
Yonhap news agency said Mr Carter left the US in a private jet on Tuesday.
Foreign Policy magazine reported on Monday that Mr Carter would leave for North Korea "within days".
There has been no confirmation of the visit from Mr Carter himself.
CNN said the White House knew about the trip, citing unidentified officials who described it as a humanitarian mission.
On Tuesday, White House spokesman Bill Burton said he would not "comment on anything that could have a negative impact on any private humanitarian mission that might be happening".
"We obviously think that Mr Gomes should be released. There will probably be more information on this sometime in the future," he said.
Mr Gomes, who had been teaching English in South Korea, reportedly crossed the border from China into North Korea in January.
He was visited by a US official and two doctors in a hospital in Pyongyang earlier this month. North Korea said in July that Mr Gomes had tried to commit suicide.
If Mr Carter did negotiate his handover by North Korea, he would be following in the footsteps of another former US president, Bill Clinton.
Last year Mr Clinton secured the release of two US journalists detained in North Korea for crossing the border.
The journalists, Laura Ling and Euna Lee, were pardoned and returned to the US with Mr Clinton.