Trump has alienated his party and failed to establish a competent campaign operation, the paper said in an editorial.
The Journal's editorial board, which generally favors Republicans, has been critical of Trump and has questioned his conservative credentials, but its warning on Monday was its strongest attack yet. It echoed growing alarm about Trump's candidacy among many leading Republicans who have been slow to embrace him or have completely distanced themselves.
The New York real estate developer, who has never held elected office, has been mired in weeks of controversy and opinion polls show him falling behind Democratic candidate Hillary Clinton in the race for the Nov. 8 election.
The Journal urged Trump's backers to push the candidate to conduct himself with a more presidential demeanor and begin running a more disciplined campaign.
"If they can’t get Mr. Trump to change his act by Labor Day, the GOP will have no choice but to write off the nominee as hopeless and focus on salvaging the Senate and House and other down-ballot races," it said.
Labor Day, which falls on Sept. 5 this year, marks the end of U.S. summer vacations and traditionally launches the final phase of the long U.S. election season.
"As for Mr. Trump, he needs to stop blaming everyone else and decide if he wants to behave like someone who wants to be president - or turn the nomination over to Mike Pence," it said, referring to the Indiana governor, who is Trump's vice presidential running mate.
Trump has repeatedly provoked controversy in the weeks since his formal nomination as the Republican presidential candidate in July, despite appeals from party leaders for him to focus on issues that could win him the election.
He picked a fight with the parents of a Muslim U.S. Army captain who was killed in Iraq and falsely accused President Barack Obama and Clinton of being "co-founders" of Islamic State. He later said he was being sarcastic but has continued to repeat the remark.
In an effort to right his campaign, Trump will deliver his second policy speech in as many weeks on Monday, this time an address on foreign policy. Speaking in Youngstown, Ohio he will outline his plan to defeat Islamic State.
Most controversially, Trump has long said he will impose a temporary ban on Muslims entering the country, and has said he would "knock the hell out" of Islamic State.
Adding to Trump's woes this week was the news, first reported by The New York Times, that the name of his campaign manager, Paul Manafort, was on secret ledgers showing cash payments designated to him of more than $12 million from a Ukrainian political party with close ties to Russia.
Manafort denied any impropriety in a statement on Monday. "I have never received a single 'off-the-books cash payment' as falsely 'reported' by The New York Times, nor have I ever done work for the governments of Ukraine or Russia," he said.
Artem Sytnik, the head of Ukraine’s anti-corruption bureau, confirmed in a briefing with reporters that Manafort's name appeared on a ledger and that more than $12 million had been allocated as an expenditure, referencing Manafort.
But Sytnik said that the presence of Manafort's name "does not mean that he definitely received this money."
The Clinton campaign said the news was evidence of "more troubling connections between Donald Trump's team and pro-Kremlin elements in Ukraine."
Trump has spoken favorably in the past of Russian President Vladimir Putin. Last month he invited Russian hackers to find "missing" emails from Clinton's time as secretary of state, when she used a private computer server to conduct government business, although he later described that comment as sarcasm.
Trump has increasingly begun to portray himself as a victim of the media.
"If the disgusting and corrupt media covered me honestly and didn't put false meaning into the words I say, I would be beating Hillary by 20 (percent)," he wrote in one tweet in a series of complaints about media coverage on Sunday.
The current RealClearPolitics average of national opinion polls puts Clinton 6.8 points ahead of Trump, at 47.8 percent to Trump's 41 percent. Opinion polls also show Trump trailing in states such as Pennsylvania that are likely to be pivotal in the election.