One of the officials cited by the newspaper said the alleged misconduct involved a male and occurred during Hastert's time as a high school wrestling coach and teacher in Yorkville, Illinois, before becoming a lawmaker.
Hastert could not immediately be reached for comment.
The New York Times also reported that Hastert was paying a man to not say publicly that Hastert had sexually abused him decades ago, citing two people briefed on evidence uncovered in an FBI investigation.
Announcing the charges on Thursday, prosecutors said they related to Hastert's alleged effort to hide $3.5 million in payments he was making to a person to conceal past misconduct, but did not detail the nature of the misconduct.
Asked why Hastert, an Illinois Republican who become his party's longest-serving House speaker before leaving Congress in 2007, was making the payments, the official cited by the Los Angeles Times told the paper it was intended to conceal Hastert's past relationship with the male.
"It was sex," said the official, who spoke on condition of anonymity.
Hastert, 73, was charged with structuring the withdrawal of $952,000 in cash to evade the requirement that banks report cash transactions over $10,000, and lying to the FBI about his withdrawals.
Each count of the two-count indictment carries a maximum penalty of five years in prison and a $250,000 fine.
Before his terms in Congress, Hastert served three terms as an Illinois state representative and was a teacher at Yorkville High School about 45 miles (70 km) southwest of Chicago for 16 years.
The Yorkville school district that employed Hastert from 1965 to 1981 said in a statement on Friday it "has no knowledge of Mr. Hastert’s alleged misconduct, nor has any individual contacted the District to report any such misconduct."
The district said it was first made aware of any concerns regarding the former congressman when the indictment was released.
Judge Thomas Durkin has been assigned to the case, but has not yet set a date for Hastert's first appearance. The former congressman has not been arrested and is not considered a flight risk, a spokeswoman for the U.S. Attorney's Office in Chicago said.
The indictment covers bank transactions and Hastert's responses to investigators and not the potential misconduct.
Kelly Griffith, general counsel for the Illinois Coalition Against Sexual Assault, said that if any sex crimes were committed, it was unlikely a person could face charges for such crimes in Illinois from 1965 to 1981 because they would likely be viewed as expired offenses under a statute of limitations.
The indictment alleges Hastert agreed sometime in 2010 to pay the person $3.5 million in compensation to conceal the misconduct, withdrawing more than $1.7 million from various domestic bank accounts from 2010 to 2014.
He made 15 withdrawals of $50,000 each from June 2010 until April 2012 when bank representatives, who are required to report cash transactions of over $10,000, asked him about the withdrawals, the indictment said.
Hastert then began to make cash withdrawals in increments under $10,000 each, totaling $952,000, starting in July 2012, the indictment said.
The FBI and the Internal Revenue Service began investigating the withdrawals in 2013 and the indictment alleges Hastert lied to FBI agents when they interviewed him about them in December.
Hastert has resigned from the board of advisers to the J. Dennis Hastert Center for Economics, Government and Public Policy at his alma mater Wheaton College in Wheaton Illinois, the school said on Friday.
He graduated from Wheaton College in 1964, where he was also a wrestler.
Hastert stepped down from the board of directors of ethanol producer REX American Resources Corp (REX.N) on Friday, according to a regulatory filing.
On Thursday, he resigned from the Washington lobbying firm Dickstein Shapiro, which he joined in 2008, and resigned as a board member of the exchange operator CME Group Inc.
White House spokesman Josh Earnest, during his daily briefing, declined to comment specifically on the allegations against Hastert but said, "There's nobody here who derives any pleasure from reading about the former speaker's legal troubles at this point."