A third swimmer, James Feigen, is in contact with Brazilian authorities and plans to make further statements to them on Thursday, said a spokesman for the U.S. Olympic Committee (USOC).
Federal police also want to question U.S. gold medallist swimmer Ryan Lochte, one of swimming's most decorated Olympians, but he had already flown home to the United States on Monday, a police spokesman said.
The four swimmers have said they were robbed by gunmen while returning to the Athletes' Village in a taxi in the early hours of Sunday after a party - an incident that stoked fears for the safety of athletes and visitors at South America's first Games.
The Rio Olympics have been dogged by a series of security scares, including the robbery of two visiting government ministers, a mugger being shot dead outside the opening ceremony and stray bullets being fired into the equestrian centre.
On Wednesday, The Guardian newspaper reported that a British team member had been held up at gunpoint while enjoying a night out in Rio in the early hours of Tuesday. The person was not seriously hurt, the newspaper added.
However, in the case of the four swimmers, police sources have said investigators have not found any evidence so far to back up their accounts.
Television images showed U.S. Olympic swimmers Gunnar Bentz and Jack Conger walking into a police office at the airport to give testimony. A spokesman for the U.S. Olympic Committee (USOC) confirmed the two men were taken from their flight.
"We can confirm that Jack Conger and Gunnar Bentz were removed from their flight to the United States by Brazilian authorities," said Patrick Sandusky, the USOC spokesperson.
"They were released by local authorities with the understanding that they would continue their discussions about the incident on Thursday."
Earlier on Wednesday, a judge had ordered police to seize the passports of Lochte and Feigen to prevent them leaving the country, apparently unaware that Lochte had already left.
None of the swimmers face charges, the source said.
Police have taken the passports of Bentz and Conger pending their testimony, O Globo newspaper said on its web site.
People magazine reported that Lochte was spotted at an airport in North Carolina on Wednesday with his girlfriend, Kayla Rae Reid, a Playboy model. Reuters could not verify this.
Lochte's attorney did not return calls for comment.
Judicial sources said the judge would consider whether to ask Lochte to return to Brazil to give testimony or allow him to do so in the United States. He had already given testimony to Brazilian police before leaving the country.
TAXI DRIVER SOUGHT
On Sunday, Lochte told U.S. media the robbers had carried police badges when they pulled the taxi over. They ordered them to drop to the ground and demanded their wallets and belongings, he said, adding he had initially tried to resist.
"The guy pulled out his gun, he cocked it, put it to my forehead and he said, 'Get down,' and I put my hands up, I was like 'whatever'," he told NBC. "He took our money, he took my wallet - he left my cell phone, he left my credentials."
In a fresh interview with NBC that has yet to be aired, Lochte had slightly modified an aspect of his story, Today Show host Matt Lauer said.
Instead of being pulled over by the men, Lochte said the swimmers stopped at a gas station to go to the bathroom when the robbers showed up. He also said a gun was not put to his head but was pointed in his direction.
"I think he feels it was more of a traumatic mischaracterization," said Lauer.
Police sources have told Reuters in recent days they have been unable to find the taxi driver or to corroborate any testimony provided by the swimmers. On Wednesday, police issued an appeal for the taxi driver to come forward.
The judicial sources said the judge in her ruling pointed to an inconsistency between Lochte, who had said only one man was involved in the robbery, and his team mate, who testified that several men had robbed them.
The judge also noted that video footage of the swimmers showed them returning to the Athletes Village joking with one another and in possession of many of their belongings, including cellular phones and Olympic accreditation.
She also highlighted that some of the swimmers testified to leaving a party at the French Olympic delegation just after 4 a.m. but security cameras showed them arriving at the Athletes Village at around 7 a.m., the sources said. That journey would normally only take around 30 minutes at that hour of day.
On his official Twitter account, Lochte said: "My hair is going back to its normal colour tomorrow", in what appeared to be a tongue-in-cheek reference to changing his appearance. The swimmer competed in Rio with his hair dyed peroxide blond.