The dead included the assailant and the policeman he stabbed, while the other two victims were among the pedestrians hit by the car as it tore along Westminster Bridge before crashing into railings just outside parliament.
"We've declared this as a terrorist incident and the counter-terrorism command are carrying out a full-scale investigation into the events today," Mark Rowley, Britain's most senior counter-terrorism officer, told reporters.
"The attack started when a car was driven over Westminster Bridge, hitting and injuring a number of members of the public, also including three police officers.
"A car then crashed near to parliament and at least one man, armed with a knife continued the attack and tried to enter parliament."
Reuters reporters inside parliament at the time heard loud bangs and shortly afterwards saw the knifeman and the stabbed policeman lying on the ground in a courtyard within the gates of parliament.
A Reuters photographer saw at least a dozen people injured on the bridge. His photographs showed people lying on the ground, some of them bleeding heavily and one under a bus.
A woman was pulled alive, but with serious injuries, from the Thames, the Port of London Authority said. The circumstances of her fall into the river were unknown.
Three French schoolchildren aged 15 or 16 were among those injured in the attack, French officials said.
The attack took place on the first anniversary of attacks by Islamist militants that killed 32 people in Brussels.
Prime Minister Theresa May was being kept updated and was due to chair a meeting of the government's crisis response committee.
"The thoughts of the PM and the government are with those killed and injured in this appalling incident, and with their families," her office said in a statement.
Eyewitnesses described chaotic scenes during the attack.
"I just saw a car go out of control and just go into pedestrians on the bridge," said Bernadette Kerrigan, who was on a tour bus on the bridge at the time, in an interview with Sky News.
"As we were going across the bridge, we saw people lying on the floor, they were obviously injured. I saw about 10 people maybe. And then the emergency services started to arrive. Everyone was just running everywhere."
The House of Commons, which was in session at the time, was suspended and lawmakers were asked to stay inside.
Interior minister Amber Rudd said the government's top priority was people's security.
"The British people will be united in working together to defeat those who would harm our shared values: values of democracy, tolerance and the rule of law," she said.
May spoke to U.S. President Donald Trump about the attack, the White House said.
U.S. Secretary of State Rex Tillerson condemned what he described as "horrific acts of violence".
Polish former Foreign Minister Radoslaw Sikorski, who witnessed the attack from a taxi as he crossed the bridge, said he saw five victims and made a video of the scene.
"I heard something that sounded like a small car crash. Then I looked out of the window and saw that there was one person lying on the asphalt," he told Reuters.
"I did not see the face of the person lying on the asphalt, but the person was not moving, it was not showing any signs of life. One of the men I saw, his head was bleeding very badly. But the person I filmed - no, that person was not showing any signs of life."
Journalist Quentin Letts of the Daily Mail newspaper told LBC radio that he had witnessed the stabbing of the policeman and the shooting of the assailant from his office in the parliament building.
"He (the assailant) ran in through the open gates ... He set about one of the policemen with what looked like a stick," Letts said.
"The policeman fell over on the ground and it was quite horrible to watch and then having done that, he disengaged and ran toward the House of Commons entrance used by MPs (members of parliament) and got about 20 yards or so when two plain-clothed guys with guns shot him."
In Edinburgh, the Scottish parliament suspended a planned debate and vote on independence as the news from London came in.
Britain is on its second-highest alert level of "severe" meaning an attack by militants is considered highly likely.
In May 2013, two British Islamists stabbed to death soldier Lee Rigby on a street in southeast London.
In July 2005, four British Islamists killed 52 commuters and themselves in suicide bombings on the British capital's transport system in what was London's worst peacetime attack.
Wednesday's attack was near the location where, in 1979, Airey Neave, a Member of Parliament who was a close aide to Margaret Thatcher, was killed by a car-bomb claimed by the Irish National Liberation Army.