The gunman in the parliament building was shot dead, and Harper was safely removed.
Canadian police said they could not "at this point" confirm whether the man who shot dead the soldier, who was guarding the National War Memorial in central Ottawa, was the same person who shortly afterwards attacked the parliament building.
Witnesses said a flurry of shots were fired after a gunman entered the parliament building, pursued by police.
The assault took place very near the room where Harper was meeting with members of his Conservative party, a government minister said.
"PM (Harper) was addressing caucus, then a huge boom, followed by rat-a-tat shots. We all scattered. It was clearly right outside our caucus door," Treasury Board Minister Tony Clement told Reuters.
The incident, shocking in Canada's normally tranquil capital, began shortly before 10 a.m. ET and was not overlate on Wednesday afternoon. Parliament and buildings in downtown remained on emergency lockdown.
Canadian police were investigating a man named as Michael Zehaf-Bibeau as a possible suspect in the shootings, a source familiar with the matter said. Two U.S. officials said U.S. agencies had been advised the suspect was a Canadian convert to Islam.
Security in Ottawa came under criticism after the gunman was able to run through the unlocked front door of the main parliament building. Police said an operation was under way to make parliament safe.
"It caught us by surprise... If we had known that this was coming, we would have been able to disrupt it," Gilles Michaud, assistant commissioner of the Royal Canadian Mounted Police, (RCMP) told a news conference.
It was unclear whether there was any connection between Wednesday's shootings and an attack on Monday when a convert to Islam ran down two Canadian soldiers with his car, killing one, near Montreal, before being shot dead by police in the first fatal attack on Canadian soil tied to Islamic militants.
No group, Islamic or otherwise, claimed responsibility for either the attack in Ottawa or the one near Montreal. Monday’s attacker, 25-year-old Martin Rouleau, who converted to Islam last year, was among 90 people being tracked by the RCMP on suspicion of taking part in militant activities abroad or planning to do so.
Canada announced this month it was joining the battle against Islamic State fighters who have taken over parts of Iraq and Syria.
GUNS DRAWN IN PARLIAMENT
From witness accounts it appeared the suspect dashed into parliament, ran past the room where Harper was speaking and was gunned down outside the entrance to the library, only about 60 feet (20 meters) away.
Dramatic video footage posted by the Globe and Mail newspaper showed police with guns drawn inside the main parliament building. At least a dozen loud bangs can be heard on the clip, echoing through the hallway.
Veterans Affairs Minister Julian Fantino, a former policeman, told the Toronto Sun that parliament's head of security, Sergeant-at-Arms Kevin Vickers, shot dead a suspected gunman.
Canadian cabinet minister Jason Kenney said a guard in parliament buildings was also wounded in the incident.
Harper stressed that government and parliament should continue its work, a spokesman said. "While the prime minister stated that facts are still being gathered, he condemned this despicable attack," the spokesman said.
Harper was expected to make a statement later in the day.
Canada said on Tuesday it had raised the national terrorism threat level to medium from low because of a rise in "general chatter" from radical groups such as Islamic State and al Qaeda but said there had not been a specific threat.
The RCMP's Michaud said the threat level on Parliament Hill had been on medium for some time.
The soldier who died in the shooting at the War Memorial was identified as Cpl. Nathan Cirillo, according to his aunt.
Cirillo was a member of the Argyll and Sutherland Highlanders, a regiment of Reserve Forces based in Hamilton, and was training to join the Canada Border Services Agency, a federal agency responsible for border and immigration enforcement, his aunt told The Globe and Mail.
It was not clear if Cirillo was armed or not when he was shot.
'YOU ARE IN DANGER HERE'
As the drama in Ottawa unfolded, police in dark bulletproof vests and carrying automatic rifles flooded the streets near parliament, clearing several blocks of downtown Ottawa.
Some took cover behind vehicles and shouted to people to clear the area, saying: "We do not have the suspect in custody. You are in danger here."
When the shooting started, most members of parliament were in the two caucus rooms past which the gunman ran. Members were told to lock or barricade themselves in their rooms or offices, and stay away from the windows.
A tweeted picture sent from the room where the opposition New Democrats were holding a weekly caucus showed a pile of chairs jammed up against the main door to prevent anyone from entering.
In Washington, a White House official said U.S. President Barack Obama had been briefed on the situation. He said the United States had offered assistance to Canada.
Canadian and U.S. stock markets declined after the shootings in Ottawa. The Toronto Stock Exchange's TSX index .GSPTSE dropped 1.6 percent, while the S&P 500 .SPX gave back 0.7 percent.
Mass shootings are relatively rare in Canada, which has stricter gun laws than the United States, and the regulations at one point included a national registry of rifles and shotguns. Legislation was passed in 2012 to scrap the registry.
Ottawa also has a low murder rate. There were nine homicides in 2013 and seven in 2012, in a city of 885,000 people. Compared with Capitol Hill in Washington, security on Parliament Hill is also fairly low key. Anybody could walk right up to the front door of parliament's Centre Block with arms and explosives without being challenged before entering the front door, where a few guards check accreditation.
Centre Block is the main building on Parliament Hill, a sprawling complex of buildings and open space in downtown Ottawa. It contains the House of Commons and Senate chambers as well as the offices of some members of parliament, senators, and senior administration for both legislative houses.
A construction worker who was on the scene in Ottawa when the shooting began told Reuters he heard a gunshot, and then saw a man with a scarf over his face running towards parliament.
"He was wearing blue pants and a black jacket and he had a double barreled shotgun and he ran up the side of this building here and hijacked a car at gunpoint," construction worker Scott Walsh told Reuters.
The driver got out safely, then the man drove the car to the Centre Block, where construction work is under way, Walsh said.
The Canadian military closed its bases across the country to the public following the events in Ottawa, CBC TV said.
The attacks in Ottawa and Quebec took place as the Canadian government prepared to boost the powers of its spy agency, the Canadian Security Intelligence Service. Public Safety Minister Stephen Blaney said last Thursday the new legislation would let the agency track and investigate potential terrorists when they travel abroad and ultimately prosecute them.