The attacker, who was also wearing an explosive vest, reached the second floor, where the defence minister and the army chief of staff have offices.
He managed to shoot aides to the minister and the chief-of-staff before being killed by bodyguards.
The Taliban said they had intended to kill the "puppet" defence minister.
The BBC's Bilal Sarwary in Kabul says this is the fourth deadly attack in the last five days, but the first time a gunman has got so close to an such senior officials.
On Saturday, five Nato soldiers, four Afghan soldiers and an interpreter were killed when a Taliban suicide bomber wearing a military uniform hit an Afghan army base near the eastern city of Jalalabad.
On Friday, a suicide bomber dressed as a policeman blew himself up inside the police headquarters complex in the city of Kandahar, killing the police chief of the southern province, Khan Mohammad Mujahid.
A senior Afghan security official told the BBC that the attacker was wearing the uniform of a colonel in the Afghan National Army (ANA).
He managed to pass through checkpoints outside and inside the ministry building with a weapon and suicide vest because he had a valid ID, the official added.
After reaching the second floor of the building, he was confronted by bodyguards deployed there and a fire-fight erupted.
The assailant was eventually killed, but not before he had shot dead two ANA soldiers and wounded seven others, including the two senior aides.
However, he was not able to detonate his suicide vest, the security official said. Local media have reported that there was an explosion.
A Taliban spokesman said they were behind the attack and had intended to kill Defence Minister Abdul Rahim Wardak. Mr Wardak was not inside the building at the time, the official added.
Separately, the Taliban told the Associated Press that the attack had been planned to coincide with the visit of French Defence Minister Gerard Longuet, who they believed was meeting officials inside the ministry.
Mr Longuet, who met French troops in eastern Afghanistan on Sunday, had been scheduled to meet Afghan officials on Monday at an unspecified location, French military spokesman Lt Col Eric de Lapresle said.
Officials in Paris told the AFP news agency they had no evidence that the attack was an attempt to kill Mr Longuet.
The defence ministry is located in the centre of Kabul, close to the presidential palace and other ministries.
The road leading to the building was closed off on Monday afternoon, with even ministry employees being turned away.
Our correspondent says serious questions will be asked how the attacker got inside one of Afghanistan's most important buildings.
On Sunday, a defence ministry spokesman said it had been able to stop potential suicide bombers and the Taliban infiltrating the army.