They've been scorned in televised congressional hearings for unfairly abusing tax laws, threatened with questioning in a criminal investigation and accused of using federal jobs to push a political agenda.
At this point in the saga surrounding the Internal Revenue Service and its use of "Tea Party" and other search terms to flag conservative groups while reviewing their applications for tax-exempt status, no one appears to have lost their job.
The official who has become the scandal's public face is on paid administrative leave. The former acting director, who was relieved of his job by President Barack Obama on May 15, departs on Friday.
The names of low-level officials who carried out the practice have been closely guarded by IRS higher-ups and agency's inspector general. No criminal charges have been filed.
There is some caution about a rush to judgment.
"We don't know all the facts yet," said R. Scott Oswald, an attorney who represents federal workers.
Protections exist so workers "are not treated in an arbitrary and capricious way merely for political purposes."
Dean Zerbe, a lawyer who from 2001 to 2008 worked for Republican Senator Charles Grassley on the Finance Committee that oversees the IRS, said the fact that employees are still working "leaves everybody with a bad taste in their mouth. But they do have their due process."
Grassley was among the lawmakers who had complained about the pace of action against IRS officials including Lois Lerner, the director of the tax-exempt division, who became aware in 2011 that some conservative groups were subjected to special scrutiny.
The senior Democrat on the Ways and Means Committee in the House of Representatives also called for Lerner's resignation.
Lerner, who earns about $177,000-a-year, according to a congressional investigator, was placed on paid administrative leave last week. Grassley last week urged a swift review of her case so she is not "in limbo indefinitely on the taxpayers' dime."
Lerner, who set the IRS scandal in motion by apologizing on May 10 at a Washington briefing for tax attorneys, could not be reached, and her attorney had no comment.
Former acting commissioner Steve Miller was pressed by lawmakers to hold individuals accountable in several recent hearings, but he would not name specific employees.
Miller, a near 30-year IRS veteran who was acting head of the agency but was not nominated as permanent director, has earned an estimated $180,000 a year. His government pension remains intact.
"As you sit here today, you were not fired from your job, and I can tell you in my private experience you would have been fired on the spot," Republican Representative Tom Reed told him when Miller gave testimony on Capitol Hill.
Miller's attorney William Burck, with the Quinn Emanuel law firm, said, "he had every right to stay on at the IRS and has done so to facilitate the transition to new leadership."
For a timeline of events in the IRS controversy, see:
Days after Lerner first revealed the IRS targeting practices, a report by the Treasury Inspector General for Tax Administration detailed the controversial IRS strategies to streamline review of applications for tax-exempt status.
At least three Congressional committees and the Department of Justice are now investigating whether any laws were broken.
Federal employees are protected by a 1972 Supreme Court ruling and a 1978 job security law.
The court ruling ensures due process for public sector workers and the law furthered those rights, including giving workers 30 days to respond to charges.
"The notion that you can't fire federal employers - that is a lot of hyperbole. But they can't summarily terminate an employee - because of these rights," said Joseph Kaplan, a lawyer at Passman & Kaplan, who represents federal workers.
MERIT BOARD APPEAL
If the IRS fires an employee, the worker can appeal to a Merit Systems Protection Board, which has 120 days to uphold or overturn the decision.
The General Services Administration, which operates government facilities and purchases supplies for agencies, went through a similar incident last year after Congress investigated its $820,000 expenditure on a lavish conference, prompting the resignation of the agency's chief and a management shake-up.
At least one executive was reinstated after the merit protection board found he had been fired in error.
It may be easier to fire an IRS worker compared to other federal workers due to the 1998 IRS Restructuring and Reform Act, which sets out what is known in tax circles as the "Ten Deadly Sins," according to experts.
The "sins" include providing a false statement under oath and threatening to audit a taxpayer for personal gain.
"These are offenses that are supposed to call for termination as the only penalty," Kaplan said. "It hamstrings the commissioner's discretion in terms of what penalty to mete out."