Now, even the Trump administration says those cuts may have gone too far.
Adjusted for inflation, IRS funding has been cut by about 25 percent since the beginning of the decade. And staffing for tax enforcement has fallen by nearly a third.
"Right now we have about the same number of auditors at the IRS that we had back in the 1950s when the economy was about one-seventh the size," said Chuck Marr, who studies tax policy at the left-leaning Center on Budget and Policy Priorities.
The IRS has always been something of a political punching bag. When Republicans took control of the House in 2010, they began pushing for deep cuts in the agency's budget.
"I know that when we were in the majority, that we took great pleasure in cutting the amount of money that was going to the IRS every year," Rep. Dave Joyce, R-Ohio, said last week.
The IRS made an easy target. The agency was the enforcer of the Affordable Care Act, which Republicans despise. There were charges the IRS unfairly penalized conservative groups that tried to apply for tax-exempt status. And even in the best of times, people don't have warm and fuzzy feelings about the tax collector.
"There's always a concern among Americans that you have the jack-booted thugs coming in to your business or office or home and auditing away," said Mark Mazur, who was assistant Treasury secretary for tax policy in the Obama administration.
Over time, both the criticism and the budget cuts have taken a toll.
Morale at the agency is down, experienced agents are heading for the exits, and less than 3 percent of the IRS workforce is under 30, said Tony Reardon, national president of the National Treasury Employees Union, which represents IRS workers.
"The IRS has essentially lost an entire generation of employees," he said. "When you look at pay freezes, retirement cuts, government shutdowns, it makes it incredibly difficult to recruit and retain new talent."
With the IRS cutting back sharply on tax enforcement, the federal government is leaving money it's owed by taxpayers on the table.
The number of individual audits fell 42 percent between 2010 and 2017, according to ProPublica. During that time, IRS scrutiny of taxpayers making more than $500,000 dropped more sharply than its audits of those making under $200,000.
"This clearly indicates a policy shift of some sort — cracking down on people who have little and going easier on people who are doing very, very well," Rep. Charlie Crist, D-Fla., complained during a congressional hearing last week.
Now, after nearly a decade of cutting, the Trump administration is calling for a course correction. President Trump has suggested side-stepping budget caps to spend an extra $362 million on tax enforcement next year.
The new IRS commissioner, Charles Rettig, argued that additional spending is necessary to reassure Americans that tax cheats stand a good chance of getting caught.
"Personally, I have both eyes focused on enforcement," he told lawmakers last week.
Rettig notes the vast majority of Americans — 83 percent — already pay the taxes they owe voluntarily. But if a more visible cop on the beat could boost compliance by just one percentage point, that would mean an extra $30 billion a year in revenue for the government.
On the other hand, government revenues could take an equally big hit if the lack of enforcement triggers even a small drop in Americans' willingness to pay.
"Americans don't really want to feel like chumps," said Mazur, who now directs the nonpartisan Tax Policy Center. "They want to make sure like everybody is paying their fair share."