According to conventional neoconservative, Free Market ideology, reducing the size of government and eliminating regulatory mechanisms leads to increased efficiency (government is defined to be inefficient as an axiom of the belief system whether or not, in any given instance, it actually is).
Like many "common sense" propositions, "cutting fat to improve efficiency" is, at best, only partly true, of limited (though real) applicability. Like all good, but limited, ideas, carrying it to illogical limits leads to destructive, counterproductive blowback (unintended consequences).
A good example of the such blowback was revealed in the Charbonneau Commission's investigation into corruption in the construction industry in the Province of Québec.
Wiki: Charbonneau Commission
Budget cuts can lead to increased expenses in the long run, if they are not judiciously implemented and monitored to optimize the benefit to cost ratio of services provided to the public sector (that is, if cuts are implemented in a hamfisted, ideologically driven fashion).
One of the witnesses being grilled before the Charbonneau Commission into construction industry corruption in the city of Montréal observed that staff reductions in the Montréal Public Works Dept meant that oversight of public work contracts was reduced. The reduced oversight had the perverse, counterproductive effect of making it easier to inflate cost estimates and pay off corrupted officials. For example, cost overruns on projects would be inflated or simply invented with a kickback payed to a corrupted civil engineer so he would approve the additional billing. In addition, with oversight delegated to private sector consulting firms, the city lost the inhouse competency to carry out its own monitoring. This lead to increased costs for oversight and / or to reduced capacity. So the net effect of ideologically inspired budget cutting is to save a few bucks with the right hand while the left hand picks the right hand's pocket. Hidden costs - economists call them "externalities"..
internal blog link: http://transparencycanada.blogspot.ca/2012/10/charbonneau-commission-when.html