I’ve always found this to be one of Adam Smith’s most powerful quotations:
Little else is requisite to carry a state to the highest degree of opulence from the lowest barbarism, but peace, easy taxes, and a tolerable administration of justice: all the rest being brought about by the natural course of things.
But is Smith right? Well, not exactly. These policies or “institutions” are certainly a good start. But I’d argue that certain types of virtue, mores, character, and values are also vital preconditions for opulence. Indeed, one might argue that these variables are prior conditions for the peace, easy taxes, and administration consistent with both a long-lasting free society and the opulence that flows from it. And they are hardly natural but things that take a lot of effort, education, and wisdom to develop/build. Individuals, families, churches, and other civic associations are the forges of these things. Unfortunately, the state often acts in ways destructive of them – an unintended consequence of the smothering love of the leviathan.
I tread in dangerous waters here given that Jim Otteson, my fellow blogger here at Pileus, has clearly spent a few more moments thinking about Smith than I have! Perhaps a signed copy will enlighten me if I’m wrong. 😉
PILEUS 
I’ll defer to Jim’s judgment here too, but I would read Smith here in just the light that you do. That is, these kinds of character and culture are necessary conditions for peace and tolerable administration of justice. The overall thought, then, is that if you have those, you’re done. In particular, you don’t need people trying to decide what way to run the country to get it back on the right track, etc. No men of system required, in particular.