Why do big IT projects fail?

It’s over thirty years since The Mythical Man Month and we’ve made huge strides in the IT industry in formalising methods and tooling. Nevertheless looking around, we still see projects failing. Why is this?
If you listen to industry leaders like Grady Booch, during the 9th Annual Turing Lecture, they will remind you that building complex systems is intrinsically hard. You’re engineering with peoples’ ideas - fundamentally abstract concepts. You’re then turning those rather amorphous things into instructions that can tell millions of tiny super hot switches whether to turn on or off. The gap in perspective is immense. We have an ever increasing number of layers and technologies to help us make the translation, but each layer adds some complexity to the ultimate solution.
Business has got more complex too, of course. The ability to put complex decisions and processes into computer systems makes it possible, for example, to assign a mobile phone to an individual in minutes. Each mobile that’s commissioned takes thousands of individual steps which need to be co-ordinated. It’s not just the phone and the network; it’s the billing and the accounting, plus the customer support and service. If it was down to a human, it would never happen reliably – and certainly not in minutes. Business is complex and technology has enabled it to become ever more so.
Therefore what we do in engineering big systems is hard and in many was it’s a minor miracle that any very large system gets delivered.
The troubled projects that Kevin and I see are failing mostly because of the need to co-ordinate people and the intrinsic complexity of what those people are trying to achieve. So my hypothesis of the day is: projects don’t tend to fail because of technology flaws or poor solutions; they fail because we don’t contain the complexity or enable it to be communicated.