Brownfield's everywhere

Last week I was privileged to present to the UK’s National Research and Training Initiative’s Large Scale Complex IT Systems (LSCITS) project. LSCITS is a £14.6M programme looking addressing some of the problems of complex systems and creating a new breed of Engineering Doctorates (EngD) skilled in dealing with them.

To quote their own Initiative Rationale:

The motivation for the LSCITS Initiative is the on-going growth in the size and complexity of information technology (IT) systems. Our ability to develop, maintain and manage such systems is falling behind the growth in their complexity. There is a high risk that we will find ourselves reliant on IT systems that we don’t fully understand and that we cannot effectively manage.

The roots of complexity in IT systems are their increasing size; the increasing involvement of many different organisations in their construction and use; and the increasing rate of business and social change that they have to accommodate. To manage and control complexity, we need better technical tools and methods of system development.

People after my own heart!

Presenting alongside me were representatives from Rolls Royce and BT. Each company had been asked to provide its own perspective on the problems of systems complexity and where they thought LSCITS might be able to help.

Extraordinarily, despite the fact the companies were presenting from very different backgrounds (complex aero-engineering, national telecoms provision and commercial system development) we all had very similar things to say. In particular:

• We all suffered from a lack of rigor caused by lack of formal tooling use. Even when tools were available, people were not happy using them. As a result the level of documentation was poor, reducing the opportunity for reuse of existing assets.
  
  
  
• People were much happier using tools that they were familiar with (Word, Excel etc.) it was unlikely that we could move large numbers of people off them, so we need a better way to control and share information using such informal tooling (one representative lamented the failure to establish a single source of truth almost verbatim from the Eating the IT Elephant book!)
  
  
  
• There was an acknowledgement that no-one can start from a Greenfield perspective. Incremental change and re-engineering is the only cost effective route going forward. No-one can afford a lengthy design or implementation period for a new system or product.
  
  
  
• Not only that, but the self-contained design and engineering teams are a thing of the past too – no-one can afford to create everything in house, so components and other elements need to be sourced from outside the organisation. This causes all kinds of problems when the capabilities and limits of these sourced elements are not known.
  

All in all, we were all looking at new degrees of both organisational and technical complexity. Communication of that complexity was the root of each of our problems. Perhaps Brownfield is a pattern for success with all complex IT systems, not just for use by Systems Integrators?

Where are the requirements hiding?

I've just been on a architecture Masterclass - which was nice. Unfortunately the class did little to dissuade the attendees that the IT industry has more than two types of requirement. It stuck to the line that we basically have functional requirements and non-functional requirements.
The Functional requirements are probably the best known. They tell the designer what the system has to do, usually from the perspective of the end user. Nearly everybody agrees these days that Use Cases are a pretty good way of capturing these requirements. I’ve used some alternative more technical approaches on some big projects, but we’ve nearly always had to create use case like perspectives anyway, just so we can communicate to back to the users. If you can get away with agile Joint Application Development techniques where the users and developers sit down together in small grounds then you’ll probably do even better, but often the high burden of detail, testing and sheer size of the projects we deal with means that such an approach gets ruled out.
The second form of requirement – the non-functional requirement are rather more poorly treated by tooling and there are few widely accepted good practices. The non-functionals tell the designer what the characteristics of the solution should be. Not what it has to do, but how it has to do it. The system must be always available. The system must provide sub-second response times 100% of the time. These are just two of the often impossible and conflicting sets of requirements that we deal with as lead architects.
NFRs are the requirements that make designers proud. You’ll rarely hear them boast that the system they created had the best implementation of a “Find Customer” Use Case, but they’ll never stop crowing that it could support 80,000 concurrent users whilst enabling a full recovery from a plane landing on the data centre within fifteen seconds without the loss of any data…
The odd thing is, the more projects I do, the more I am convinced that we miss most of the requirements that we actually need to deal with. They don’t fit into the functional or non-functional space. They are called constraints. Constraints outnumber the other requirements 10:1. They are all those Entreprise Architecture standards you need to conform to, they are the existing data you need to migrate, they are the interfaces to the existing systems, they are the budget, they are the timeframe. Basically they are the difference between the system working in its environment and context, or not. They are the Brownfield aspects of requirement. The NFRs and Use Cases talk about the system as if it lived on its own little planet. They define the boundaries and shape of an isolated bubble. What they conveniently ignore are all the provisos that actually make the system hard to deliver. It’s here that we see implementation projects fail: they adopt a domain model which is incompatible with the rest of the organisation and make integration impossible; they ignore the real complexity of the interfaces with other systems, assuming that an interface is just the definition of a data structure, they ignore the fact that the data used in the system is now duplicated in multiple places.
What’s worse is that these things are often found out late in the project lifecycle resulting in unexpected expense and delays. Why don’t we just accept the fact that systems have to be deployed into complex environments and do what any respecting building architect would do: commission a Site Survey and use its constraints to influence the design? It’s the Brownfield way, you know its right.