The New Normal – Agile and Architecture

I have had the chance to gain experience of leading a small architecture team working in an “agile” (notice the lower-case ‘a’ here) environment. This experience has had me trying to find the middle ground between two largely conflicting approaches to IT – the waterfall approach I have become accustomed to and the agile approach I found myself in. Over time I have become a big fan of the agile way of working given the satisfaction that seems to be demonstrated by the stakeholders to the progress made compared to what I experienced working in a waterfall environment.

Make no mistake, the whole point of architecture within IT is to make the organisation more agile. I would go as far as arguing that any architecture function not achieving this is failing, but that’s probably another post for another day. Architecture brings about this agility by defining common and fit-for-purpose models which act as archetypes for future endeavours in the same space making subsequent requirements cheap(er) and in some cases free.

Enterprise IT in general has recently latched onto “Agile” (upper-case ‘A’ here) as a method to enable faster time to realisation which essentially means reduced cycle times to get from inception to production, particularly the early demonstration of progress. This has largely been driven by the success of this model in the web application development space where the prototyping of user interfaces demonstrated significant value in delivering a fit-for-purpose user experience built quickly and meeting the customer’s requirements. Within an Agile development process, sensible prioritisation ensures the requirements don’t become a laundry list of nice to have features which drive up analysis time and cost with no real benefit. Now the powers that be in IT want to apply this to any and all development. Surely if the Facebooks and Twitters of the world can do this then we can too, can’t we?

I believe, admittedly having never worked in a start-up, that enterprise IT in an established organisation is a whole other animal because of a legacy of interconnected systems which, let’s face it, are often a delicate house of cards lacking any significant fault tolerance (in the business logic sense rather than the technical sense) where even a small amount of misbehaviour can cause significant issues and require a heroic effort to repair the data corruption that ensues – that is if it can be repaired at all. It is therefore understandable why architecture as a process has developed a level of risk adversity demanding a significant effort given to planning and fore-thought as to how to introduce a new jigsaw piece into the puzzle going against the grain of the agile way of thinking which pushes the mantra that “the correct architecture will emerge”. Such a high level of risk adversity is often completely unnecessary unless the systems being designed or maintained are safety control systems for nuclear power stations or the defence industry.

Whilst I can see how the mantra that “the correct architecture will emerge” may apply to the internal architecture of an application when considered in isolation, when it comes to enterprise IT I don’t think that it is quite time for architects around the world to cross-train to another vocation quite yet. There is plenty for architects to do in this new agile world but it requires a change in thinking and approach to work in an environment that seeks to remove the heavily gated waterfall method that is causing the lack of agility.

Let’s examine the typical waterfall process:

  1. Business stakeholder writes a “concept paper” of some description typically 5 or so pages.
  2. The team responsible for making the concept a reality analyse the concept paper in room with the stakeholder in a room – this somewhat morphs the concept, but the written down concept doesn’t change. The business analyst documents 20-30 pages of requirements as a result.
  3. The architect analyses this set of requirements and puts together a design of 30-100 pages including a compliance matrix of how the design meets the requirements the business analyst describes.
  4. Each developer group, of which there are often 5 or 6 involved, take this document and create a system specific design each of about 30-50 pages.
  5. The architect reviews the documents created by the developer groups to ensure compliance with the design he wrote.
  6. The developer groups build what they said they would (or at least broadly), often changing small details to overcome challenges they come across, sometimes involving the architect – they often don’t update their design document as this happens, neither does the architect unless absolutely necessary.
  7. The ‘product’ is tested and released into production, often subtly different to what the architect wanted (even though the architect doesn’t know), and in turn subtly different from what the stakeholder wanted (and it is often painfully clear to him).

This doesn’t include dealing with change requests or pushy project managers. It is quite possible that I am painting a rather ‘doom and gloom’ picture but at best this process is slow and optimistically produces around 250 or so pages of documentation which no-one will ever read again. Worse still the documentation is often out-of-date before the project is even complete. Conversely, the document that people do want to read, the current state architecture, doesn’t get updated with the rigour it deserves because everyone has moved onto the next project and doesn’t have time. The lack of visible pace in this process, demonstrating what is actually being delivered toward the end, means that it doesn’t deal well with the fact that the concept isn’t a static entity and evolves over time, often evolving after development has started but before the project is finished. Often the project manager has committed an end date to the leadership which has now become immovable (as moving it demonstrates incompetence) and development of the solution to the original concept continues unabated, meaning the stakeholder hasn’t got what he wants as what he wants has changed.

One of the key reasons for the success of agile ways of working is the responsiveness of this change and the collaboration that ensures that everyone involves understands what the current view of the concept is, have the power to influence it and are working towards it.

My list of what needs to change for architects to be successful in this environment (and we don’t do all of this in my group, because change takes time) in no particular order is this:

1) Stop being a document production factory and start getting involved with the development teams.
It’s very normal for architects to try to remain disconnected from the teams that do the actual work, often choosing to support development until a system level design is done and even then only to assure compliance to the original design that the architect himself has produced. It’s not that there is anything wrong with development teams (most of the time), indeed some architects were once developers themselves and understand the development process and developer mindset very well, but staying at an arm’s length helps keep an architect pragmatic and agnostic. The speed that agile environments demand means there is less time available to spend writing documents to throw over the fence remedied by more time spent with the people that build stuff, working with them until code hits production, to communicate the approach/design verbally or on a whiteboard. A document which would have been 30-100 pages becomes a summary (or several depending on the size of the project) of about 5 pages in addition to an optional data model diagram – with the rest being done collaboratively. The architect therefore knows what is actually being built, including every subtle change being made, and has the ability to change the course of what is being built if issues arise. The move from BDUF (big design up front) to SDUF (sufficient design up from) is the largest and most difficult paradigm shift architects must deal with.

2) Understand the requirement.
Requirements are often not understood by anyone but the business owner who, quite unlike a good business analyst who has honed his skills in this area, may write conflicting statements as to what the ‘product’ needs to do. It therefore becomes very necessary to spend time with the stakeholders throughout the process to discuss the requirement, to challenge them, and evolve an understanding of what they really want. A business analyst (if one is used at all) should not be used as a human shield by the architect against stakeholders. This should not be treated as a ‘box ticking’ activity resulting in a requirements compliance matrix and it must be remembered that a requirements compliance matrix doesn’t necessarily equate to satisfaction of the stakeholders regardless of whether the resultant product complies with all of the requirements that the human shield documented.

3) Be bloody minded as to how requirements should be solved.
I don’t think there are many architects that struggle to be bloody minded but I do think it’s high time to release this part of their personalities, a part that they often have to work very hard to keep locked up. There is a massive difference between being bloody minded and telling everyone that they don’t know what they are talking about, often being necessary to ask for a little trust and reference a track record of success – it is also important to recognise that everyone else is trying to do a good job and also play their part in enabling the success of the initiative. In order to ensure that embarrassment doesn’t ensue, and trust isn’t lost, it is important to get a good understanding of the requirements. Furthermore, architects are often required to quickly understand how a requirement is likely to evolve over time beyond the current project and this is where the bloody mindedness comes into play with the duty to ensure what is put on paper and how the product is built takes this into account. It isn’t uncommon to have to fight tooth and nail for this, to ensure the door to an appropriate evolution remains wide open. Any architect that doesn’t have the appetite to do this will fail.

4) Actively document the current state in retrospect.
Working in an agile environment means that the time from the start of an initiative to the point where something can first be shown is far shorter than a waterfall environment. The collaborative approach that needs to be put in place to make this happen means that documentation is typically very light at the start of the project. Let’s face it – after a project has been completed no-one ever willingly reads requirements and designs produced unless they need clarity on a small detail. Just like any other project environment, architects are re-assigned to a new project as soon as they have enough bandwidth. The light documentation during projects can be problematic when it comes to new joiners as well as our audit friends who need to understand the architecture, particularly where it currently stands and broadly where it is going. It isn’t particularly useful to be pointed at a very weak current state architecture document, as is often the case, and a whole pile of designs to understand the current state. This means some, if not the majority, of the documentation effort needs to be in actively maintaining a current state architecture as projects hit production to ensure the document of the current state, the document that people are actually interested in reading, has sufficient detail in it so as to be the only document they need. Pretty much all architecture practices say this needs to happen, but in an agile environment it really really needs to happen. This is often a tough sell to leadership as it is seen to be superfluous to delivery and its value not appreciated.

5) Document a broad long term target architecture as far in advance as possible.
Document here is a looser expression referring to painting a picture that everyone has access to understand where the broader programme of work is going. This may well be a sketch on a whiteboard that isn’t removed for quite some time until a paper document is produced. It helps that this sketch exists on a whiteboard at first as it is easy to change and should be drawn in the first few weeks of the first iteration in parallel to development itself. The detailed paper document version of the high level long term target architecture may run in parallel to several iterations but serves as a great guide for new staff being on-boarded as well as a point to reference for everyone else working on the programme. This whole process requires that the architects involved have strong visionary skills making assumptions that often must be staunchly defended.

6) Be comfortable with and sensitive to change.
Change happens, deal with it. I could easily end this point right here and not write any more. I think architects struggle with change because of the gargantuan effort it takes to document the interim steps required to get to the long term target architecture. Change seldom makes significant changes to the long-term target architecture but does usually change the steps that need to be taken to get there. This is partly resolved in an agile environment by not making a huge effort to document the interim steps until it becomes actually necessary and usually a step at a time. This will require rework, but typically only in thinking rather than on paper. Constant and open dialogue with stakeholders and development teams ensures sensitivity to change so that it is not a complete surprise when it does finally come up and the platform can be built to accommodate it as much as makes sense.

7) Take baby steps to get to the architecture you need.
To counter-act the evolving nature of a concept and the short project durations in an agile environment it is often necessary to take a somewhat meandering route toward the target architecture. This means making more tactical decisions than most architects are comfortable with. The natural reaction to this is to think that money is being wasted, and whilst that may be true compared to a straight line to the strategic architecture, strategic architectures often require significant time to realise. The straight line approach is less sensitive to the continuing evolution of the concept, disillusioning the stakeholder over time.

8) Design systems front to back.
The most important parts of an architecture for a set of collaborative development teams are the integration points and underlying data model. Not having these available as quickly as possible will often impede development teams from making any significant progress. Conversely, internal business logic can be refined over time and is not an immediate requirement if there is an understanding that initially integrations presented are only structurally correct.

9) Know when the waterfall approach makes more sense.
Working in a predominantly agile environment doesn’t exclude waterfall as a method to achieve certain parts of the architecture. This needs to be understood and this again becomes something that needs to be fought for due to the perception that waterfall is inherently slow. The advantage, however, of the waterfall approach is the ‘head space’ it gives architects to understand and devise a solution to a problem space. This usually applies best to non-stakeholder facing plumbing which provides the ‘spine’ of the architecture and should be built well before any stakeholder facing elements that depend on it.

I would be very interested to hear other real world experiences on this subject.

Leave a Reply

Your email address will not be published. Required fields are marked *