Lessons Learned about Lessons Learned (Part 2)

In my last post I discussed a few of the problems that I've encountered with Lessons Learned reporting (a.k.a Project Post Mortem reviews). In this post I'm going to propose some of things that could be done to add value to a process that is often judged with contempt by project managers and team members, misused or misinterpreted by process improvement teams and managers, and generally misunderstood by the people who could most benefit from it. These ideas are intended to be method independent and can be adopted whether you're using CMMI, Agile, or any other model or lifecycle.

Before I go any further however, I want to pick up on some of the comments made on my previous post, namely the issues of Timeliness (@flowchainsensei) and Trust (@bruhland2000), because these have a significant bearing on any proposal for improvement of this process.

In many organisations the Lessons Learned process only kicks in at the end of the project. There are several issues with this:

• the project cannot benefit because it is already finished
• project team members cannot properly recall details of things that could be of value in the future, and anyway they are already busy thinking about their next project
• funding is no longer available to do the job properly

The other problem about timeliness is that project lessons learned reviews rarely coincide with process update cycles. Valid improvement opportunities (or fixes to incorrect processes) may not be made available to projects for months. Projects who need the fixes and therefore use them before they are "official" face the prospect of being "non-compliant" unless process waivers can be used (another process that is often so bureaucratic and time-consuming that projects would prefer to risk a non-compliancy than jump through hoops to get the appropriate waiver!).

The second issue regarding trust (or lack of it) is common in organisations where there is still a dominant blame culture. Whilst good facilitation of reviews can minimise some of the issues around trust, a more stable and long term way to deal with the issue is to be more explicit about what constitutes a valuable lessons learned review in the first place and focus on the processes, system interactions and organisational quagmires that need to be changed to give projects a better chance of succeeding.

So what are we really trying to achieve in a lessons learned review? Who are the beneficiaries? How can we make the process more robust and of value to the participants? And most importantly, how can we effectively use the information gathered in these (and other reviews) to ensure that the process returns more benefit than it costs, and therefore adds value to the organisation? I'm not going to try and define a one-size fits all process here. What I'd prefer to do is offer up some ideas for consideration that you may be able to build into an existing process set, or to discuss with senior management to bring around a mindset change if that is what is required.

Some of the issues that an organisation needs to addressed (and you need to be honest  when you answer these questions) are:

• What is the real purpose of the process ? Is it intended to generate genuine improvement or is it in place to comply to some abstract model requirement? If you are not getting any tangible benefit from your lessons learned then you are doing this for compliance purposes and the process needs a major overhaul
• Does the lessons learned review do anything to address systemic process or organisational issues in a timely fashion or does it simply generate outputs which are subsequently ignored because there is no real substance to them
• Who is empowered to make process changes based on the lessons learned review? Can project teams make local changes to meet their specific issues? If they must wait for official changes to be released there is little incentive in them taking the time to participate in any such review - and the organisation is building a failure mechanism into its operations
• Are successes identified by projects deemed to be "best practice" and built into the process set regardless of the specific set of circumstances which may have led to a specific success?
• Do you have a repository of lessons learned reports and assets which consists primarily of similar findings and alternative templates and is rarely accessed by any project teams? Any genuine lesson learned should take the form of an actionable item leading to a direct improvement in the process asset library or change to company policy
• Does your lessons learned report template contain sections like "what worked well", "what went wrong" and "what could be improved"? Focus the review away from these subjective descriptions and follow through on specifics like "which processes / interactions are causing us trouble and what can we do to resolve the problem?"
• Have you considered adding some quantitative elements into to the report? This can be difficult as scoring issues is still very subjective, but if the same people contribute by scoring the same questions at regular intervals in a project you can at least begin to visualise how your improvement efforts are working. However, this shouldn't be used as for cross project comparisons. (I'll post an example of this type of template on my website in the downloads section in the next few days. This template has proven useful in the past and is still being used today)
• Do you have any measures in place to assess the value of the process? If you assess the value using compliance audits or counts of reports submitted than you haven't. If you have a count of actionable items generated from lessons learned reviews you are at least on the way
• Do process owners (or their equivalent) ever meet with project teams or do they rely on the SEPG representatives? Project managers with process issues should invite process owners to their lessons learned reviews (as non-project stakeholders) to hear the issues first hand (and not to defend their process)
• Do you make a distinction between internal project reviews (status meetings) and lessons learned reviews? If projects have regular daily, weekly or monthly review meetings which address the purpose of the lessons learned process why should they be compelled to perform another process review just to tick the box. Don't penalise good practice for the sake of compliance or terminology!

As I wrote in the previous post many lessons learned processes are in place to fulfil a misinterpretation of a requirement, whether it be CMMI or ISO or whatever. The process is often designed to create evidence to demonstrate that the process is been executed, but it fails to address the real objective which is to provide feedback into the management system to try to stop bad history repeating itself. Far more effort is put into creating the outputs and monitoring compliance to the process than in addressing the real needs of the organisation - which is to improve. If the current effort wasted on Lessons Learned processes can be re-channeled into addressing the issues uncovered by the process then the people who are required to use it may start to respect it more because they will actually begin to reap some benefit from it.

As will the whole organisation.

Lessons Learned about Lessons Learned (Part 1)

My last post about "Best Practice" was something of a personal peeve rather than a major hurdle to successful business improvement, although lessons should be learnt from the story I told, namely:

• your improvement focus shouldn't be about creating templates and completing documents to provide evidence for appraisals and audit
• measures that you put in place need to be thought through from the perspective of those being measured if you want to avoid costly and wasteful dysfunctional behaviour
• best implies that there is no room for improvement

Ultimately, I also suggested a relatively quick fix; simply stop using the term "best practice" and come up with something more appropriate.

"Lessons Learned" (learnt?) is a different kettle of fish however. Most organisations have some kind of lessons learned mechanism in place (very often associated with their "Best Practice Repository"!) and in my experience very few of these mechanisms add any real value to the organisation. They are put  in place to meet the "requirements" of CMMI, etc. and as is so often the case, fail to even to do this adequately because people fail to understand what the model is really trying to help the organisation achieve.

There are a whole bunch of issues associated with the Lessons Learned process, many of which would appear to be associated with the origins of the idea. When I first started working in Software Development over 25 years ago we never did Lessons Learned - but then again, we didn't do much of the stuff commonly considered as integral to the development process today. My first memory of anything resembling the concept of a lessons learned review was when I read an article by Tom deMarco on Project Postmortem Reviews * some time towards the end of the 1990s. This made a lot of sense back in those days - at least in the organisations I worked in where we had a small teams of developers who stayed together and worked on project after project. It made sense in projects that lasted for years, where phase postmortems could be used to identify and correct mistakes prior to starting the next phase. And in the days before I started getting involved in SW-CMM initiatives I introduced the concept and practice into several groups with some small success.

On a small scale the Project Postmortem Process defined in that paper can be very useful as long as two conditions remain in place:

• The same teams of people are used in a "product development environment" so that they have a shared understanding of the issues which cause problems
• There is a management environment that allows the problems to resolved internally within the teams rather than imposing  inappropriate solutions on the team without really understanding the underlying issues.

The Project Postmortem Process fails as soon as:

• It is scaled up to become an organisational process - e.g. the focus changes from an internal review and change exercise to a management mechanism for fixing organisational issues
• Project teams are constantly rearranged and staff treated as interchangeable resources
• Emphasis changes from internal and potentially informal communication within a team, to a demand to codify the findings for wider use
• Managers and process experts attempt to implement CMMI without understanding what the model really implies - i.e. that the process is part of the feedback mechanism to improve the business, not an exercise to generate paperwork to fill up a repository
• The process is used to apportion blame

Over the years since first reading about the Project Postmortem Process I have been seen numerous lessons learned systems in numerous organisations. Most of them suffer from exacly the same problems.

• Lessons Learned reports are documented and filed away and the feedback element to improve process rarely occurs
• Lessons Learned are purely qualitative and can easily be distorted by the strongest or most vocal members of a team
• Not all project team members or stakeholders are invited to participate in the process so not all perspectives are represented
• Most staff perceive the process as a waste of time (and in most cases they are right)
• Reviews are usually focused on project failures and successes and the process improvement opportunities are not realised
• Managers or SEPGs that do any analysis on the results attempt to implement heavy handed changes based on their interpretation of the results without understanding the underlying circumstances or, most significantly, the differences between the environmental characteristics of the originating team (project or programme, agile or waterfall, etc.)

In fact many Lessons Learned reviews and reports have become so worthless that I can often predict the findings without knowing anything about the project or people. Typically the section on Things That Worked Well include:

• Excellent teamwork and communication between project team members
• Everyone went the extra mile and worked hard to complete the project under extremely difficult circumstances
• The pizza brought in by senior management on the nights we had to work was really tasty and much appreciated

And in the Things That Could Have Been Better you'll see:

• Really difficult relationship with the key stakeholder made this project a major challenge
• Customer was never available
• Requirements were so poorly specified we had to make them up as we went along
• Technical procurement issues led to long delays
• The CMMI initiative caused us a huge amount of extra work which didn't add value to the project

So very often, project lessons learned reviews tell us what we already knew, and probably have known about for some time. And the only thing that future projects can learn is that they are doomed to follow a similar pattern because nothing is being done to correct the organisational issues. The most likely outcome is that new status report and requirements specification templates will be imposed on the projects.

In part 2 I'll have a go at look at some of the things we might be able to do to make Lessons Learned a more valuable tool for the organisation, by extracting value for both projects and processes.

* Collier, DeMarco and Fearey: "A Defined Process for Project Postmortem Review" IEEE SOFTWARE 0740-7459/96/$05.00 © 1996 IEEE Vol. 13, No. 4: JULY 1996, pp. 65-72