Improving the Odds of Large Program Success

The Idea: I try to feature one idea in each blog post, but this time I’m going to go for two: 

  • First idea: Internal auditors can improve the likelihood of large program success.
  • Second idea: Never miss an opportunity to capitalize on a failure.

The Execution: “IT systems development project failures are expensive, with an estimated cost to the world economy of US $6.2 trillion annually. This is not surprising given the dismal success rates — only 32 percent of major systems development projects are reported as completed successfully. It doesn’t have to be this way!” No, this quote was not ripped from the headlines — it’s from The IIA Research Foundation’s (IIARF’s) Systems Development Projects: How Internal Auditors Can Improve Success Rates.

Internal audit departments are increasingly getting involved in “large programs” (I define these as strategically vital programs designed to unleash new capabilities within a company, typically enabled by large-scale IT implementations). Because large programs are both strategically important and involve a significant outlay of resources (dollars, people, time/focus), they carry with them significant risks. Risks can occur anywhere in the process, from project inception (soundness of the business case) to project implementation (rigor of the governance process; excellent execution of project management techniques; attention to financial stewardship), to handover to the business (moving from incubation to steady state, at scale).

Internal auditors play a role in large program governance by providing their perspective to project steering committees and executive management on how project risks are being mitigated as the work unfolds — in flight — rather than solely through postmortems after the project has been implemented. Some departments call this a project health check, others do pre-implementation reviews, and many are partially involved through testing around system development life cycle controls for Sarbanes-Oxley. If your department is new to this, the IIARF’s book referred to above is a great place to start. It identifies the factors really critical to ultimate project success that internal auditors can assess.

So, once you’ve decided to engage in these projects, how do you convince the project team and its stakeholders that this is a role internal audit can play? This is where “never fail to capitalize on a failure” comes in. Now that the United States is being educated daily on the catastrophic risks of large program failure, you’ve got the perfect example to showcase how internal audit can help. I bet there are a lot of folks who would’ve liked to have heard an objective point of view on how the healthcare.gov website project was going before go-live!

Posted on Nov 4, 2013 by Carolyn Saint

Share This Article:    

  1. Carolyn, congratulations on your new blog and the interesting topic. With a 68% failure rate for system development projects one cannot but wonder if the internal audit opportunity should not regularly include a review of the “tone at the top” for indications (or perceptions?) that negative feedback, even when valid, is viewed by senior leadership as defeatism and thus considered unacceptable behavior for those tasked to execute a major project (or achieve a key objective). If so, a credible internal audit function (backed by a supportive Audit Committee) should be in an excellent position to get buy-in from the responsible project team and its stakeholders that IA can help them, if in no other way, by being someone who they can go to express their concerns if and when they believe they are unable to go to senior management when they have reason to believe the successful execution of the project is at significant risk of failure.
  1. Mike--thanks for reading and commenting!  I like your idea...whether conducted by Internal Audit, HR as part of an annual employee engagement survey, or through the ERM process, these concerns on corporate culture should be visible to and mitigated by senior management.  I think sometimes the project team pushes back on timing of the audit work (in-flight vs post-implementation), as well as internal audit's role in large program governance vs. that of other functions (e.g. PMO).  A well thought-out methodology for internal audit's involvement, embedding them in the governance process (perhaps reporting project status at specified funding gates) would go a long way toward gaining greater acceptance of the role internal auditors can play in ultimate project success.

  1. Hi Carolyn.  Great insights!  The IIA has also published the GTAG "Auditing IT Projects" which includes many ideas for helping auditors engage with large project teams.  Would be great to get more of your ideas as we update that GTAG. 

  1. Steve--you're right, the GTAG you mentioned is another great resource for understanding and auditing large program risks.  Definitely looking forward to the update and very happy to discuss when  you're ready!  Thanks for the comment and the valuable reference to the GTAG series.

  1. Carolyn - you've identified an area where I believe internal audit can really contribute to an organisation's objectives. I've been involved in major projects both as an internal auditor and in a project team implementing a financials package. The problem with large programs is that they are usually supported by 'senior management' thus any bad news (unmanaged risks) is buried. This is a quote from a recent UK failure, 'Pressure to deliver a programme of this magnitude within such an ambitious timescale created a fortress culture where only good news was reported and problems were denied.' (See http://www.parliament.uk/business/committees/committees-a-z/commons-select/public-accounts-committee/news/universal-credit-report/). The internal auditor has two objectives in order to improve the likelihood of large program success. The first is to ensure that the risks threatening the delivery of the project are being managed; the second is to ensure that the risks threatening the system when it is implemented will be managed. The first objective is achieved by ensuring that the project team assess the risks, own them and regularly update the risk register which is published, so there is no danger of buying the bad news. This is where the internal auditors' objectivity and independence will really be tested! The second objective is achieved by assessing the risks which will be present when the system 'goes live'. When I was a member of the project team, one useful strategy I found was to write the user manual while the project was in progress, not after implementation. This forced out the detailed problems, which are often the cause of major failures, and the manual could be used as the basis of testing and training.
  1. David, I wholeheartedly agree and your comments align with Mike's as well. Your example of the UK case where root cause analysis was used to highlight project reporting limitations is exactly the kind of story that helps management to see the benefits of an objective perspective.  Your idea of writing the user manual while the project was in progress is interesting.  Can you share more on that? Was this done as a part of User Acceptance Testing by the business process owners? Or would this be something the project manager owned?  Thanks for advancing the conversation!

  1. Carolyn, I'd be happy to provide more details about the writing of a user manual during implementation. Just give me a few days to fire up the relevant brain cells...
  1. Hi Carolyn. I'm afraid the answer to, 'Why write the user manual during the implementation of an IT system?' takes some space. However, first some background. Having been the CAE of a large UK company for several years, I was appointed as the head of the transaction processing departments (AP, AR, payroll etc) for a newly formed division. It needed new financial systems and I headed up the team that chose Oracle Financials. I then became a member of the implementation team, with the first date in my project diary (keep a diary) of 12 June 1990. From what I remember we had some basic principles: change the package as little as possible; forget the existing systems; think of the scenarios the package will have to process (for example foreign currency invoices, recurring allocation journals) - this is where most of the analysis work had to be done. The first system to be implemented was the general ledger, followed by purchase ordering and accounts payable. From a user perspective, a package is a series of input screens, consisting of fields which require the input of data. The users' challenge is to set up systems to get the data to the input screen, for all the scenarios likely to be encountered. This may be relatively easy for invoice input screens but not so obvious for the foreign currency exchange rate screen. (Who sets the rates? Who inputs them? How often are they changed? How are profits and losses accounted for?). In order to set up systems to deliver the data, it's worthwhile categorising the screens into set up data (used for setting up the system and rarely used afterwards e.g. the accounting timetable), standing data (new suppliers, new account codes) and transaction data (journals, invoices). Continued...
  1. Continued from previous... At this point you need to document the systems necessary to deliver the data to the screens). But you will have to do this for the user manual, so why not write the manual at this point? The advantage gained by doing this is that you will have to consider the detail, and it's the details which can derail implementation. What's the essential point to remember about manuals? People don't read them. They certainly don't read them when they should, because they're probably panicking and can't find the right page. A manual must be easy to use, so it must be a reference document, not a novel (think Wikipedia not 'War and Peace'). It must be targeted, so people only get the parts of the manual necessary to complete their tasks, not the whole 969 pages. (I'm assuming paper here because users can write their own notes all over the manual). The best way to target people is to write separate instructions for each screen and report, and only give them the instructions for the screens and processes for which they are responsible. The structure of the manual sections is similar. There is a section for each activity (from sorting the mail to generating cheques (checks)). Each section has a similar structure: A front page showing the individual tasks within the activity which then might have a reference to other documents such as examples (manual journals) and input screen details (showing what information to be keyed into each field). An example of one section of the manual, for the setting up of monthly journals transferring a fixed sum between cost centres, can be viewed at http://dmgriffiths.com/systemsimp/glmanualexample.pdf . Necessary controls are included in the list of tasks. Continued...
  1. Continued from previous...When each section is complete, it is scrutinised line-by-line in a meeting with the author by knowledgeable representatives from the users concerned. This ensures that the suggested procedures are thoroughly checked by staff who know the actual scenarios they face every day. It also involves them early in the implementation, hopefully avoiding late, nasty surprises. The manual can now be used for user acceptance testing and for training. The training we did, for example on invoice entry, was to photocopy typical invoices and give the users these with the relevant section of the manual. After going through one example we let the users get on with the rest, asking questions when necessary. We then made any amendments to the manual. Did this approach work? Yes, the general ledger was implemented with the Oracle consultant spending far less time than he anticipated.
  1. Continued from above. I forgot to mention how the above methodology might be used for completely new systems. In this case the decisions that the system needs to inform must be defined (What benefits do I pay in these circumstances?). (My site at www.managing-information.org.uk provides some ideas). Having decided on the output, the database and input screens can be designed. Then proceed as above.
  1. Wow David, you really gave us some good insights on the benefits of writing the user manual at an earlier point in a system implementation.  I'm going to share this with my team and my IT partners to get their thoughts on how we could use this on our major projects. Definitely will check out your web site too. Thanks again for taking the time to share your expertise and wisdom!

  1. Carolyn. Thanks for your comments. You have my email address from the reply form, feel free to contact me with any questions. I still have the notes from my systems implementation and I will give them a reprieve from the recycling bin!
  1. Been there many times, but always followed this 6 point framework. Internal Audit Objectives: 1. Project Plans are sufficient to complete the project as scheduled (time, deliverables, cost, and includes the right people involvement). 2. Training of personnel is complete, timely and adequate. 3. System programs will process data accurately, completely and in a timely manner and there is adequate separation of duties between input, processing and output. 5. Ample system testing was performed before user signoff for acceptance. 5. System documentation is adequate for user, help desk and operations needs. 6. Security features are in place and data files can be complewtely reconstructed if necessary.

Leave a Reply