How to Assess the Effectiveness of Internal Audit

Norman Marks, CRMA, CPA, is a vice president for SAP and has been a chief audit executive and chief risk officer at major global corporations for more than 20 years.

 

There are some interesting discussions on LinkedIn (including this one and this one) and elsewhere about the value of internal audit and even calculating a return on investment in internal audit.

As you might expect from me, I don’t like the traditional measures or KPI that many use. I just don’t see them as indicators of effectiveness.

I believe that in order to establish how we measure the effectiveness of internal audit, you have to start with agreement among the head of the function (CAE) and his stakeholders (primarily the audit committee) on the role and the objectives of the activity.

As explained in the IIA’s definition of internal auditing, the role — and therefore the objective — of the activity should be as “an independent, objective assurance and consulting activity designed to add value and improve an organization’s operations. It helps an organization accomplish its objectives by bringing a systematic, disciplined approach to evaluate and improve the effectiveness of risk management, control and governance processes.”

Internal auditing is effective if it provides the audit committee and executive management with the assurance they need, namely that they can rely on the organization’s processes and systems to manage risks to the achievement of the organization’s objectives. That means providing assurance on the risks that matter to the organization today, in a form and timeframe that is useful.

Additional value is provided through the role of internal audit as a change agent, making recommendations for improvement that are embraced and acted on by management.

How do you put a value on assurance? You don't worry about the quality of the water you drink (at least where I live) because you know that the company providing the water has to comply with strict regulations, and the water is tested frequently to ensure it is safe and to standards.

How much would you then pay, as a board member or top executive, for assurance that the processes and systems that you rely on to run the business are working properly? Assurance that is so reliable that you don't even think about it?

It's hard to put a value on "peace of mind," but in my mind (pun intended) that is the greatest value an effective internal audit function can provide.

I believe that the only way to determine whether internal audit is effective is to ask the stakeholders whether they are comfortable that they are receiving the assurance they need, when they need it, and in a useful form on the risks that matter to them and to the organization. Only then do you start looking at additional value that is provided.

For a moment, let’s examine some traditional measures and discuss their value and relevance. The table below is for a hypothetical organization. At first glance, this looks like an effective internal audit department.

 
Metric
Achievement
Percentage of audit plan completed
98%
Number of audit findings
Up 10%
Recommendations accepted and implemented
90%
Auditee survey results (average from 0 to 5)
4.3
Cost savings (duplicate payments, vendor overcharges)
$3,000,000
Internal audit budget
2% below budget
IIA Quality Assurance Review
Generally complies
 

This department completed 98% of the engagements in its audit plan. But, if that was (as most are) an annual audit plan then this may be an indication that they continued to remain glued to their plan even when risks changed. They failed to audit what matters now; instead they blindly continued to audit what used to matter. When you have a flexible audit planning process that adjusts to changes in the organization’s risk profile, percentage completion is meaningless.

An increase in audit ‘findings’ does not indicate productivity. If the audit department has been around for a while, this is an indication that they haven’t been getting their message across, addressing the root causes of issues and effecting lasting change. An effective internal audit department will, over time, contribute to the improved maturity of governance, risk management, and internal control systems — such that, in time, exceptions and so-called "findings" will diminish.

When 90% of recommendations are accepted and implemented, 10% are not. A 10% defect rate is abysmal. Was internal audit getting the recommendations wrong? Were they not accepted because they didn’t make good business sense? Or was internal audit not able to persuade management to effect the change? When you have a defect rate of 10%, the quality of the audits and reports are called into question. Frankly, the acceptance rate should be above 99%.

Cost savings of $3,000,000 are excellent, but only if they are not delivered by diverting resources from essential assurance activities to efforts to demonstrate that internal audit "adds value." Too many organizations have focused on the latter but failed to address critical risk areas such as ineffective risk management, poor information to support decision-making, and governance issues.

Staying within budget is, at least on the surface, very good. But, internal audit should be prepared to go to the audit committee for additional funds if new or changed risks emerge. Budget limitations are not a valid excuse for failing to engage and address unanticipated high risk areas.

Passing the IIA’s quality assurance review (QAR) is all well and good, but it is not a guarantee that the department has delivered the necessary assurance and consulting services. Many departments have passed the QAR but failed to audit risk management, or to report the lack of risk management to the audit committee.

Where does this all leave us?

Going back to the objectives of providing assurance that matters on what matters, the CAE should propose measures and metrics that support an assessment by the audit committee and top management that internal audit has been effective.

I would ask these questions of my stakeholders at least annually:
  • Do you believe internal audit has provided you with the assurance you need, in a useful way, when you need it, on what matters?
  • Do you have the assurance you need that management has effective and efficient processes and systems to manage the more significant risks to the success of the organization and the achievement of its goals and strategies?
  • Has internal audit been sufficiently responsive to changes in risk, ensuring it remains relevant and on point?
  • Has internal audit been an effective agent for change, improving business efficiency and effectiveness?
  • Are you satisfied that the cost of internal audit is less than the value of the assurance and consulting services it provides?
  • Are there activities that internal audit should stop performing? Have there been activities you would have preferred not to pay for?
  • How can internal audit improve its services to the audit committee, management, and the organization as a whole?

I welcome your comments, stories, and opinions.

 

Posted on Jan 30, 2013 by Norman Marks

Share This Article:    

  1.  Very insightful report

     

  1. Thanks the last few pointers are really helpful.

  1. Very good insight Norman.  I struggle withe traditional measurements too.  Also, I really like the questions you ask annually.  However, the main issue is how do you measure these things?  i think it is important to have qualitative and quantitative factors.  Any suggestions on how to measure? 

  1. Norman;  Great topic.  Thanks for raising it.  Coincidentally I am working in the UK this week and readers should be aware that the Sept 2012 UK Governance Code calls on Audit Committees to take specific steps to evaluate the effectiveness of the internal and external audit function.  The new Code takes effect in 2013.

    Unfortunately, in the case of assessing internal audit effectiveness, I think many QA reviews, at least in the past, have assessed IA functions against the type of traditional evaluation metrics you list above - planning, executing and reporting of traditional direct report audits.  

    I believe internal audit departments should be primarily evaluated on whether they are helping the board meet their risk oversight expectations.  The best summary of what a good board should do in the area of risk oversight I have seen is from the NACD Blue Ribbon Commission report "Risk Governance: Balancing Risk and Rewards".   The six most important risk oversight responsibilities are described in my new Conference Board Director Notes article on Board Oversight of Management's Risk Appetite and Tolerance. It can be found at:

    http://blogs.law.harvard.edu/corpgov/2012/12/17/board-oversight-of-managements-risk-appetite-and-tolerance/

     

     

     

  1. Thanks for the comment and question, Robert.

    While we all like the quantitive, that is not always going to work well in practice. Whether the audit committee feel that you are providing clear reports that get to the point and communicate the assurance they need is always going to be subjective. It is also subjective whether they feel you have tackled the right risk areas.

    All I can suggest is that you understand what constitutes success and effective performance from the perspective of your stakeholders, and then see if you can agree with them at the start of each period how it will be measured at the end of the period.

    In practice, I use some metrics (such as the time to release a report after fieldwork is completed) as an additional internal measure. But I also realise the limitations of such measures (such as the issues are difficult and it is better to get the report right and the right corrective actions sold, than to get it out there quickly), and that what really matters is the perspective and opinion of the stakeholders.

  1. I DO as those questions you present at the end of your article, and often, because the answers are important to me.  Over time the answers have become more positive (a good think, I hope), but they weren't always, and using this qualitative assessment has helped drive the IA function to be more effective.

    Hoever, I don't struggle with traditional measures and actually use some of them because, as you point out, The IIA requires us to provide a "...systematic, disciplined approach..." and I need certain metrics to ensure we are doing so.  Also, our stakeholders expect  us to complete evaluations we've agreed to perform on their behalf, even when those agreements change quarterly, or at a more appropriate frequency (which ours do at times).

    So, in my opinion, a good method of evaluating Internal Audit's effectiveness utilizes relevant measures that incorporate both qualitative and quantitative assessments.

  1.  Well said, Rob

  1. Norman - I am trying to understand how other Internal Audit functions provide assurance on the "EFFICIENCY of processes."  Can you share how you have provided this assurance? 

    "Do you have the assurance you need that management has effective and EFFICIENT processes and systems to manage the more significant risks to the success of the organization and the achievement of its goals and strategies?"

    "Has internal audit been an effective agent for change, improving business EFFICIENCY and effectiveness?"

  1. Ad hoc audit requests (normally urgent or compelling) by the Audit Committee or Management, should also be considered as in a number of engagement, such requests could form a sizable part of the internal audit services. 

    How satisfied is the customer, i.e. Audit Committee, with the audit department being effective or not is what matters most. The seven (7) questions are very powerful to determine the Audit Committee members' perception of the department's effectiveness.  The metrics, in my opinion, could be applied but not relied upon so much for measuring effectiveness. Perception of audit performance, its effectiveness and impact to the organization of audit recommendation and implementation, in my opinion, lean more on qualitative assessment. 

    Audit Department being a support and staff function (not operational) could only, but ,"influence" those  performing functions for compliance, esp. on those identified as having high risks. 

    We could be more complacent on recommendations would address low or negligle risks, but not those that have high probability of occuring and huge impact to the bottom line. Of course, recommendations, esp. those involving additional outlay of huge some of resources would take bit longer to implement.

    Cheers!

     

     

  1. I think you have to look at the impacts of internal audit's recommendations to see if it adds value to the organization.  The recommendations should improve managing the significant risks in the process. For example, if internal auditors recommend documentation updates to reflect what the organization is currently doing, then this adds less value than updating it to manage emerging risks.

  1.  Dear Anonymous,

    We have been performing operational audits focusing on the efficiency of processes for a very long time. Many of the companies where I led IA had very thin margins and so efficiency was a major risk area. While this is sometimes different from looking at effectiveness, internal auditors should learn how to assess and comment on efficiency as well.

     

    Norman

  1.  A very insightful article.Most Audit Commitees do not measure what counts.The position of the entity on the governance maturity model is also very critical.Internal Auditors may try hard to highlight risk and control issues to audit clients and fail to archive the 99% + acceptance rate for observations and recommendations.This reflects on the position of the entity on the governance maturity model as it does on the IA function.I have dealt with an audit client (Senior Executive) who stated in the opening meeting that the audit had to at most two hours.It would be very difficult to obtain the buy in and acceptance of  clearly articulated audit findings from such a client.My contribution in summary is that acceptance rate is an inter play of several factors key of which are  the Management of the entity  and the IA its self.

  1. Congratulations. Excellent articulation of evaluating internal audit effectiveness. There will be challenges on getting time and attention from Chairperson and members of The Audit Committee . What is most important is the dialogue and engagement with major stakeholder . Evaluation parameters will get finetuned as we move forward.

  1. The simple measures almost always boil down to one of three: less costly; better; and/or faster.  So let's talk from the top down.  Management and board hold onto resources such as internal audit when that resource regularly provides information that is needed but but not known and not readily available through other means.  If audit simply repeats what is already regularly known they really do not provide value and therefore will be considered as a waste of resources in a down cycle.  Auditors that understand this measure go the extra distance to be sure they understand the level of information that a process owner already has.  When the auditor detects trends and issues that are to the contrary or have not been identified and can professionally communicate timely the auditor has made a difference and will be more likely to be invited back.  You use the duplicate payment analogy.  Most likely management is already aware that duplicate payments are slipping out the door from sources of information such as from the payee's that return such payments without depositing the checks.  What management may not be knowledgeable about is the frequency and the severity of such payments when made are cashed in by the payee which presents an opportunity for a cheaper, faster, and better internal audit function to add value with conclusive and defined answers.  PS: auditors can and will easily defeat the "metrics" you defined in the matrix!

  1. Dear Sirs,

    This topic are really helpful to me.

  1. I found your article to be very interesting.  Even if the metrics stated are good measures, you have really raised some issues that every CAE and senior management should be considering.  I am yet to see though an organisation where the heads of departments are enthused about the Internal Audit function.  They do admit at times that value is being added but ask another time and auditors are viewed as a pain.  You can let me know if your experience (at least lately) has been one of total acceptance and appreciation.

  1. You've hit on another great topic, Norman, which has lead to a great discussion.  Here are a few things that I've used to address the points you raise. 

    We do have an annual audit plan, and we report to the Audit Committee how we are progressing, but the plan is flexible.  If priorities change during the year, we will defer or cancel a planned audit and focus on new, higher risk areas.  However, we still report that as a failure to meet the plan.  It's explainable, but why not acknowledge and track the effectiveness of the audit plan itself?  It certainly takes enough time to create, and the AC becomes more aware of risk and IA's ability to address it timely.  This is not, as you sugegst, a meaningless metric.

    Also, the IIA's quality program includes similar questions to yours in their interviews with senior management.  And since an effective QAIP will include anual self-assessments, it seems like these questions would already be being asked at least annually.  (Our process is to ask about 25% of the senior managers each quarter to get regular updates as to our value.)

    We also have some of the standard metrics we include in a balanced scorecard, theoretically because it's how senior management tracks all departments -- but I secretly believe it's because people in general like to see colors rather than text <g>.

  1. Very good article.  I fully agree that measures such as completion of the audit plan and number of findings are not good measures of IA's effectiveness.  No one has ever critized my department for conducting one less audit.  You want to have broad coverage, but quality is more important then quantity.  Being graded on the number of findings encourages more findings.  Trivial and low risk findings are of very little use to the organization.  Also, they tend to distract from the more critical findings.

    One of the best measures for an IA department is the number of consulting requests received.  I have use this metric for many years.  This shoud increase over time and an increasing number of requests shows that other parts of the organization view IA as a true partner.

  1. Congratulations Norman! Not on the article so much but rather on the response you received and interest you generated.  A major achievement if not the first time I‘ve seen up to fifteen comments. 

    No one can disagree; its’ all good.  
     
    If I may add….an area where IA can contribute to significant savings is in the advisory role. If in retrospect, IA Advisory Services were asked for a quote for all the times the company had contracted out special improvement-and-efficiency studies to outside bureaus or one-time service providers, overhead cost would likely fall should they win a few,  Getting resources and the Audit Committee onside to balance the trade-off may be a problem besides others like management acceptance, first-choice, etc.

     

  1.  to measure performance and effectiveness of internal audit department is bit challenging with traditional kpis. too many kpis also loose the focus on subject matter. this problem will multiply when we have weak audit committee as well as demanding. this lead in interference of management role. despite these difficulties if we measure performance from the point of view that help provided to audit committee in risk oversight role is very vague and i believe parameters should be designed to measure performance. if we dont have critical successs factors and kpis agreed with ac and board then like always we will hear from board about audit role and reason for existence when audit is unable to identify breached control.

  1. I agree with your effectiveness indicators.  The traditional measurements cited in your article are actually efficiency indicators.

  1. Excellent Info! I really thankful to discussing here on the topic "How the internal audit process is managed is a key factor to ensuring the effectiveness of a quality management system." http://www.zipquote.com/
  1. Insightful article. I will be undertaking a research project for my MSc Finance and would like to develop the idea further by looking at how Internal Audit Functions across a wide spectrum of industries are managing the following additional aspects within the broader domain of performance measurement;

    Annual appraisals- content,form and effectiveness

    KPI deployment within the Audit functions -Implementation,review and effectiveness

    Use of other tools like Balanced Score Card in appraisal of Internal Audit staff

    How the Audit Commitee assesses and evaluates the performance of the Internal Audit Function among other issues. I have a keen interest in the subject and hope to share with peers the results of my findings

     

  1. Agree in general, but have a question. Does it create a conflict of interest for IA to ask Audit Committee as part of IA assessment on the effectiveness of processes maintained by management (question N2 from your list)? Thank you for your response
  1.  Ilya, it is not a conflict but I would generally ask them for their opinion after I have management's assessment.

  1. Very insightful article. However, I believe there is need for both quantitative and qualitative assessment of the Internal Audit function

Leave a Reply