How Do You Determine Whether the Risk Management Process Is "Effective"?

Tim Leech and I have been sharing our own perspectives on this question and would like your views.

Tim’s view is:
An “effective” risk management system is one that produces materially complete information on a timely basis on the organization’s residual risk status. This system needs to be re-evaluated each time a significant risk materialized that wasn’t foreseen. In cases where frequency and severity estimates were materially wrong, efforts need to be taken to see if the information system can be improved.
Norman’s thinking is that whether the risk management process is effective should be measured by looking at results, not just the processes:
An “effective” risk management system is one where there is an appropriate risk culture, decisions (at all levels) are based on an understanding and consideration of risks, and risks that are either above or below risk targets are managed towards that target. This implies continuous monitoring of risk levels and adjustment of responses, with appropriate communication throughout the enterprise. Management’s processes have to provide a reasonable level of assurance that risks are identified on a timely basis, fairly assessed, and appropriate actions taken. Obviously, a lot has to happen within the risk management processes/systems to support the above.
What do you think? We asked a group of experts and here are their answers:
  • The institutional process and good business practices of minimizing possible losses to the organization's operations through collaborative and supportive efforts of management, staff, and customers in their planning, execution, and monitoring of their roles and responsibilities for the short-term and long-term welfare of the business. (Prof. Frederick Gallegos)
  • An effective risk management system is embedded within formalized, mature governance and management processes. It is not a system to be externally applied. Organizational culture and formal processes in place promote understanding of risk, definition of appropriate risk appetite, and approval for decisions that exceed the risk appetite. Effective risk management systems are maintained by reporting that promotes a transparent view across the organization — of the formality within management processes, and the effectiveness of risk consideration and communication. (Dan Clayton)
  • Effective risk management consists of repeatedly electing a course of action from available options (including the option of doing nothing) consistent with an accurate understanding of stakeholders’ risk appetite and time horizon (which implies communication of expectations by stakeholders) carried out by competent (skilled and experienced) personnel with timely monitoring by those personnel and by the stakeholders or their representatives so that appropriate adjustments can be made as conditions change. (Charles Yates)
  • Effective risk management is maximization of the company's potential-to-pain ratio. (Cass Brewer)
  • Effective risk management is when each risk event identified is examined through the lens of both the direct loss to the firm and indirect losses that may arise because of damage to the firm’s reputation associated with the event. (Deon Binneman)
  • Risk management is about bringing a perspective to the management of complicated issues in complex organizations. It is about the management (and not the avoidance) of risk. It helps to prioritize your work and that of others in a fast-moving context with an approach that is better than simple intuition and which facilitates communication between people. It is a style of thought, and is definitely not a paper chase. (Richard Anderson)

We would love to get your views and perspectives.

Posted on Jan 4, 2010 by Norman Marks

Share This Article:    

  1. Norman:


    Happy New Year to you!

    Have been doing a fair amount of research recently on the state of ERM in the business world today.

    I absolutely agree with your elegant synopsis, the tenants of which I chose as a starting point for my consulting practice, Westport Business Solutions and why I have embraced the thought leadership of the GRC Model.

    The biggest disconnect at the moment is that while there is consensus among "experts" as to the merits and benefits of an integrated risk management framework, there appears to be little thought devoted as to how to convey this as a compelling concept to management. It's not so much whether a process is effective, it's whether or not it's formally adopted in the first place.

    You quoted Dan Clayton as saying:

    "An effective risk management system is embedded within formalized, mature governance and management processes. It is not a system to be externally applied."

    I absolutely agree. Therefore, as "consultants" or "experts", if we believe in the value and importance of this concept, we now need to focus on developing and articulating strategies and tools which allow members of management within an organization who do "get it" to "market it."

    Once implemented, then our role is to help them ensure that the adopted framework is effective and continues to remain current and relevant.

     

    Thanks again for a thoughtful article

  1. Norman:

    Here's my perspective.  Nothing in this world is static - business conditions change continuously.  Therefore, one of the main tenets of "effective" would have to include the ability of risk management to adapt to changes within the business conditions an organization faces. 

    While frameworks, tools and strategies will help risk managers mature the practice of RM within an organization, effectiveness does not necessarily need to be a formalized set of policies and procedures.  Rather, I would argue, a consistent understanding across the organization of what risk management is (and isn't), coupled with a culture of continually assessing the potential impacts of changing business conditions and the management responses needed as a result, makes a solid foundation on which to build an "effective" risk management process.

    Thanks for the interesting article!

  1. Tim's view: "This system needs to be re-evaluated each time a significant risk materialized that wasn’t foreseen. In cases where frequency and severity estimates were materially wrong, efforts need to be taken to see if the information system can be improved."

    Should include the probably much rarer inverse cases: re-evaluated... each time it was forseen and estimates were materially right. Will we ever see the results of such evaluations?

  1.  Very good question you pose.

    I tend to agree more with the view that the effectiveness of the process is related to quality of the outcome/deliverable?

    I would focus especially on the ongoing management of the controls/mitigations that have been identified (ensuring that these have in fact been implemented and they actually work as intended).

    Learning from near misses and the continued improvement of the process would also be features of an effective process.

  1. A broad understanding by key management of the enterprise's risk appetite is vital.  Each organization's pinch points are different and risk management must understand this and adjust as needed.  ERM and its attendant appetite for risk is in a state of flux, dependent upon the everchanging business environment and other considerations and issues.  We, as internal auditors, must help ensure these realities are recognized and dealt with appropriately. 

  1. To your original question of how to determine whether the organization's risk management process is effective, my answer would be that you would have to (be able to) independently confirm whether all important uncertainties to the organization achieving its objectives are managed down to what is deemed to be the appropriate level, in the most cost-effective way.

    The most important reason that this is typically easier said than done is that, oftentimes, there will be very limited visible (and therefore verifiable) information on what all of the different variables are in this short sentence. So what exactly are the most important objectives, what are the relevant uncertainties and what is the most cost-effective way. Anyone?

    To me then, the challenge lies in getting management to carefully articulate these variables, including the 'managed' part, and this will be the starting point for not only auditing, but more importantly, improving the state of risk management in an organization.

  1. Directorship magazine includes excellent materials from Goldman Sachs on risk oversight. See my comments at http://normanmarks.wordpress.com/2010/01/11/goldman-sachs%e2%80%99-10-principles-of-effective-risk-oversight/

  1. Criteria for an effective risk mgmt program:

    1) Objectives are clearly defined, measured and aligned for the enterprise and business units.  COSO: "Objective setting is a precondition to risk assessment."  Metrics would be "drivers" (e.g., number of subscribers at a magazine) and not "results" like profit.

    2) Key risks to realizing these objectives are identified. Risks can be internal or external, conditions or events.

    3) Risks identified are prioritized by mgmt in accordance with some a methodology that considers likelihood and magnitude, or leverage and volatility. Contrary scenarios and assumptions are considered.

    4) Responses are identified that management considers effective in managing the risks within tolerance.

    5) Parts 1-4 are reviewed with the Board (strategic & operational risks) or Audit Committee (financial & compliance risks).

    6) Effective monitoring of progress towards the objectives occurs. The extent to which risks are affecting progress are discussed.

    7) Incentives are aligned with these metrics and discounted based on the degree of risk taken by the firm, business unit or individual.

    8) Continuous improvement mechanisms are in-place.

  1. I think it's a combination of Tim's and Norman's defintions. Risk management processes can be evaluated by applying a standard control framework (adequacy of process control). How well the risk management process has functioned requires identifying instances where the process has worked.  Absent that, “effectiveness” only can be stated as negative assurance.  Example: management of the terrorist bomber risk. At a strategic level, Al Qaeda has not accomplished another 9/11. One might conclude the strategic risk management process has been effective. However, tactically, Al Qaeda has placed suicide bombers on international flights to the US; and succeeded in mounting serious attacks in Britain, Spain, etc. So there’s room for improving risk management at the tactical level. I guess this leads to the conclusion that adequacy of a risk management process can be assessed objectively; but effectiveness has to be defined by the risk owner before internal auditors can evaluate for them how well the risk management process works.

  1. Mark:
    I agree 100% with you that the criteria to determine when a process is "effective" should be agreed with senior management and, and I repeat the word "and", the board.

    Many board members when surveyed have stated that a priority outcome from an IA function is "no surprises".  I believe that an "effective" risk management system should strive to deliver no significant surprises where surprises are defined as risks that have the potential to materially impact the value of the entity. If senior management and the board are aware of areas of high retained risk and have decided to accept the retained risk it shouldn't come as a surprise if a negative identified risk outcome occurs.  If estimates of frequency/severity/consequence/liklihood used to base the decision are proven to have been materially wrong at the time the original risk acceptance decision was made, the risk management process should be reevaluated.

    I encourage IA departments to survey senior executives and the board to seek agreement on the criteria to define what constitutes  "effective" risk management.  Once the criteria, including desired performance metrics are agreed, an audit of an organization's risk management processes will be a much easier next step.

  1. Tim, this is a suggestion that needs to be handled with caution. If the intent is for the auditor to assess the risk management process, the auditor should be responsible for deciding the 'yardstick' with which to measure it. Certainly, the views of management should be solicited, but the assessment will be flawed if management determines the standard for 'effective'.

    However, if the intent is to provide consulting support in the establishment of a risk management process, executive management and the board should be more in the driving seat.

  1. Good question and some thoughtful responses but nobody has mentioned the international standard, ISO 31000: 2009 Risk Management – Principles and guidelines. This sets out 11 principles for effective risk management as described by an international committee representing diverse management activities. It also sets out an agreed risk management process and other guidance. Electronic copy of the standard can be bought from the Standards New Zealand website (look for AS/NZS ISO 31000: 2009).

    In 24 pages it so much more than the ERM documents from the USA.

  1. Chris, thank you for raising ISO 31000 . I agree this is a valuable document and recommend it to all who have not seen it. It is based on the ANZ 4350 standard and, IMHO, much better than COSO ERM.

    However, it is not easy to use ISO 31000 as a yardstick with which to measure the effectiveness of risk management. Consider that the ISO standard says that each organization should tailor its risk management program to suit its culture, etc. While is has some valuable principles, I am not convinced that is enough.

    For example, ISO 31000 does not insist that you need to have an adequate risk-culture - and without one, risk management will not be effective. Instead, ISO says that risk management has to be adapted to the organization's culture.

    I found ISO 31000 overly focused in its steps to periodic risk assessment and not focused enout on risk-intelligent decisions. The latter are included in the principles, but not in the implementation guidance.

    Having said all of which, with the possible exception of 4350, its the best out there I have seen - and will improve as the practical guidance is completed and published.

  1. We all believe that there are state owned enterprises. must they practice best corporate governance and if so, are the issues of remuneration, disclosure hence transparency, laws ,policies and regulations be in the interests of the public and the tax payer?

    What is corporate practice in a state owned enterprise and how must it be assessed?

  1. All these interesting comments do nothing to demonstrate empirically that risk management adds value.  There appear to be no real data or studies that show application of risk management actually reduces risks; e.g. either the known risks happen anyway despite mitigation steps, or the "unknown unknowns" vastly outweigh the known risks anyway, or that the total expected loss is accurate enough to determine how much resources to put into their mitigation. 

    With all the time and energy put into this topic, shouldn't someone be able to prove something about risk management effectiveness in a peer reviewed journal?

  1. Charles, you make an interesting point. I have only seen studies that show that having a risk management process adds value by increasing share price, enhancing results, and improving resiliance. I have not seen any studies that show that risks are actually reduced.

    All I can say is that everybody who has a solid risk management process in place believes in it. Their experience is that it works by reducing the likelihood of an adverse event (e.g., at SAP we have withdrawn from $100m of bad deals in the last year) or reducing the potential impact of that adverse event.

    The only way to determine whether the risk assessment is accurate is to decide not to take any action and see what will happen - and people don't do that unless the risk is within their tolerance levels.

  1. I understand the issue of fuzzing the data by actually working the risks.  However, I think these 3 things are measurable, and because they have not been measured, it makes the process suspect:

    Of the known risks, how many were actually realized?  In other words, do we really successfully manage many of them?

    Of the realized risks, how severe were they versus expected?  If we thought it might be a 4 month problem, was it really 1 month, or 12 months?

    Of the other problems that crop up, how big were those versus the known risks?  In other words, are the known risks minimal compared to what really happens?

    This data is measurable.  Does anyone have any non-anecdotal data to demonstrate risk management effectiveness?

    Chuck Weis

  1. Charles,

    You raise interesting points. In many ways we are facing the challenge of building a bridge to Governance transparency needs, from organizational operations. We have lots and lots of great building materials, and many experts yet we have not found the lasting bridge design.  We have engineers (Management) and analysts (Auditors). Engineers who are experts at building things and Analysts who can critically value and report.

    In my mind we the Analysts are still emerging from a traditional skill set focused on valuing completed bridges against external standards. We are unfamiliar with the process of building bridges. We are just now coming to a broad recognition that we have to better understand engineering processes. (After all risk management is about managing and acheiving business objectives). The value will come in helping the engineers take the next best step. This thrusts the analyst into an unknown world of defining best practices and standards around strategic development, objective implementation, people, process and technology maturity-all the while maintaining independence.

    In my mind, risk management will never be fully effective until we have incorporate standards for measuring good management as part of the process. This is where the capability maturity model has such promise. If we can capture the maturity of people, processes and technology along a standard spectrum we have created a way to communicated vulnerability to management. Risk Management is one part vulnerability awareness, and one part risk identification...

     

  1.  Getting the definitions right, and easy to understand at all levels within a business could be the place to start. If everyone has a reasonable definition within the organisation, in their own language, then maybe <a href='http://www.riskmanagement.org.nz/'>Risk Management</a> is effective within that business. In New Zealand we have a lot of work to do, especially as it applies to health and safety, as instead of talking about Risk Management the terminology is Hazard Management!

     

     

Leave a Reply