The Problem With Risk Heat Maps and Dashboards

Norman Marks, CRMA, CPA, is a vice president for SAP and has been a chief audit executive and chief risk officer at major global corporations for more than 20 years.

 

Most risk practitioners seem to use some form of visualization, such as a heat map or dashboard, to communicate risk levels. But I have my doubts as to whether these reports are as valuable as they seem.

 

 

 

 

 

 

 

 

 

 

 

In the heat map above, two risks are shown. One is asssessed as having a high probability/likelihood of a high impact, the other medium probability of a high impact. The question is whether that is useful information by itself.

To illustrate the issue, let's take a speedometer that shows that the vehicle is moving at 100 mph.

What does this tell you? Does it tell you whether the vehicle is moving at the right speed, the speed you want? Not really.

To be able to assess whether 100 mph is good or bad, and whether you need to change the speed, you need more information — more context. For example:

  • What kind of vehicle is this? Is it a car, an airplane, or a boat?
  • If it's an airplane, why is it going so slow? Is it in the air or on the ground?
  • Let's assume it's a car. What are the road conditions?
  • What are the traffic conditions?
  • Where is this? On the freeway or in your driveway?
  • Is the driver experienced and able to drive safely at this speed?
  • If the car is on the autobahn in Germany, and there is no speed limit, is this fast enough?
  • When do you need to reach your destination? Is there any benefit for arriving early or penalty for arriving late? What will you do if you arrive early — will that actually cost more (e.g., for parking), or will you be able to use the time to prepare for a meeting?
  • Is the vehicle safe to drive at this speed? Does it have enough gas/petrol, and has it been maintained such that the brakes and everything else will work properly?
  • What is the speed limit, and is there a police car right behind you?

Knowing the speed is not enough to know whether action is required.

In the same way, knowing the risk level (likelihood and impact) is not enough. It needs to be reported in the context of risk appetite/criteria (I prefer using risk criteria as discussed in ISO 31000:2009). In other words, is 100 mph an acceptable level? Or is it either too high or too low?

Rather than using risk heat maps or similar, I think it is better to find a way to report whether the risk is within acceptable levels — satisfying your risk criteria.

What do you think?

Posted on Jun 20, 2012 by Norman Marks

Share This Article:    

  Comments (34) un-categorized
  1. Comment by: Lawrence Ellefson   (6/20/12 06:14 PM)   http://www.orcasaudit.com

    Absolutely, positively 100% correct. Every risk needs to be considered in context and within defined acceptable tolerable criteria. Been preaching that since the 80's.

  1. Comment by: Jose Duran   (6/20/12 10:23 PM)   http://da.com.ve

    I think the first step is answered all questions about the context. Next, you build the heat map (If all criteria is totally defined).  

  1. Comment by: Rajarshi Ghosh   (6/21/12 05:13 AM)   

    The process of risk rating done is how we rate them and it sometimes is to the perception of the individual. As sometimes we take the severity level very high which obviously we should but after applying the control measure thats when we need to decide to the appropraiteness and suitability of those. Even after that , it still remains very much qualitative subject to interpretation but at the same time risks do remain which has to be tolerated and at the same time acceptable.

     

    Rajarshi

  1. Comment by: Paul Nyirenda   (6/21/12 09:23 AM)   

    I agree, the context is so critical. This also applies to reviewing of risks, we need also to review the context as it is likely to change

  1. Comment by: Anu Alladi   (6/21/12 10:27 AM)   

    If the tolerance levels are already inbuilt into the definitions for the impact levels(high, medium and low), wouldn't the heat map appear complete then? 

  1. Comment by: karl green   (6/21/12 10:40 AM)   

    If the readers of the dashboard have a common understanding of the parameters that determine the ranking of a risk, and the parameters used reflect the risk tolerance of the organization, then a dashboard such as your illustration provides a useful snapshot.

    My experience is that the 2 conditions I mentioned rarely exist, and it takes tremendous effort to get to a common understanding. If the providers of risk metrics aren't careful, the rankings provided merely reinforce previously held opinions without generating the types of discussions useful to either management (in general) or those in risk / compliance positions.

  1. Comment by: Norman Marks   (6/21/12 10:53 AM)   

    Anu and Karl,

    I agree that the report should indicate not just the level of risk but whether it is acceptable. Can you share how you do that?

    Thanks for your interest

    Norman

  1. Comment by: Brian Spires   (6/21/12 11:24 AM)   

    I agree that all of the factors you describe on acceptable levels of risk should be considered.  But that is not a problem with the use of risk heat maps.  That is a problem with deciding what risks go on the risk heat map and what supplemental information you supply with the heat map.  If risks 1 and 2 in your example are that risky to an organization they deserve some level of attention.  But if the tolerance for risk 1 is "high-high" then perhaps it is a simple monitoring report.  If the tolerance on risk 2 is "medium-low" I'd expect a lot of detail on what the organization is doing to improve their mitigation strategy related to that risk.  And if risk 3 didnt make the map because the residual risk is "medium-medium", but the tolerance is "low-low",  then the process of putting together the risk map has not matured yet.  You can muddy up the heat map by trying to balloon the acceptable range of a risk and planting the residual inside or outside the balloon or you can handle in the supplemental materials.  So, while I agree with your concepts, let's not disparage heat maps - if done correctly they are effective tools at starting a discussion around risks.  And after all, isn't that what we are trying to accomplish?

  1. Comment by: Robert Berry   (6/21/12 12:04 PM)   http://www.thatauditguy.com

    I have to agree and disagree.  I agree that the heat map is only part of the complex process.  I also agree that everything must be put into the appropriate context.  But I do not think we should abolish heat maps.  I think they are part of the puzzle.  We need to invent a better reporting process that includes either the heat map or some more relevant variation.  I am glad you asked Anu and Karl if they could share how to do that.  This is something I've struggled with for quite some time as well.  I thank you for another great post.

    Robert 

  1. Comment by: Norman Marks   (6/21/12 12:11 PM)   

    Brian, I am a little troubled by the idea that somebody decides what should be on the heat map. Do you mean that only risks outside acceptable levels are reported?

    I like the idea that all risks other than Lo-Lo are reported, but together with related acceptable levels - and that includes where the risk level is lower than desired.

    But, how do you do that?

    Norman 

  1. Comment by: Jacquetta Goy   (6/21/12 12:44 PM)   

    I think that the fundamental problem is the reductionist nature of many reports, and the concept that 'less is best' when it comes to reporting. For reporting risk I think this is particularly problematic because risk is complex and generally speaking heat maps are a very crude tool (and dashboard dials possibly even more so). As an illustration to a more detailed report I think that they can be effective, likewise if they are well explained in a PowerPoint presentation, but I'd be concerned if they were used on their own.

     
    Having said that I do use a heat map variation, which maps risk against opportunity and also shows the effect of planned treatment. I have found this approach works as good way to summarise the outcome of all the analysis and can stimulate a good discussion. If you develop and use risk criteria based on your unique organisational objectives then the plot should show you how concerned you should be about every risk, as all 'red' risks should only be red if there is a real possibility that they could seriously impact on objectives to such an extent that you need to take immediate action to reduce the risk. You do need to provide the criteria alongside the map, and when developing the criteria make sure that the senior leadership understands and endorses the levels.
     
    I am a little confused by Brian's scenario of different tolerances for individual risks that appear to have no relationship to the risk rating. I would think that the risk criteria were not working if a risk rated as medium was considered to be more concerning than one rated as high.
     
    The only issue with reporting all risks using a heat map is that you are likely to have a very crowded map (assuming an organisational focus) which is fine if you are wanting to promote an understanding of the totality of risk, but probably less helpful if you wish to look at individual risks.
  1. Comment by: Jacquetta Goy   (6/21/12 12:45 PM)   

    Sorry about the bolding, I have no idea how that happened!

  1. Comment by: Brian Spires   (6/21/12 01:42 PM)   

    Jacquetta - sorry that I confused you!  My point was that you could have a relatively high residual risk that you accept that risk because perhaps it is primarliy external and there is little to do to remediate or perhaps it is cost prohibitive to drive the risk lower.  On the other hand you could have an individual risk that has a moderate level of residul risk, but you have no desire to accept that risk level and want to drive it lower - that risk needs more attention.

    Maybe an example.  Since it is vacation time for a lot of us, let's say your company owns rental vacation homes on the east coast of the US.  One risk that is relatively high inherently is the risk of hurricane damage.  Yes, you can make sure they are built to code, have storm shutters, etc., but your rersidual risk is still high becuase you just can't control the weather.  But you've factored that into your business model and you have insured to a level that you are comfortable with (still would have a significant impact after insurance).  That risk sits in the red, but I'd argue does not deserve much attention other than watching The Weather Channel.  Another risk might be that your cleaning service does not clean your homes well between rentals and that has a negative impact on your re-rentals and reputation.  You might have that risk at a moderate-moderate level residually, but really only have a tolerance of low-low because it is easy to mitigate that risk away entirely.  I'd say that risk, even though lower rated, should be much more concerning to management and deserving of immmediate attention.

    I agree with Jacquetta that putting all risks on a map would be too crowded to do much more than give an overall sense of the risk environment.  You need to have an established process to determine what risks are elevated and reported on.

  1. Comment by: Norman Marks   (6/21/12 01:54 PM)   

    Brian, I 100% agree that there can be high risks that don't require additional action (perhaps monitoring) and lower risks that can and should be addressed. That is why I prefer to report not only the level but whether action is required because the risk level is outside desired parameters.

    Years ago, I wrote a controversial audit report on the security of the company's mainframe computer system. I reported that while the security was lacking, no immediate action to fix the security gaps was justified on business grounds - because the company was in the process of replacing the system and relocating the data center. Management and the board needed to be aware of the level of risk, but taking action would have cost more than the risk justified.

  1. Comment by: Norman Marks   (6/21/12 01:54 PM)   

    Jacquetta, I will continue to disagree with you that risk management should focus on risk as a negative effect, and believe that management should be aware of risks that are over-controlled, or where the organization is overly risk-averse. I like the idea that risk reports can indicate where action is required because risk levels are below desired levels as well.

    As an example, in a prior life one of my team audited the company's investment policy and pointed out that the Treasurer was more risk-averse in his investment decisions than merited by the business. The organization's overall risk attitude was far more entrepreneurial than the investment policy.

  1. Comment by: Rommel Verdugo   (6/21/12 02:22 PM)   

    Perhaps you can start by defining the risk appetite (risk assessment criteria matrix). If the rules of the game have been previously agreed with all players then there is little chance for failure or misunderstanding, no matter how you report it.

    If 100 mph is not a tolerable level of risk (risk appetite), then I would set up the control strategies that will allow me to maintain this speed wherever I want/need. You can use a heat map to report all of this knowing that, anything above a certain threshold of impact and likelihood (reputational, legal, financial, operational, strategic, etc.) will not be accepted and in accordance the risk should be brought down to a level the company feels more comfortable with.

    The heat map then will have several layers indicating what risks are above my risk appetite and need to be brought down (reduced inherent/residual risk), what risks are inside the company’s risk appetite (I’m ok with their impact and likelihood) and what is below, and perhaps I need to redirect controlling opportunities to higher risk areas (better control/risk return investment).

    Regards,
  1. Comment by: Mark Mitchell   (6/21/12 06:01 PM)   http://www.nyserda.ny.gov

    This article could just as well be entitled the advantage of heat maps.  Every tool has its advantages and disadvantages.  The advantage of the heat map is that it gets attention. An internal auditor must find a way to break the preoccupation that an Audit Committe or management may have with other matters in order to have a meaningful discussion about risks and risk tolerance.

    In order to present how best to present whether or not the risk is presentable, I like to provide a risk rating criteral map that puts each level of risk into context.  But that alone is not enough.  I have found that because management must be involved in determining whether or not they accept each level of risk.  They often have an opinion as to how to put each risk or risk grouping into context.  Sometimes is is a descriptive paragaph or number of paragraphs in a memo that accompanies the heat map.   Other times is is comments in a column within a spreadsheet that appear next to each risk described.

    I like to use the heat map to direct my audiences to a spreadsheet or memo that shows the business function, the objective being evaluated, and a brief description of the impact of the risk rating.

     

  1. Comment by: Mark Mitchell   (6/21/12 06:14 PM)   http://www.nyserda.ny.gov

    This article could just as well be entitled the advantage of heat maps.  Every tool has its advantages and disadvantages.  The advantage of a heat map is that it gets attention.  An Audit Committee or management often have other matters on the forefront of their minds.   in order to have a meaningful discussion about risks and risk tolerance an internal auditor must sometimes find a way to break that preoccupation with other matters.  A heat map can do this nicely.

    In order to present whether or not the risk is acceptable, I like to start by providing a risk rating criteral map that puts each level of risk into context (from a financial, operational, and compliance perspective).  But that alone is not enough.  I have found that because management must be involved in determining whether or not they accept each level of risk, they often have an opinion as to how they would like to see each risk or risk grouping put into context.  Having them do this gets them engaged in the ownership and consideration of the risks.

    Sometimes it is presented as a descriptive paragaph or number of paragraphs in a memo that accompanies the heat map.   Other times it can be comments in a column within a spreadsheet that appear next to each risk described. I like to use the heat map to direct my audiences to a spreadsheet or memo that shows the business function, the objective being evaluated, and a brief description of the impact of the risk rating.

    NB.  This is an edited version of my previous post.  I hope this presents my views more clearly.

     

  1. Comment by: Grant Purdy   (6/21/12 10:24 PM)   http://www.Broadleaf.com.au

     Norman

    I think have either forgot or don't know what a matrix is for.  It is a simplifed graph that describes how we can combine qualitative measures of consequences and their likelihoods. The only inferences to be drawn from it are:

    • the level of risk and therefore the priority for attention;
    • whether treating the risk by modifying the likelihood or the consequences or both is preferred.

    It cannot be used to determine whether a level of risk is 'acceptable' and unless they are very simplistic, drawing lines on the matrix to describe risk criteria is pointless and missleading.

    Risk criteria are the 'lens' through which we can see and understand risks.  They help us decide what is a risk and also how it should be analysed and evaluated.  Rarely are they just simple limits and, even then, there are implications for how the level of risk should be derived.  In most cases, risk criteria are in a system where there may be upper and lower bounds but where, between those two levels benefit cost analysis is applied that looks at at the change in risk compared with the net cost to achieve that change in order to determine the desirability of risk treatment.  

    None of this can be conveyed by drawing lines on a matrix!

  1. Comment by: Norman Marks   (6/22/12 10:07 AM)   

    Grant, I agree with your comment. Unfortunately, some companies only share information about risk levels (in the form of a heat map or similar) with executive leadership and the board. My hope is that this post will point out the need to share additional context (criteria) so that the stakeholders can appreciate whether the level of risk is of concern or not.

  1. Comment by: Anthony Garnett   (6/22/12 01:27 PM)   http://chiefauditexecutive.wordpress.com

    Norman

    I agree, of course all data needs context to turn it into information. The heat map is useful in my view for the fact that the very process of actively putting a spot on the map prompts mature risk management thoughts. Much more so in my experience than ascribing a number to it. It also speaks to what risk management really is, a qualitative, artistic, judgement. 

    The pseudo scientification of risk is not helpful or useful in my view thus criteria and rules always fall short in my view. We are qualitative human people and whilst we can and should assess risk with some structure, it is not a mathematical process for most strategic risks.

  1. Comment by: Norman Marks   (6/22/12 01:54 PM)   

    When it comes to providing the board with reports on risk, what do you give them?

    • Complete details on all risks, or just those that are above an agreed level (P*I)?
    • Only those risk areas requiring their attention - above an agreed level and/or outside risk criteria/tolerance?
    • A summary that describes only those meriting their attention (significant to the organization as a whole), with an attachment so they can drill down into all risks above an agreed level and/or outside risk criteria/tolerance?
  1. Comment by: Wally Walter   (6/23/12 09:06 AM)   

     Absolutely agree with the "context" to which the risks are associated. While some may be a risk for an entity , might be an opportunity for other . Some may be an acceptable risk for an entity , could be converse for other. So perfectly agree with what you state. 

    But at the same time, could I state that such reporting dashboards/ maps are meant to be succinct in an understandable language. It may not fit within a map/ graph/ dashboard, but definitely can be supported with a supporting note/ assumption/ concrete observation. 

  1. Comment by: Jacquetta Goy   (6/25/12 02:04 AM)    One of the traditional uses of heat maps is to allocate particular actions or responses to risks that are plotted in certain zones. For example all red risks must be treated, or all green risks ignored. I think that this is a terrible idea. We use an ALARA approach and consider what response is appropriate for each risk individually (and one form of action could be to consider removing some of the controls if they appear excessive to the level of risk). This is not done by the board however, they see those risks that are assessed as high or critical according to the escalation criteria they have set and they see the action plans too. Some risks will always remain high regardless of what action an organisation might take, but I don't think that this is a good reason not to remind the board that the risk remains, even if it is just to trigger a conversation about whether the context has in any way changed, so when the hurricane happens the board is not taken totally by surprise. My approach to reporting generally follows your third option Norman. I'm not totally sure how risk reporting could easily show risk aversion of the nature that you describe. I suspect that IA would be better placed for this type of drill down. Generally if you are facilitating a group and they score a risk as very high even after you have helped them to test their assumptions and instructed them to follow the criteria then that is the level of risk reported. Of course their risk attitude will then show through the disproportionately high number of risks being reported up to the next level and the discussion ensuing.
  1. Comment by: Tim Leech   (6/25/12 02:24 PM)   http://riskoversight.ca

    Norman: 

    You raise an excellent point.  I have commented on the dangers of heat maps in our March 2012 white paper THE HIGH COST OF ERM HERD MENTALITY.  A central point in my comments is the need to maintain individual risk information in the context of the objective or objectives being assessed.  When risk information is orphaned from the objective or objectives being assessed and/or other risks that may impact the uncertainty of achieving one or more objectives it can result in wrong conclusions on the overall acceptability of the objective(s) residual risk status.  The white paper can be accessed at:

    http://riskoversight.ca/wp-content/uploads/2011/03/Risk_Oversight-The_High_Cost_of_ERM_Herd_Mentality_March_2012_Final.pdf

    Having said that, and having been in the ERM software business for over a decade, I recognize that people are fond of heat maps and more than a few senior executives and boards are accustomed to them and are conditioned to see them.   They should however recognize that risk map information alone can cause problems. 

  1. Comment by: Tony Stephenson   (6/26/12 07:56 PM)   http://www.icomply.com.au

    I have always found the heat map limiting in usefully determining risk.  One reason is that the nature of different risks you are evaluating are very different, and using a single heat map approach does not conveniently reflect these subtleties.  For example, a logistic business might be considering a move into intercontinental liquid fuels transportation and needing to assess the risks of the venture, while also evaluating a new type of lifting device. Using a heat map, both might be high - high, but this says nothing about scale, nor does it provide a priority since both rank equally on the heat map.

    The (software) approach my company has taken is to abandon the traditional heat map in favour of a multi-dimensional "matrix", recognising that not all risks are equal.  The model we use allows the definition of "Risk Types" - as many as are appropriate - for example, IT Security Risks, Fraud Risks, Asset Risks, Workplace Safety Risks, etc etc.  Each risk type can be configured with the usual evaluation criteria like Likelihood and Consequences, but addition criteria can be added also, such as Frequency, Financial Exposure, and any other impacts or risk factors that might be envisaged.  Each "row" in the matrix can have as many named elements as appropriate (eg <100k, 100-500k, 500k-2m, 2-5m, etc), with contextual descriptions for each element popping up to guide the person doing the evaluation.  Each element can be individually scored, and a weighting or multiplier given to each criteria, resulting in a final risk score. 

    Now you can have a meaningful ranking system, because your intercontinental fuel transport project could end up with a risk score of 10000, whereas the lifting device risk score might be 150 - even though they are both labelled "high risk".  And the board will have something meaningful to debate.

     

  1. Comment by: Ian Lyall   (6/26/12 08:48 PM)   

    I think this is a circular discussion. The organisation's risk appetite should determine the criteria to be used to evaluate risk and therefore any risk assessed as red is outside the organisation's risk appetite and must be treated. Insurance is a control and could be used to reduce the consequence and hence the risk level. But, getting back to Norman's speedo example. Doing 100 in a busy suburban street might have a risk level in the red zone. Doing 100 on a country highway might be assessed in the red zone. Putting both of these situations on a heat map would tell you something - you need to treat the risk of having an accident while driving at 100 on busy suburban street.

    Heat maps are not the answer to everything but they are a simple graphical way of presenting information to busy executives on what the highest risks that are, or should be, actively treated.

     

  1. Comment by: J Paterson   (6/27/12 11:57 PM)   http://riskai.co.uk

     There is some very interesting stuff here

    First thing to say is the Uk HM Treasury have some good definitions of risk appetite e.g Hungry, Open and aware  ,cautious and risk averse

    second i think this is very important since line management, senior management and the board may all have different rusk appetites on specific areas some affected by irrational perceptions of dread, familiarity or self interest (to name just a few factors) 

    I 

  1. Comment by: J Paterson   (6/28/12 12:04 AM)   

     Finally the value of making these views explicit is that it mat reveal that some risks are being kept off the risk register because they are "within acceptable limits" from a governance and accountability point of view its dangerous to "collapse" this thinking since thus encourages an "off the books" culture which is a common flaw in risk processes - this is not to imply this task is easy but it is an area for vigilance!

  1. Comment by: david k waltz   (7/2/12 09:31 AM)   http://treasurycafe.blogspot.com

    Norman,

    The instrument dashboard analogy is an item I have mentioned in my blog posts as well. There seems to be a desire to get a metric that essentially "makes the decision for us", when this cannot ever really be the case due to the points you raise above. While metrics can inform us about a situation, the manager's job is to make a judgment about it -  usually in the context of balancing many different factors. This responsibility cannot be avoided by delegating it to a metric. Thanks!

  1. Comment by: David Horn   (7/5/12 07:07 PM)    Norman, As always, great discussion. I think the challenge is to put the results in perspective, but if you have an agreed upon approach, whomever is receiving the heat map, should be happy with how it was reported, since they should have been involved in those design discussions. Where I have also seen the challenge is in the role area around this reporting. Should it be us telling whomever here are your results in the format we agreed upon? or should it be us telling them here are the items we think you need to be worried about? or both? David
  1. Comment by: Matthew Green   (8/25/12 11:43 PM)   

    I very much enjoyed this article.  Before, I go into my explanation on possible solution, let me briefly argue my issue with your 'speedometer' reasoning.  You actually answered your own questions in your speedometer example - 

    When you ask what vehicle is it when you show a speedometer - it cannot be a boat (pit log on a boat) or a airplane - its an air speed inicator on airplane - so speedometer measures land-based vehicles - but not racing cars (Nascar, F1) as they dont use speedometers - so you have a standard manufactured land-based vehicle when showing a speedometer.

    Next, the autobahn is 130km per hour the speed-limit (81mph) so even if you were driving 100mph you would be braking the speedlimit law in all countries which says - yes you are driving too fast...

    Now on to my explanation - 

    The issue is well the heat map is much like a map of world pre-early world exploration - its flat!  Its a single dimension - well its not that easy - as measuring probability and impact are two just many factors to say the very least is the appetite/acceptability, strategy and so on...to be fair the heat map should be mapped to a heat cube - so its 3 dimensional in essence you have something like a rubiks cube  with exception that you have only 3 colours and each space would need to be weighted as it would be possilbe to have say 5 sides Red and yet still be a Green risk...why because you account for the extensions, however you still have account for seperability.

  1. Comment by: Norman Marks   (8/25/12 11:59 PM)   

     Matthew, thanks for the comment. Just one point: that 130kph speed limit in Germany is advisory only. Depending on the conditions, it may be just fine? 130kph may also be legal in France, Australia, Italy, and several other European countries.

  1. Comment by: Gavin Lawrence   (2/20/13 03:58 AM)   http://www.subsea7.com

    I am not convinced by the speed analogy for risk.

    I opine that it is not speed that kills but either dramatic change of direction or extreme decelaration and, of course, both combined.

Leave a Reply