How to evaluate an operational risk management framework?

Operational risk managers are required to set up a robust risk management framework for the business and firm, internal auditors must audit the design and effectiveness of it, and regulators will evaluate the framework across all three lines. A robust, efficient framework get accepted by the business and embedded more easily, it avoids audit findings and costly closures, it keeps regulators satisfied, away from unwanted scrutiny or, worse, expensive sanctions or forced remediation.

For firms to achieve this goal, but also for internal auditors and external supervisors to perform their necessary roles with confidence, all parties need to know what good looks like and to use objective, observable and measurable criteria to evaluate operational risk frameworks effectively.

In our experience though, framework evaluation is puzzling for many: clear benchmark is lacking, evaluation is left to personal judgement, auditees get defensive, auditors feel uneasy, balancing between too little oversight or undue severity.

Together, as a team, we cumulate over a century of experience in operational risk practice, oversight and supervision. From sitting on the Basle Operational risk sub-committee, to developing the operational risk oversight for some of the world’s largest banks, but also setting up frameworks in tier 3 organisations, we have seen practices of all types, framework of all forms, and experienced risk culture from best to worst, on every continent.

To share our experience and provide guidance on what to look for in an operational risk management function and framework, we have put together for firms a set of criteria and an assessment's structure articulated around three dimensions: design, maturity and performance. We present the essence of it below.

1. Design; relates to the comprehensiveness and consistency of the framework:

• Comprehensiveness: Do you have all the elements expected in a risk management framework? Regulators and auditors expect to see, especially in the financial industry, the usual building blocks such as: incident data collection, risk appetite and supporting metrics (often called KRIs), risk assessment and control assessment (often called RCSA), action plans and follow-up, scenario analysis, and the internal evaluation of capital. 
• Consistency:Are these elements linked together logically? You build a consistent framework around two pillars: risk appetite and governance. Risk appetite defines the axes and tolerance levels of the heatmap, the thresholds of the monitoring metrics (KRIs), of incident reporting and triggers the action plans. Governance defines the roles and responsibilities of the three lines, organises the structures of decision-making, the ownership and maintenance of policies and documentation, to bring coordination and actionability to the structure. 

2. Maturity; relates to the operability of the design and to the quality of implementation of the framework parts. Maturity criteria include:

• Collection of incidents and near misses is comprehensive and reliable;
• Risk reporting is fed from and back to business lines, focusing on priority issues, deviations and comparisons across similar activities and through time;
• Executive Directors understand the concepts of risk appetite and limits and actively monitor them;
• Risk assessments are comparable across business lines and as objective as possible, using business data and clear rationale for the assessment of risks and controls;
• Scenarios are used to improve decision-making and risk mitigation;
• KRIs and monitoring metrics are preventative and relevant, linked to the risk drivers, to the weak points of the firm's process and to its risk appetite. They are actively used to prevent incidents;
• Risk management culture is valued throughout the organisation.

3. Performance; addresses the essential question of the value of risk management: how do you know, and can demonstrate, it works? Operational risk managers attempt too rarely to assess the impact of their work on the organisation, using the common argument that you can’t show what you have avoided. This is a real missed opportunity to demonstrate the value of risk management through the positive impact of better decision-making in terms of better revenue stability, higher growth, improved process efficiency, project timeliness and reduced delays, higher returns on investments. We can only encourage firms to scrutinise the efficiency and effectiveness of their risk management framework, for better value protection and creation of value.