Do you find it interesting that the opening requirement of the safety assurance element of API 1173 is optional? “The operator should evaluate the application of its PSMS and determine whether expected progress toward effective risk management and improved safety performance is being achieved.” The word “should” is used within the requirements of API 1173 12 times. In most cases, the word is used as a qualifier to identify a potential outcome from executing some portion of the recommended practice. Since the two primary goals of API 1173 are to reduce risk and improve safety performance, this optional requirement is essential and necessary to execute.
SAFETY ASSURANCE OUTCOMES
In our blog on operational controls, we discussed the need for processes to validate if changes have been implemented effectively and completely. This includes ensuring that the improvements being made, from the risk management and continuous improvement processes, are achieving the defined goals and objectives of the safety management system (SMS), including reducing risk and improving safety performance.
The safety assurance element of API 1173 is about making sure that your SMS is:
It is necessary to put a structure in place that allows you to measure the progress and effectiveness of your SMS against an established baseline. This is important not only to show how you are complying with the requirements but, more importantly, to demonstrate that you are reducing risk.
Additionally, the safety assurance process feeds the cycle of continuous improvement and should drive your safety goals and objectives for the coming year as part of the management review. This, combined with the specific risks identified for mitigation, can provide much or all of the safety improvement plan for the upcoming year.
SMS PROGRAM EFFECTIVENESS
There are two ways an SMS should be measured–how effective is the SMS, and are you reducing risk?
The starting point for this assessment is to determine whether procedures have been developed to meet all requirements of API 1173. Auditing is one of the more effective ways to determine this and can be done using your internal audit team or external resources.
Internal audit processes use well-exercised frameworks with defined management action plans and closure expectations. These create the continuous improvement cycle and maturity plan for the elements and support compliance with the requirements. Additionally, internal audit departments have established connections and processes that inform executive leaders and board-level committees of audit results. These processes provide opportunities to gain support for the development and maturity of the SMS.
Leveraging an external resource to perform these audits can provide added value and accelerate the maturity of your SMS by recommending other practices your internal audit team may not be familiar with.
Going beyond compliance is not only critical but also is why there are requirements to assess the maturity of your SMS. This measurement is subjective and is why an outsider’s perspective is important when evaluating the quality of the process and not just ensuring compliance.
Perhaps the most important aspect of maturity is the assessment of safety culture. Safety culture has traditionally been evaluated from the perspective of occupational safety. Adding the elements of pipeline, process, and public safety to the occupational focus provides greater insight into safety culture, processes, and specific areas for improvement.
RISK MANAGEMENT EFFECTIVENESS
Measuring the effectiveness of your risk management efforts is what the SMS is all about. Are your risk mitigation activities reducing your operational risks? Your risk management program should be able to provide easy-to-understand metrics that quantify your change in risk over time.
At an aggregate level, your risk level will be influenced by your risk mitigation efforts and emerging risks. These often reflect the quality and maturity of your risk management processes, stakeholder engagement, and leadership’s commitment to risk management. You should be able to demonstrate a change in risk, and your processes, documentation, and record-keeping efforts will explain the difference.
Additionally, a well-structured risk register will allow you to evaluate the change in risk over time at various levels–departmental, functional, or by category or process.
Measuring risk at an individual level allows you to show progress over time and evaluate if your efforts were effective. This is especially true early in the implementation when your risk register is more volatile, stakeholders are engaged, and you can demonstrate improvement even if your aggregate risk score increases.
There are other ways to measure the effectiveness of your risk management program and your SMS. Such as measuring the quality and timeliness in completing your risk mitigation projects, the level of engagement from your stakeholders, and the results of your quality control testing. These provide additional insights into the effectiveness of your program and show that your mitigation activities have been successful.
KEY PERFORMANCE INDICATORS
A question we often receive from our clients relates to establishing key performance indicators (KPIs) and measuring their SMS. There are multiple ways to leverage KPIs within an SMS. The graphic below provides a framework for how KPIs may be established and organized.
SMS Program Metrics demonstrate the progress of your SMS program. Results are typically reported annually as part of the annual management review process. Examples may include the implementation progress, audit results, maturity, and safety culture assessment results.
Risk Management Metrics demonstrate the change in risk from the baseline assessment and the prior year’s results at the aggregate, departmental, function, and individual risk levels. This includes a change in risk associated with specific risks where mitigation activities were completed. The specific risk metrics monitored are likely to change every year as the risk mitigation efforts mature and the prioritization of risk management activities changes.
Engagement Metrics demonstrate the level of engagement in the SMS from various stakeholders. Employees or contractors submitting risks or corrective actions, employees participating in continuous improvement and risk mitigation projects, near misses submitted by employees or contractors, and leadership SMS engagement meetings are examples of stakeholder engagement metrics.
Pipeline Safety Program Metrics are ongoing metrics associated with high-risk processes that are likely to be a continued priority. Excavation damage rates, emergency response times, leak backlogs, integrity assessments, and bare steel or cast-iron mains replaced are examples of program-level metrics that are actively tracked and monitored throughout the year as part of an SMS.
Ultimately, you are expected to be able to show that you are reducing risk and improving safety, and your metrics help tell that story.
MEASUREMENT CHALLENGES
Certain annual metrics don’t lend themselves to production on a monthly scorecard, such as maturity or safety culture assessment results, which are once-a-year or every-other-year activities.
Additionally, change in risk often takes a while before the results are evident in the metrics. It may take many cycles to see the value of risk mitigation activities before determining if your efforts were effective.
These longer-term metrics lend themselves to annual reviews and effectiveness assessments and should be incorporated into management reviews. On a short-term basis, it is important to focus your reporting on the progress and activities you are trying to complete within the year, particularly related to risk management.
When evaluating metrics, answer these questions: Are you completing your risk assessments? Are you implementing your continuous improvement projects on schedule? Are your excavation damage rates or emergency response times better than planned? Activities that occur frequently and are easily measured are often included in monthly SMS scorecards. This would include items such as control testing results.
The goal is to quickly communicate to leaders if additional actions are needed to address any emergent issues and tell key stakeholders that the plan is being executed and making the desired progress.
LEADING VS. LAGGING INDICATORS
There is a lot of discussion on transitioning away from lagging indicators and leveraging leading indicators as a measure of SMS success. It is important to incorporate leading and lagging KPIs to understand if your SMS is meeting its objectives, particularly when evaluating the effectiveness of risk mitigation and continuous improvement efforts.
The risk assessment process allows you to identify the preventive controls and improvements needed to reduce risk. Metrics associated with those preventive controls need to be identified and evaluated for effectiveness in design and execution. It will take time to see the effects of the process improvements in the lagging indicators, which is one reason the leading indicators are essential.
The additional benefit is judging if there is a cause-and-effect relationship between the control and the outcomes to ensure you’re investing time in the processes most closely tied to the desired outcomes. Measuring leading indicators associated with preventive controls and aligning those results against the lagging indicators will provide the visual cues needed to understand the effectiveness of your actions.
At ENTRUST Solutions Group, we help clients develop their SMS metrics, scorecards, and safety assurance procedures. We also work with clients to develop systems, metrics, dashboards, and reports that allow them to see the effectiveness of their SMS.
Our expert team has the experience you need to conduct audits, complete effectiveness assessments, evaluate maturity, and assess safety culture.
For more information about safety assurance or implementing an SMS, please contact Jim Francis at jfrancis@enstrustsol.com or 713-324-3950.