Operational Risk Loss Forecasting Model for Stress Testing

©2014CRISILLtd.Allrightsreserved.
Operational Risk Loss Forecasting Model
for Stress Testing
A Three-Stage Approach
James Lu
The 17th Annual OpRisk North America 2015, New York
March 2015

Agenda
 Operational Risk Loss Forecasting for Stress Testing
– Regulatory Expectation and Challenges
– Industry Practice and Methodology
 Rebuilding the Operational Risk Loss Forecasting Model
– Background and Challenges
– A Three-Stage Approach
– ARIMAX Model to Project External Losses
– OLS Model to Project Internal Losses
– Simulation to Estimate Tail Risk Using Scenario Analysis
– The Model Results
– Conclusions and Limitations
2

Operational Risk Loss Forecasting for Stress Testing
Regulatory Background and Expectation
3
 The Fed’s CCAR requires banks to have internal capital planning processes
– Operational risk loss is a component of PPNR, which is the first building block of the CCAR process
 Operational risk loss estimation methodologies vary among banks and so do the Fed’s expectations
 The Fed has given guidance regarding estimation methodologies
– Stronger practices vs. lagging practices
 Banks should use internal op risk loss data as a starting point to provide historical perspective
– They can then incorporate forward-looking elements, idiosyncratic risks, and tail events to estimate
losses for the CCAR projection window
Operational Risk Loss Forecasting Challenges
 Limited internal loss history
 Link loss history with macroeconomic drivers, including bank-specific risk drivers
 Quantify extreme severe but plausible loss events

4
Operational Risk Loss Forecasting for CCAR
Industry Practice and Challenges
Operational Risk Loss Estimation Flowchart
Banks’ approaches vary in each of the following steps due to data richness and business
sophistication. A few banks conduct benchmark analysis.
Internal loss historical database, supplemented by
external historical loss
Macroeconomic variables
Scenario analysis add-on to capture tail risk which internal loss history may have not experienced
Analytical models or judgment-based estimation to predict future ops risk losses for the baseline
scenario
Analytical models or judgment-based estimation to predict future ops risk losses for stressed
scenarios and bank idiosyncratic risk
Management overlay for legal/litigation/compliance risk and other risks
Benchmarks to either support or challenge the loss projections

Industry Practice and Challenges
Common operational risk loss estimation methodologies
 Modeling Approaches
– Regression model using macroeconomic variables
– The ARIMA/ARIMAX model to capture the auto regressive nature of loss history
– Modified loss distribution approach
– Scenario analysis to capture tail risk
Criteria for Selecting the Final Model
 The selected modeling approach should have the following characteristics:
– Segmentation needs to be granular enough given the data availability
– Model design and outcomes should be empirically robust and statistically sound
– Model performance needs to exhibit reasonable levels of stability and accuracy
– Scenario analysis add-on and management overlay should be fully reviewed, challenged and well
documented
5

Agenda
 Operational Risk Loss Forecasting for Stress Testing
– Regulatory Expectation and Challenges
– Industry Practice and Methodology
 Rebuilding the Operational Risk Loss Forecasting Model
– Background and Challenges
– A Three-Stage Approach
– ARIMAX Model to Project External Losses
– OLS Model to Project Internal Losses
– Simulation to Estimate Tail Risk Using Scenario Analysis
– The Model Results
– Conclusions and Limitations
6

Modeling Challenges
Rebuilding the Ops Risk Loss Forecasting Model
Background and Challenges
7
 The bank has less than four years (i.e.15 quarters) of operational risk data in its database,
which reliably and consistently captures operational risk loss dates, loss events, lines of
business, recoveries, etc.
 ORX ops-risk database has quarterly loss data since 2002. How do we link external loss
history with internal loss history?
 What is the granularity of modeling, given both the richness of external loss data and short
history (and data sparseness) of internal loss data?
 Literature and industry experience indicate a lack of strong predictors for ops-risk loss
model, so developing statistically sound and empirically robust models is a challenge
 Quantitatively-driven scenario analysis add-on as a tail risk is still evolving and therefore a
challenge for most banks to factor it into their models
New Modeling Approach
 The newly developed three-stage model and quantitatively-driven scenario analysis
respond to the challenges faced by both the bank and the industry

Three-Stage Modeling Methodology
8
 The three-stage model utilizes both external and limited internal ops-risk loss data to
forecast the future nine quarters’ (9Q) ops-risk loss for the bank in both the baseline and
stress scenarios
 Management overlay is added to capture legal/litigation/compliance risk
 The implemented model meets the regulatory guidelines for stronger practices regarding
ops-risk loss forecasting methodologies
Ops-Risk Loss Forecasting Model
Stage 1 Stage 3Stage 2
The ARIMAX model to
forecast future
9Q external
ops-risk loss by
geography and LoB using
macroeconomic variables
The OLS model to forecast
future
9Q internal ops-risk loss
by geography and LoB
using projected external
quarterly loss from Stage
1 and
macroeconomic/bank
idiosyncratic risk drivers
Simulate scenario analysis
results to capture tail risk
that was not experienced
in internal loss history

9
Three-Stage Modeling Process Flowchart
The ARIMAX model to
predict 9Q ops-risk
losses for CCAR
horizon by geography
and LoB
ORX
Loss
History
Macro
Variables
Projected external
losses by
geography and LoB
Mapped
Macro
Variables
Mapped
Bank
Specific
Variables
Bank
Internal
Loss
History
Projected 9Q of
internal losses by
geography and LoB
The regression model
to predict 9Q ops-risk
losses for CCAR
and LoB
Scenario
Analysis by
Event Type 1
Scenario
Analysis by
Event Type 2
Scenario
Analysis by
Event Type 3
Projected 9Q internal
losses with tail risk
Projected 9Q internal
losses with tail risk
and management
overlay
Stage 1
Stage 2
Stage 3

Stage 1: ARIMAX Model Using External ORX Data (1/3)
10
The ARIMAX model to
predict 9Q ops-risk
losses for CCAR
horizon by Geography
and LoB
ORX
Loss
History
Macro
Variables
Projected external losses
 Map the bank’s portfolio to ORX data structure by geography and LoB. There
are six geography and LoB combinations. This mapping is based on the
assumption that the bank and other banks in ORX have similar risk drivers by
geography and LoB combination
 Data integrity and cleansing include data series normalization, missing data and outlier treatment,
specifically quarterly external loss is normalized by the number of banks in the ORX.
 Variable transformations are tested, and the selected dependent variable is the logarithm of quarter-over-
quarter change of ops-risk loss per bank
 The ARIMAX model offers the following benefits:
– Captures the relationship between observations across time
– Estimates relationships and produces forecasts that utilize both the information in past values of the series (quarterly
loss) and the information contained in independent variables (macroeconomic variables)
– Methodology is mature and easy to diagnose and estimate
– ORX data series has 12 years of data and covers economic cycles
 The ARIMAX model has one limitation:
– In its data history, ORX data does not capture the bank’s portfolio size, which could be a driver
GEO 1 – Retail
GEO 1 – Commercial
GEO 2 – Retail
GEO 2 – Commercial
GEO 3 – Retail
GEO 1 – Corporate & Others
Bank
(Geography & LoB)
GEO 1
GEO 2
GEO 3
GEO 4
GEO 5
ORX Data
Geography LoB
Retail
Commercial
Corporate & Others

𝜆 𝑡,𝑝
𝑂𝑅𝑋
= 𝑙𝑜𝑔
𝐿 𝑡,𝑝
𝑂𝑅𝑋
𝐿 𝑡−1,𝑝
𝑂𝑅𝑋 = 𝑙𝑜𝑔𝐿 𝑡,𝑝
𝑂𝑅𝑋
− 𝑙𝑜𝑔𝐿 𝑡−1,𝑝
𝑂𝑅𝑋
= 𝑓 𝑥 𝑝, 𝜙 𝑝, 𝜃 𝑝
Stage 1: ARIMAX Model Using External ORX Data (2/3)
11
 The ARIMAX model specification
– AR: Autoregressive – Time series is a function of its own past
– MA: Moving Average – Time series is a function of past “shocks”(deviations, innovations,
errors, etc.)
– I: Integrated – Differencing provides stochastic trend and seasonal components, so
forecasting requires integration or undifferencing
– X: Exogenous – Time series is influenced by external factors
 The selected dependent variable is a logarithm of QoQ ops-risk loss ratios (𝜆 𝑡,𝑝
𝑂𝑅𝑋
) for a proxy of
geography and LoB combination, and it measures the relative change of ops-risk loss. It can be
modeled as a function of macroeconomic variables 𝑥 𝑝 along with lagged values of ORX log-
returns (ϕ) and the lagged forecast errors (θ) which are specific to that proxy (i.e.𝑓 𝑥 𝑝, 𝜙 𝑝, 𝜃 𝑝
 The ARIMAX model combines time-series analysis and regression analysis. Hence, in many cases, it can produce better
forecasts than using either technique alone. The ARIMAX model includes a structural (economic) explanation for the part
of the variance of the dependent variable that can be explained structurally while explaining the part of the variance that
cannot be explained structurally with a time-series model.
 We choose the best model based on
– Low AIC
– Low MAPE
– Statistical significance of macroeconomic variables, MA, and AR terms
The ARIMAX model to
predict 9Q ops-risk
losses for CCAR
and LoB
ORX
Loss
History
Macro
Variables

Stage 1: The ARIMAX Model Using External ORX Data (3/3)
12
 The model outcomes indicate that all macroeconomic and structural variables are significant
and economically explainable
 The same model is used to project 9Q external losses for baseline and stress scenarios
The ARIMAX model to
predict 9Q ops-risk
losses for CCAR
and LoB
ORX
Loss
History
Macro
Variables
GEO 1 – Retail Portfolio
Conditional Least Squares Estimation
Parameter Estimate
Standard
Error t Value
Approx
Pr > |t| Lag Variable
MA1,1 -0.47548 0.15491 -2.88 0.0066 1 log_loss_ratio_Rtl
VAR1 6.84435 3.19104 2.68 0.0110 0 CPI_SA_YY_Q8
VAR2 1
GEO 1 – Commercial Portfolio
3.02520 4.12118 3.65 0.0008 0 Unemp_YY_Abs_Q4
VAR4 0.25252 0.05616 3.78 0.0005 0 GDP_QQ_Rel_Q8
Conditional Least Squares Estimation
Parameter Estimate
Standard
Error t Value
Approx
Pr > |t| Lag Variable
MA1,1 -1.15427 0.43507 -2.75 0.0100 1 log_loss_ratio_Cml
AR1,1 -1.12522 0.37971 -2.81 0.0084 1 log_loss_ratio_Cml
VAR1 27.27729 12.40279 2.04 0.0502 0 Unemp_QQ_Abs_Q6
VAR2 -0.0008847 0.0004363 -2.27 0.0300 0 GDP_QQ_Abs_Q2
VAR3 0.0552E-7 9.5156E-8 -2.18 0.0369 0 Pers_Bk_Filings_YY_Abs_$_Q4

13
Projected
external
losses by
geography
and LoB
Mapped
Macro
Variables
Mapped
Bank
Specific
Variables
Bank
Internal
Loss
History
Projected 9Q
internal losses
by geography
and LoB
Regression model to
predict 9Q ops-risk
losses for CCAR
and LoB
Stage 2: Regression Model for the Bank’s Internal Loss Forecasting (1/2)
 The Stage 2 model uses the forecasted external losses from the ARIMAX model in Stage 1, along with
macroeconomic and bank-specific risk factors to forecast the bank’s internal losses. The regression
model was selected over the ARIMAX model due to the short history of internal loss data. Internal loss
forecast model is specified as
𝐿 𝑡,𝑝
𝐼𝑛𝑡
= 𝐿 + 𝑥 + 𝜆 𝑡,𝑝
𝑂𝑅𝑋
That is, the bank’s internal loss is a function of the average quarterly historical loss 𝐿, macro and bank-
specific variables 𝑥 (including balance sheet size), and the forecasted values of the ORX losses, 𝜆 𝑡,𝑝
𝑂𝑅𝑋
 It is worthwhile to note that
– We include forecasted external loss as an internal loss driver. This is based on the assumption that
the bank and other banks in ORX share similar operational risk loss trends
– We include the bank’s internal loss average in the regression model on the assumption that the
historical average is the starting point of future loss. In other words, future loss will be around the
historical average, adjusted by forecasted external loss, other risk factors, and balance sheet size

14
Projected
external
losses by
geography
and LoB
Mapped
Macro
Variables
Mapped
Bank
Specific
Variables
Bank
Internal
Loss
History
Projected 9Q
internal losses
by geography
and LoB
Regression model to
predict 9Q ops--risk
losses for CCAR horizon
Stage 2: Regression Model for the Bank’s External Loss Forecasting (2/2)
 The model outcomes indicate that the identified risk drivers are significant and economically explainable
 The same model is used to project the baseline scenario and all stress scenarios for 9Q of the bank’s internal losses
GEO 1 Commercial Internal OLS Model (Adjusted R 2 = 0. 52)
Parameter Estimates
Variable Label DF
Parameter
Estimate
Standard
Error t Value Pr > |t| Tolerance V.I
CPI_SA_YY_Q4 1 -176.0824 46.56550 -4.04 0.0014 0.99499 1.0
log_lossratio_hat Forecast for log_dloss_sm4 1 14.6227 7.06965 2.29 0.0397 0.99499 1.0
Analysis Variable : MAPE
Minimum Mean Median Maximum
0.0205061 1.7752406 0.8452006 9.2587490
GEO 1Retail Internal OLS Model (Adjusted R2
= 0.852)
Parameter Estimates
Variable DF
Parameter
Estimate
Standard
Error t Value Pr > |t| Tolerance
Variance
Inflation
Intercept 1 15.20999 2.94609 5.51 0.0003 . 0
Auto_Sales_NSA 1 2.42056 0.46628 4.60 0.0010 0.18560 5.38782
HPI 1 -0.10252 0.02193 -4.38 0.0014 0.61482 1.62650
Loglossratio_hat_Q3 1 -0.84000 0.31150 -2.66 0.0239 0.53596 1.86580
Analysis Variable :MAPE
Minimum Mean Median Maximum
0.0258840 0.2257054 0.1640946 0.8851183

Performance Testing
15
 Model performance testing was conducted to compare the new model with the previous
years’ models on both model stability (Adjusted R2) and predicting accuracy (Mean
Absolute Percent Error, or MAPE*)
 The new model is significantly better than the previous model on both stability and
accuracy. In addition, the new model more appropriately reflects the risk characteristics of
ops-risk and is better aligned with the supervisory guidance of operational risk loss
forecasting methodologies for CCAR
 The Fed examiner’s comment was positive: “Model better designed and performance
improved significantly.”
* Mean Absolute Percent Error (MAPE): Absolute (Forecast – Actual) / Actual
** Not based on the same model specification, therefore not a direct comparison
Model Performance Comparison – Stability**
Model Performance Comparison – Accuracy
Geography & LoB
Prior Model New Model
R2 Adj. R2 R2 Adj. R2
GEO 1 Retail 0.18 0.13 0.88 0.85
GEO 1 - Commercial 0.21 0.14 0.55 0.52
Geography & LoB
Prior Model New Model
Min Mean Max Min Mean Max
GEO 1 Retail 12% 136% 892% 2.30% 23% 89%
GEO 1 - Commercial 2% 359% 6629% 2.00% 178% 926%

16
Projected 9Q
internal
losses by
geography
and LoB
Scenario
Analysis by
Event Type 1
Scenario
Analysis by
Event Type 2
Scenario
Analysis by
Event Type 3
Projected
9Q internal
losses with
tail risk
Projected 9Q
internal losses with
tail risk and
management
overlay
 The Stage 3 of the model involves simulating scenario analysis results to capture ops-risk tail risk, which
are the frequency and severity that current internal loss history has not experienced.
 Scenario analysis is based on the seven Basel II AMA ops-risk loss event categories*. A group of SMEs
discussed and selected certain categories that are specifically applicable to the bank’s exposure and
loss experience
 For the selected categories below, SMEs determined the “Most Likely” and “Worst Plausible” loss
frequency and severity
 We use Poisson distribution to simulate loss distribution to select certain percentiles as tail risk losses
under extreme scenarios. We select the median as the baseline and 55 percentile for adverse and 60
percentile as the severely adverse and BHC stress scenarios (e.g. a scalar of 118% is applied to the
adverse scenario over the baseline scenario for each of the 9Q in the projection window)
Stage 3: Simulation of Scenario Analysis for Ops Risk Tail Risk
* The seven loss event categories are internal fraud; external fraud; employment practices and workplace
safety; clients, products and business practices; damage to physical assets; business disruption and system
failure; and execution, delivery and process management
LoB Event Category M.L Freq. M.L. Sev. M.L. Loss W.P. Freq. W.P. Sev. W.P. Loss W.P./M.L. Ratio
GEO 1 Commercial CPBP - Securitization Breach 10 0.25 2.5 0.0067 5000 33.5 13.4
GEO 1 Retail Internal Fraud 1500 0.0001 0.15 0.02 500 10 66.67
Corporate Treasury EDPM 1 2.5 2.5 0.02 200 4 1.60
Corporate IT BDSF - Outage 60 0.01 0.6 0.002 50000 100 166.67
LoB Event Category 45%ile Median 55%ile 60%ile 65%ile
GEO 1 -
Commercial
CPBP - Securitization
Breach
84% 100% 121% 145% 175%
GEO 1 - Retail Internal Fraud 93% 100% 106% 117% 142%
Corporate
Treasury
EDPM 75% 100% 132% 168% 215%
Corporate IT BDSF - Outage 90% 100% 113% 129% 151%
Average 86% 100% 118% 140% 171%

Ops-Risk Loss Forecasting Results (Bank Total Ops Risk Loss)
17
 The bank’s ops-risk loss forecasting for the baseline scenario and three stress scenarios
 Tail risk scalars of 118% and 140% are applied to adverse and BHC stress/severe adverse scenarios,
respectively, over the baseline loss projections
 Finally, management overlay of $150mn (scaled) is added to the three stress scenarios to cover
legal/litigation/compliance risk for the projection window (not included in the graphs below)
0
20
40
60
80
100
120
The Bank Ops Risk Loss Forecasting
Baseline Adverse BHC Stress Sev. Adverse
0
20
40
60
80
100
120
140
160
180
The Bank Ops Risk Loss Forecasting with Tail Risk
Baseline Adverse BHC Stress Sev. Adverse

Conclusion and Limitations
18
 The link between Stage 1 and Stage 2 of the model is based on the assumption that the
bank’s ops-risk losses and other banks’ losses have similar loss drivers within the same
geography and line of business
 Historical average as the starting point for internal loss projection should be on the
conservative side and also in alignment with regulatory expectations (historical
perspective view)
 Scenario creation is not based on quantitative analysis, but rather based on forward-
looking of stress scenarios pertinent to the bank’s portfolio and also based on known loss
experience
 Tail risk scalars are judgment-based
 This ops-risk loss forecasting does not incorporate internal control (e.g. Risk Control Self
Assessment or RCSA) and other risk indicators into the estimation methodology.
However, a few banks have incorporated these into their estimation methodologies
 Benchmark and sensitivity analysis have not been conducted, but they are Fed
expectations in the long run

www.CRISIL.com/gra

Operational Risk Loss Forecasting Model for Stress Testing

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

En vedette

En vedette (20)

Similaire à Operational Risk Loss Forecasting Model for Stress Testing

Similaire à Operational Risk Loss Forecasting Model for Stress Testing (20)

Operational Risk Loss Forecasting Model for Stress Testing