This document discusses best practices for model risk audits, which provide assurance that a bank's model risk management is adequate. It focuses on how a model risk audit team can effectively examine stakeholders in the first and second lines of defense. Regarding the first line, the audit team should assess model development, data quality, usage, performance, output, and human resources. For the second line, the team should evaluate the model risk management function, including the model inventory, validation process, and governance. The goal is to test the overall quality and timeliness of model validation and risk management.

1. The RMA Journal March 201636
BEST PRACTICES
IN MODEL RISK AUDIT
IMPROVING MODEL CONTROL PROCESSES
AROUND THE THREE LINES OF DEFENSE
OPERATIONALRISK

2. March 2016 The RMA Journal 37
BY JACOB KOSOFF
A model risk audit function adds value
by providing assurance to key stakehold-
ers—including the audit committee of
the board—that a bank’s model risk man-
agement is adequate and effective. That
includes its governance, policies, proce-
dures, controls, practices, and operations.
While this article generally describes
a “model risk audit team” at institutions
where it is part of the broader internal
audit function, a credit review team or
analytics audit team could also perform
these functions as long as the reviewers
did not design, implement, or operate
the models. In other words, the review-
ers must be independent of the processes
and controls they evaluate.
Quantitative models drive decision
making in terms of lending, reserve
requirements, capital adequacy, deposit
pricing, instrument pricing, transfer pric-
ing, and compliance diagnostics, to name
just a few critical areas. Consumer lend-
ers approve billions of dollars in loans
each year, relying partly, and in many
cases fully, on analytical models.
Moreover, economic capital adequacy
and allocations, as well as reserves, are
determined—in no small part—by model
output. Since the economic crisis, a three-
tiered process has emerged to manage
model risk within the financial services
industry to ensure that banks are basing
decisions on a sound, evidence-based
analytical framework. Meanwhile, regu-
latory scrutiny of model risk has become
a regular part of the annual examination.
Three Lines of Defense for Model Risk
This article highlights best practices for
internal audit as the third line of defense
for model risk by answering the following
questions:
• What is model risk?
• How can a model risk audit function ef-
fectively examine stakeholders within
the first line of defense?
• How can that same function effectively
examine stakeholders within the sec-
ond line of defense?
It must be emphasized that the fourth
line of defense (external auditors and
regulators) should not be the primary
identifier of significant weaknesses in
model risk management. Material weak-
nesses in the earlier defense mechanisms
are increasingly the source of regulatory
enforcement actions.
What Is Model Risk Audit?
Traditionally located in the internal au-
dit department, a model risk audit team
can be an independent and effective
third line of defense to address model
risk. However, other organizational
configurations are possible, including
a cross-functional team that draws on
resources from credit review, analytics
audit, IT audit, enterprise risk audit, or
other teams that combines quantitative
backgrounds with institutional subject-
matter experts.
Shutterstock,Inc.
The appropriate configuration varies
based on the needs of the institution,
although it is expected that the internal
audit function will report through a line
of control separate from the model risk
management function to ensure inde-
pendence. Traditionally, internal audit
will report to a general auditor or chief
audit executive, who in turn will report
directly to the board of directors.To carry
the appropriate influence, findings from
internal audit should be well documented
and reported directly to the audit com-
mittee of the board.
OCC Bulletin 2011-12, “Sound Prac-
tices for Model Risk Management: Super-
visory Guidance on Model Risk Manage-
ment,” and its Federal Reserve equivalent,
“Supervision and Regulatory Letter SR
11-7,” charge internal audit with assess-
ing “the overall effectiveness of the model
risk management framework—including
its ability to address individual and ag-
gregate model risk.” As with other bank
regulatory audit obligations, internal audit
must evaluate the first and second lines of
defense as part of this process.
The first line of defense for model risk
management is comprised of model devel-
opers, model owners, model processors,
and model users. An evaluation of how the
first-line model risk management function
is designed begins with a review of the ad-
equacy and adherence to model develop-
ment policies and procedures; owner, user,
and processor policies and procedures;
and additional related controls.
The second line of defense for model
risk management generally includes a
model risk management team consisting
of a model governance/controls group, as
well as a model validation department.
The scope of an audit includes an evalu-
ation of the adequacy of and adherence to
corporate-level policies and procedures for
model risk management and validation;
the model validation department’s policies
and procedures for conducting validations
and annual reviews; and documentation
regarding governance.
QUANTITATIVE
MODELS DRIVE
decision making in terms of
lending, reserve requirements,
capital adequacy, deposit
pricing, instrument pricing,
transfer pricing, and
compliance diagnostics, to
name just a few critical areas.

3. The RMA Journal March 201638
Auditing the models within the line of
business is usually performed by a cross-
functional team that includes staff from
operations audit, IT audit, and enterprise
risk audit. References within this article
to a model risk audit team will refer to
this cross-functional team. In addition,
audit may coordinate certain resources
from within the credit review function.
Performing a strong review of the
line-of-business model activities includes
multiple steps performed by various
model risk audit professionals. The fol-
lowing are steps a model risk audit team
should consider. They are not intended to
be exhaustive, but rather to give examples
of key audit testing.
Model policies and procedures. The
audit team should first identify policies
applicable to the examination and per-
form a holistic review to ensure they ac-
curately reflect the role of various model
stakeholders, including model owners
and the model risk management and
validation department. The model risk
audit team should understand the general
scope, breadth, and policy interconnect-
edness. For example, from a consumer
lending origination perspective, policies
to review would include those related to
credit scoring, credit scoring overrides,
scorecard modification, decision engine
origination, scorecard development, and
scorecard performance for the appropri-
ate references to model risk functions
and policies.
In addition, the model risk manage-
ment unit is expected to provide strategic
metrics for the model risk process, includ-
ing an overview of the model risk portfo-
lio by model risk ranking; the number of
completed validations; progress reports for
performing model risk duties; trends in
model weaknesses and reporting to sup-
port the ongoing relevance of the model
population; and information for gover-
nance committees to address disputes
about model adequacy and model use.
A model risk audit team must work well
with external auditors and the regulators
who serve as external oversight. Since
model risk audit is charged with effectively
challenging the model risk process of the
first and second lines, it must ensure that
the review is designed to be effective and
efficient (in other words, it is neither re-
dundant, nor does it create redundancy).
This article highlights best practices
for model risk audit as it performs its
role as a check on the first and second
lines of defense.
Best Practices in Auditing the First Line
of Defense
Model risk audit’s responsibilities include
examining all aspects of the model risk
management process, including those
lines of business or shared service func-
tions that operate, build, maintain, moni-
tor, and modify models. The scope also
includes third-party models that are used,
though management may develop a sepa-
rate control approach for these.
Model development. A core responsi-
bility of model risk audit is to assess the
model development process by evaluat-
ing the control environment in which
models are developed. This is done by
first obtaining the policies and proce-
dures, as well as the model documen-
tation, directly from the model owner.
Then the model risk audit team should
determine whether the model develop-
ment process complied with regulatory
requirements (for example, Basel III
or fair lending regulations), corporate
policies and procedures, and lower-level
policies and procedures. In addition, the
team should ensure the developers have
an internal control process to monitor
their own adherence to policies and pro-
cedures during development.
The conceptual framework, modeling
assumptions, and data acquisition pro-
cess during model development should
also be examined, as well as the effective-
ness of communication from the model
developers to various stakeholders, in-
cluding the second-line model risk man-
agement function or validation groups.
Communication from development areas
in regard to potential model weaknesses,
limitations, responses to validation work
and results, and issue remediation should
be timely and transparent—and demon-
strate a mutually supportive environ-
ment between the model developer and
the second-line functions in managing
model risk.
Data quality. Another step in audit-
ing the first line is to review the controls
related to data quality and relevance. The
model risk audit team should evaluate
management’s controls in regard to the
selection of production and developmen-
tal data, including transformation of data
and how anomalies in data are treated.
This review should include controls for
evaluating the use of third-party data, and
audit testing should include reconcilia-
tion of data used in the models and the
data used in the user acceptance testing
or review of management’s reconcilia-
tion processes. A model risk audit team
should obtain model testing performed
A MODEL RISK AUDIT TEAM
MUST WORK WELL WITH EXTERNAL
AUDITORS AND THE REGULATORS WHO
SERVE AS EXTERNAL OVERSIGHT.

4. March 2016 The RMA Journal 39
body. If a waiver has been granted, the
team should ensure that the owner has
reported the waiver status to management
and all users of the model.
Change management. A model risk
audit team should assess whether the
model owner maintains an appropriate
model change log. Model code should
be appropriately restricted from modi-
fication by developers and users sub-
sequent to model approval. For cases
where changes are deemed appropriate,
provisions should be set out in policies
and procedures.
The model risk audit team should ob-
tain the model change log and perform
two tests. First, the log should be checked
to ensure that each change has a stated rea-
son and that the approval of the change is
noted and supported with testing. Second,
the model risk audit team should ensure
the changes were validated prior to be-
ing used. If changes were implemented
without appropriate validation, the team
should issue a finding.
Model implementation. Model imple-
mentationisacriticalcontrolrequirement
to ensure that data feeds are provided to
the model for computational purposes
in accordance with model specifications.
For example, for a consumer loan origi-
nation model, a model risk audit or credit
audit team should perform operational
testing of the origination decision flow to
ensure loans are assigned the appropriate
risk level as indicated in the model’s doc-
umentation. Model risk audit should also
verify that loan attributes were segmented
correctly and assigned to the appropriate
risk-based pricing channel. This may al-
ready occur in the specific business group
audits performed by other areas within
internal audit or credit review.
Model performance. Models must be
subject to ongoing monitoring to ensure
they continue to produce accurate, com-
plete, timely, and relevant results. The
model risk audit team should obtain a
copy of a model’s specific model monitor-
ing plan and a copy of the most recent
model monitoring reports for review. The
model risk audit team should determine
if the plan is appropriate, if the estab-
lished thresholds are reasonable, and if
the recent model monitoring report ap-
propriately reflects elements in the plan.
Next the team should review the model
monitoring report to determine whether
thresholds have been exceeded or if the
model has exhibited poor performance.
If so, the team should obtain documenta-
tion from management to ensure actions
outlined in the monitoring plan were
taken and adequate reporting to man-
agement has been provided. Additional
follow-up with the model owners and
model validators is warranted to deter-
mine if appropriate steps were taken to
allow continued use of a model that is
performing outside thresholds.
Model output. Review of model
development and usage includes a re-
view of model output. Model risk audit
should ensure management has adequate
controls to report output to model us-
ers and oversight groups. This control
step should also include a review of any
adjustments to model output. If qualita-
tive adjustments are applied to the data,
then model risk audit should evaluate
the governance and oversight related to
the adjustments as well as the support-
ing evidence for the adjustment. Model
risk audit should ensure adjustments are
reinforced by rigorous empirical analysis.
Testing related to the adjustments should
be performed as deemed necessary. For
example, some financial products are
subject to modeling for pricing where
adjustment to modeled prices is made to
accommodate cases of insufficient data or
model inaccuracies.
Human resources. Finally, internal
audit must determine whether model
developers, owners, users, and proces-
sors are appropriately qualified and
whether there are sufficient resources
for model development, processing, and
challenge. This step should be conducted
by identifying model developers, owners,
users, and processors and obtaining their
by developers for data in each model and
ensure that the testing was rigorous, ad-
equate, and appropriately reviewed.
Model usage. Ensuring appropriate
model implementation and usage is key.
To assess this process, a model risk audit
team can first identify the known uses
of each model in the context of the cur-
rent audit. Then the team can identify
the model owner and determine whether
the owner can identify and track all us-
ers (and uses) of the model. The model
owner must follow all user- acceptance-
testing practices prior to implementation,
and model risk audit should assess this
process. This includes the model risk
audit team assessing the IT control in-
frastructure to ensure it includes access
controls, change control processes, ap-
propriate backup and code reviews, and
other key controls.
Finally, model risk audit should de-
termine whether all model uses were in-
cluded and approved in the model valida-
tion report. If uses were not included and
have not been documented and approved
in the report, a model risk audit team
should communicate to the appropriate
bodies the finding that the model is not
being used for an approved purpose. For
credit-related models, this step may often
be performed by a credit audit team.
An additional step regarding model us-
age for model risk audit is to assess the
processes for establishing and monitor-
ing limits on model use. For example, if
automated mortgage collateral valuation
models perform poorly in rural geogra-
phies, model risk audit should determine
if the line of business included appro-
priately automated controls to prevent
model-produced values for rural homes
from being used. Values for rural homes
may instead require appraisals. In a re-
lated matter, a model risk audit team
should ensure that models receiving a
“not fit for use” validation result are not
being used and have been moved out of
production. It should also ensure that an
appropriate replacement process has been
put in place, unless a waiver for use has
been granted by an appropriate governing

5. The RMA Journal March 201640
sheets, or other tools used to facilitate
decision making. Models can support
finance, risk, treasury, compliance, mar-
keting, and other activities.
Typically, model owners are respon-
sible for notifying MRMVD about items
potentially meeting the definition of
models, while MRMVD is responsible for
determining whether something meets
the definition and also for maintaining
and verifying the completeness of the
corporate-wide model inventory. An in-
ventory includes a risk classification for
each model individually as well as in the
aggregate across model types to support
prioritization of model risk management
activities. Therefore, a key component of
testing is to evaluate and test whether
the model determination and model
tiering processes include a rationale and
supporting documentation. The model
risk audit team should also ensure that
the inventory contains all data elements
required by the guidance.
Model validation. Reviewing the
model validation process is one of the
most important functions of an effec-
tive model risk audit team. The guiding
principle is to test the overall quality
and timeliness of the model validation,
resumes or other relevant work history
and continuing-education records. In re-
viewing these items, the model risk audit
team learns if the modeling stakeholders
have the appropriate education, certifica-
tions, or work experience to adequately
develop or change the model. The num-
ber of continuing-education hours and
the content of the training should be
investigated for appropriateness. For
staffing sufficiency, auditors can review
project plans for model development and
work quality to determine if a sustainable
development process is in place.
Best Practices in Auditing the Second
Line of Defense
A model risk audit team’s responsibili-
ties include examining the model risk
management and validation department
(MRMVD). The structures of model risk
groups vary, but may be comprised of a
model governance team and model vali-
dation teams. A model risk audit team
should have various objectives when
auditing the MRMVD, including as-
sessing the adequacy of and adherence
to policies, procedures, and governance
processes surrounding the model risk
management function in order to proac-
tively identify potential impediments to
timely and full compliance with regula-
tory guidance. Specific tasks should also
focus on determining the effectiveness
and independence of the MRMVD.
Model policies and procedures. As
with the first line, a good first step when
auditing the second line of defense is to
assess and test the adequacy of model
risk policies and procedures currently in
place, including the committee structures
and a reconcilement of the policies and
procedures to SR 11-7/OCC 2011-12.
Model inventory. The model risk audit
team should review the controls support-
ing the accuracy and completeness of the
model inventory. Agreeing on the defini-
tion of a model is a pain point for many
financial institutions. While seemingly
simple, a model can comprise a variety of
computational methods, Excel®
spread-
including reviewing model validation
reports and model validation issue-
monitoring and remediation testing.
The first task in the model risk audit
assessment of the model validation activi-
ties and reports is to obtain the model
documentation used in preparing the
validation report, the model validation
work papers, and final model valida-
tion report. The model risk audit team
should evaluate MRMVD’s observations
and findings on the accuracy, relevance,
and timeliness of model development
practices, including data quality and
management. The auditor should also
determine what model validation did
to assess the following items, as well as
conclude whether model validation’s as-
sessment was reasonable:
• Modeling approach and substantiation
of the methodology selected.
• Model assumptions, including risk
factors.
• Model testing.
• Performance monitoring.
• Qualitative adjustments.
A model risk audit team should next
determine whether a validation report
accurately includes the following:
• Clear and comprehensible executive
summaries with a statement of model

6. March 2016 The RMA Journal 41
MRMVD would be expected to have the
experience and expertise to evaluate
models and to challenge assumptions
and approaches undertaken by a model
development staff that is closely aligned
with business activities. Just as with the
first-line assessment, auditors should ob-
tain resumes of MRMVD professionals,
including their work history and continu-
ing-education records. In reviewing these
items, the model risk audit team, com-
bined with MRMVD interviews, can as-
sess whether the MRMVD professional has
the appropriate education, certifications,
work experience, and self-confidence to
adequately challenge model development
professionals. An assessment and related
testing of MRMVD’s planning processes
should be undertaken to evaluate the suffi-
ciency of resources and timely completion
of model validations and annual reviews.
Conclusion
Model risk audit performs its work on
a risk-based standard. Accordingly, the
intent of the audit testing is not to evalu-
ate the full model portfolio but to perform
deep analysis using judgmental sampling.
When weaknesses are discovered, model
risk audit’s role is not to remediate the
weaknesses, but to inform the first and
second lines of defense so they can im-
prove their model control processes and
establish interim controls to mitigate
weaknesses in specific models, model
groups, or model risk portfolio practices.
Once a strong and sufficient second line
of defense is established, appropriately
designed, and operating effectively, many
of the audit tasks listed may be performed
at a lesser frequency, with more reliance
placed on the second line of defense’s
control activities.
Jacob Kosoff heads the Model Risk Management
and Validation Department at Regions Bank. He can
be reached at jacob.kosoff@regions.com.
The opinions expressed in the article are statements
of the author, are intended only for informational pur-
poses, and are not formal opinions of, nor binding on
Regions Bank, its parent company, Regions Financial
Corporation and their subsidiaries, and any repre-
sentation to the contrary is expressly disclaimed.
purpose and a synopsis of model and
validation results, including issues, ma-
jor limitations, and key assumptions.
• The model aspects that were reviewed
during the validation.
• Potential deficiencies or model limi-
tations, including a determination of
whether adjustments or other compen-
sating controls are required.
• Evidence of an independent, effective
challenge of methodologies, data,
implementation plans, monitoring, or
other areas that could lead to noncom-
pliance with regulatory requirements
and corporate policies, procedures,
and standards.
• An appropriate publication distribution
list and communication with stakehold-
ers such as model developers, business
users, and senior management.
Model limitations. The next task in-
cludes assessment of the model valida-
tion issue-monitoring and issue-report-
ing and remediation process. The model
risk audit team should select issues
noted in a sample of MRMVD’s valida-
tion reports and then confirm the issue
was correctly entered into the model
issue database. The reports should ap-
propriately describe the nature, extent,
and importance of the issue. Manage-
ment’s corrective action or remediation
plan and the related target remediation
date should also appear reasonable.
If the issue has been closed, an exami-
nation of the supporting documentation
reviewed by MRMVD should be conduct-
ed to determine whether the issue’s closure
was well evidenced and suitable. If the
issue has not been remediated promptly,
model risk audit should conclude whether
the past-due issue was included in man-
agement reporting. Finally, the model risk
audit team should confirm that models
with severe limitations are taken out of
production until remediation is under-
taken or alternative tools are developed.
Model performance. Once models
have been validated, policies and proce-
dures should provide for monitoring and
review by MRMVD to confirm that model
outputs remain accurate, complete, time-
ly, and relevant and that appropriate ac-
tions are taken to improve models failing
to meet standards. From time to time,
models will be retired or replaced with
better tools. Banks are expected to pro-
vide reporting to both management and
board governance committees that offers
a profile of models in use, residual model
risks, and overall remediation plans at the
model portfolio level.
Model issues database. A model risk
audit team should then perform a security
review of the model inventory and issues
database to ensure that inappropriate ac-
tors cannot alter the records (for example,
by back-dating issue remediation).
Model governance. A model risk audit
team would next review the general gov-
ernance of enterprise model risk manage-
ment (including board and management
committee participation) in determining
the model risk framework and model risk
reporting. The team should evaluate and
test the overall governance and reporting
processes for accuracy, completeness, and
timeliness of key reporting in coordina-
tion with the testing of the model valida-
tion function within model risk manage-
ment. It is particularly important for the
governance bodies to receive information
about the portfolio of models, the residual
model risk exposure, and status reports
evidencing how and when management
will remediate identified weaknesses.
Training on model risk. The model
risk audit team should review stakeholder
training on model risk to evaluate the
sufficiency of the technical and business
knowledge of MRMVD team members
and team capabilities. Additionally, con-
sideration of broader training provided
by MRMVD to the first line, as well as
to key management committees and the
board, should be part of the organization’s
model risk training program.
Human resources. The model risk
audit team should evaluate MRMVD’s
organizational standing and stature. The