A way to set your risk tolerance without stretching out your neck too much (i.e. let the data decide for you). This a presentation I have share a few BSA/AML round tables. It demonstrates an approach for setting multiple risk tolerances, which not only make your institution more compliant it is much more more cost effective than setting an ad hoc "conservative" one. I am an economist at heart and prefer precision over a conservative guess. Every institution needs a risk tolerance, otherwise, how do you know when to stop when performing below the line tests?
2. Motivation: Why PB & J
• Sometimes at the end of a very hard day at the end of a long
week, you may find yourself with a nearly empty refrigerator at
dinner time… Ideally you would have a nice lasagna … but…
• The world is rarely ideal….
• ….You may find some stale bread, a half jar of peanut butter and,
if you are lucky, a long forgotten jar of jelly at the back of the
fridge…PB & J sandwich it is!!!
2
3. What is the ideal way to choose your
BSA/AML Risk Tolerances
• Setting Risk Tolerances – A First Best Approach:
1. Use a nearly perfect risk assessment
• Confidently captures and quantifies all the bank’s risks
• Customer risk
• Product risk
• Measures risk mitigating factors or processes
• Identifies the likelihood of criminal activity by activity monitored or rule
2. Use a methodology consistent with what the rest of the bank is using for setting risk tolerances
• Complications:
• BSA/AML models are so different from other models both in concept and regulatory expectation
• Other risk tolerance approaches may not be sufficiently well defined for modification purposes
• There is no clear BSA/AML guidance for this.
• Bring 1 and 2 together and set your risk tolerances for each rule being as quantitatively rigorous as
possible
…Ready for the PB & J sandwich?
3
4. Second Best: A PB & J Approach
• Note this is just one approach and not the ideal approach – Instead of PB &
J you can always choose
• Cold cereal
• Ramen noodles…
• Pizza delivery…
• Similarly there are many equally valid second best approaches. This is just
meant to guide the thought process
• Given that the risk of inaction is worse than less than perfect action
approaches like this one can keep things going while the modeling
techniques mature and become more standard or until clear regulatory
guidance is established.
4
5. The Bread: Choosing the Right Set of Models
• Identify a models with realistic SAR rates
• Realistic?
• Recall SARs are rare events
• Models with SAR rates in the double digits are generally poorly tuned or have an overly
aggressive review process or both
• Model with SAR rates close to zero (less than 2%) are usually the best candidates
• Models with large volumes of alerts produce more robust SAR rates
• Like choosing the wrong bread, if you choose the wrong models they can be easily ripped apart
(or criticized) when spreading the peanut butter.
5
6. The Peanut Butter: Improving the selection
• Just having bread is not enough…we need to make it more palatable
• From the remaining subset of models we need to choose the more conservative ones.
• How is conservativeness identified and supported:
• Create a histogram of most relevant parameter for each models in the subset
• Examine where the threshold lies relative to the highest concentration of transactions
• Try to choose a model with few thresholds
• Example: More Conservative Less Conservative
• If you have more than one choice then repeat the exercise by comparing the threshold and histogram of the second most relevant thresholds
6
7. The Jelly: Measuring the Results & licking the
spoon
• You have narrowed it down to one or two models.
• Now you need to estimate the risk tolerance associated with the rule(s)
• How to measure this estimated tolerance:
• Recall risk tolerance is related to percentage of false negatives (missed cases)
• Perform one below the line tests, as if you are going to tune the model.
• Use a statistically valid sample size
• Manually review the alerts missed to estimate the case rate or productivity rate
• The below the line case rate is your estimated risk tolerance for the rule(s)
• Notice the use of estimated in italics
• Normally risk tolerance is used to set the initial thresholds, here we are making a the assumption that among the large number of rules analyzed at least one represents
the true risk tolerance.
• Law of large numbers – “the average of the results obtained from a large number of trials should
be close to the expected value”
• Sanity check (lick the spoon) – Are you comfortable with the estimated risk tolerance
• If is seems off...toss it… and… try cold cereal, i.e. another approach
7
8. The Sandwich: Putting it All Together
• After narrowing down the models and being comfortable with the results it is time to put everything
together to get your risk tolerance for all the models.
• One risk tolerance will not fit all models therefore you first need to spread your models out and segment
them into categories:
• Low, medium, high risk
• Very low, low, medium, high and very high risk
• Use your risk assessment to perform the segmentation. Think of it as the knife, where even a plastic
knife or even your finger is sufficient.
• More segments allows for better future tuning but also requires more initial judgment
• Use the estimated risk tolerance of the rule(s) and map to its corresponding segment(s)
• Depending on the risk tolerance identified and the number of segments assign a consistent risk tolerance to
each segment. This is judgment driven. Example follows…
8
9. Sandwich Examples
• Example 1
• The estimated risk tolerance is a associated with a HIGH risk rule
• Three segments were chosen
• The estimated risk tolerance for this hypothetical rule was 3%
• Based on judgment of risk and number of segments you choose a 4% increment: which results in
• 7% for MEDIUM and 11% for LOW
• Example 2
• The estimated risk tolerance is a associated with a VERY LOW risk rule
• Five segments were chosen
• The estimated risk tolerance for this hypothetical rule was 15%
• Based on judgment of risk and number of segments you choose a 3% increment: which results in
• 12% for LOW, 9% for Medium, 6% for high and 3% for very high
• These example are designed purely to illustrate the process and are not meant to guide in any way the
assignments of risk tolerance. Also note that the increment need not be linear if the difference the
perceived risk gap between different segments is variable then the increment should be too.
9
10. Enjoy your Sandwich…next is Dessert!
• Congratulations! You now have your risk tolerances for each rule!
• Now you are ready for the FUN part…tuning the model!!!
•Enjoy!
10
