Fairness audit memo
Marcus Delgado · attempt 1 of 2 · 3 of 14The lending model reports 94% accuracy on its holdout set. Before recommending deployment, I re-ran the evaluation with a customer-level split and found accuracy fell to 88%. The original split allowed the same applicants to appear in training and test data, which I believe constitutes leakage.
Subgroup analysis shows a 17-point gap in approval rates between applicants under 30 and over 55, after controlling for income. I recommend reporting the disparate impact ratio alongside overall accuracy and setting a threshold of 0.8 as a review trigger.
Limitations: I did not have access to the feature importance output, so I cannot say whether age is used directly or through a proxy such as tenure…
Comment · you
Good catch. Say what the leakage does to the metric, not only that it exists.
Rubric · 20 pts
16 / 20 suggested
Identifies leakage (Outcome 3.2)
3
4
5
Reports subgroup performance
3
4
5
Actionable recommendation
3
4
5
States limitations honestly
3
4
5
Feedback · AI draft · from the rubric and your 2 comments · edit before sending