AI in Crimnal Justice: Blog Post #3
In my last blog post, we were able to define fairness, both in the context of criminal justice and an AI model. Fairness is an easy concept for humans to understand, but not for an algorithm that lacks human morals. Using the mathematical definition, we can now try to create a model that meets the standards of the definition.
Before we do that, though, we need to talk about a crucial topic in Artificial Intelligence: Accuracy vs. Interpretability. A model that can correctly predict can be “unsuccessful” if it is hard for AI scientists or ordinary people to interpret. Would you rather have a model that predicts correctly 100% of the time but is difficult to interpret or a model that works the majority of the time but is very simple to interpret?
The answer may be situational but having a model that is most accurate with the proper interpretability is key. You may ask, how can we define interpretability in the context of Criminal Justice?
Well, Interpretability is really the What, Why, and How of an AI model. What are the results of the model? (This is the accuracy), Why did the model get to these results? Most importantly, how did the model get to its results? If an AI scientist is working on a Recidivation Model but does not know how it works, how can we trust the model’s process, especially because the model result’s may be biased? If the Judge, assessing the defendant's risk score, cannot comprehend why the model got its results, it doesn't seem fair nor reliable, even if its predictions were correct.
Thus, Interpretability allows us to assess if the model is fair or not.
Importance of Protected Features in different models:
Since we know what fairness is and the importance of interpretability and accuracy we can start creating models. However, I will not be going over how to create these models, as it is a bit complicated right now. I will show graphs to visualize the results of different types of models, which include protected features (characteristics protected by the existing equality legislation).
To the left, are the results of Linear SVMs:
Training Accuracy: 0.68
Testing Accuracy: 0.67
Protected features such as race, gender, and age, dont have a huge impact. (The higher the coefficient value, the more impact it has on the results). There are a couple of outliers, which can be due to that data and code, but overall, protected features have an insignificant impact. In addition, Linear SVMs are one of the easier types of the model so it has interpretability. However, accuracy is quite low. A 67% — 68% accuracy is unreliable. Let's look at two other models.
To learn more about Linear SVMs: http://www.linearsvm.com/#:~:text=Linear%20SVM%20is%20the%20newest,a%20linear%20support%20vector%20machine.
Above, is the impact of race in a Random Forest Model. The code specifically looked at the race because that is what we want to examine. As you can see, African Americans have the highest coefficient value which is quite alarming. However, race overall has an insignificant impact on the outcome which is a step in the right direction.
Training accuracy: 0.78
Testing accuracy: 0.69
In addition, the accuracy is comparatively high, but not perfect. However, Random Forest Models tend to be complicated, so this model lacks interpretability.
To learn more about Random Forest: https://builtin.com/data-science/random-forest-algorithm
The final model is Neural Networks (NN), like Random Forest, which is much more advanced than Linear SVMs and Linear Regressions. Even today, researchers are finding ways to understand Neural Networks.
Training accuracy: 0.72
Testing accuracy: 0.67
The accuracy of Neural Networks is no better than Linear regression, showing that accuracy is not only from advanced models. But as Neural Networks become understandable, researchers can create a more accurate Neural Network.
To learn more about Neural Networks: https://news.mit.edu/2017/explained-neural-networks-deep-learning-0414
Conclusion:
We looked at a couple of different models with varying accuracy and interpretability. Our simple model (Linear Regression) was quite biased and inaccurate. But by separating race (Random Forest, Neural Networks), we were able to get more accurate and unbiased results. However, it is nowhere perfect. Researchers today are looking to create more accurate and reliable models. Allowing for more interpretable models that come with unbiased results is key to successful risk assessment. Models like COMPAS exhibit extreme biases which is a violation of basic human rights. Hopefully, in the near future, AI Criminal Justice applications will be unbiased and treat each defendant fairly, providing understandable and accurate results.
I would like to thank Inspirit AI for this amazing opportunity. Content like this, and other AI applications in the real world can be explored in the link below.
Andrew Cho is a Student Ambassador in the Inspirit AI Student Ambassadors Program. Inspirit AI is a pre-collegiate enrichment program that exposes curious high school students globally to AI through live online classes. Learn more at
https://www.inspiritai.com/.