Guide to Building Data-Driven Organizations in the Public Sector

Bias in Prediction

Team 5

Topic Overview

I found this description of a model and found it very helpful, and I hope you do as well: “A model, after all is nothing more than an abstract representation of some process, be it a baseball game, an oil company’s supply chain, a foreign government’s action, or a movie theater’s attendance. Whether its running in a computer program or in our head, the model takes what we know and uses it to predict responses in various situations. All of us carry thousands of models in our heads. They tell us what to expect, and they guide our decisions” (p. 18).

There is no such thing as a perfect model because conditions are always changing and when this occurs the model must adjust as well. For every model we are reviewing we must ask ourselves who designed the model, and what is the company or person wanting to accomplish? After all models carry opinions and are embedded with mathematics. Some other important questions to ask are, “Even if the participant is aware of being modeled, or what the model is used for, is the model opaque, or even invisible?” (p. 28). Does the model work against the subject’s interest?” (p. 29). The third question you need to ask is whether the model has the capacity to grow.

Crime prediction software helps to provide valuable information the law enforcement so they can position their police units where crime is most likely to occur, which allows police departments to optimize resources. These types of models do not discriminate against race or ethnicity and doesn’t focus on the individual but targets geography. The flipside to the coin is that “our prisons fill up with hundreds of thousands of people found guilty of victimless crimes. Most of them come from impoverished neighborhoods, and most are black or Hispanic. So even if a model is color blind, the result of it is anything but. In our largely segregated cities, geography is a highly effective proxy for race” (p. 87).

“While looking at WMDs, were often faced with a choice between fairness and efficacy. Our legal traditional lean strongly toward fairness” (p.94). However, WMDs tend to favor efficiency because they are fed by data which can be measured and counted. Fairness is a difficult concept to quantify, and programmers don’t know how to create a code for it, while their bosses never really ask them to pursue this issue. A big question that gets overlooked is whether we are willing to give up some efficiency to gain fairness.

Chapter Summaries:

Bomb Parts

The Bomb Parts chapter goes into describing three kinds of models. The first model discussed involves Baseball and is discussed as a model because of its predictive mathematical modeling. It is considered one of the healthy case studies in contrast to some of the other toxic models often referred to Weapons of Math Destruction (WMD). The baseball model is considered fair due to the readily available statistical information that most anyone knows how to interpret, this it what makes it transparent when compared to some of the other models being discussed in this chapter.

The second model is referred to as a dynamic model which covers the use of memorizing, updating and adjusting your model based on what kinds of foods your family likes and dislikes. It’s a model that I think most of us use today to try and ensure that everyone at the table gets a healthy meal while providing them with something they will eat. This is a model that requires time, data updated daily and seeking feedback from family members. “No model can include all of the real world’s complexity or the nuance of human communication. Inevitably, some important information gets left out” (p. 20).

Finally, the last model utilized is a hidden model and lacks transparency. Level of Service Inventory-Revised (LSI-R) is one of the more popular models discussed. This gathers data from those convicted to asses their level of risk for recidivism. “This questionnaire includes circumstances of a criminal’s birth and upbringing, including his or her family, neighborhood, and friends. These details should not be relevant to a criminal case or to the sentencing” (p. 26). This brings in the questions of fairness, and transparency because without it how can this model be fair to anyone?

Civilian Casualties

Reading, Pennsylvania which is about 50 miles west of Philadelphia used to be a reach city that grew from railroads, steel, coal and textiles. After the 2008 market crash tax revenues dropped significantly and the city had to lay off 45 officers from the police department. It later (2011) became a city that had the highest poverty rate in the country (41.3%). Due to this layoff the city adopted a software program called PredPol that analyzes historical crime data and calculates where crime is most likely to occur. There was a significant reduction in burglaries the following year (23%). Other major cities followed suit and either utilized the same software program or other similar programs such as HunchLab and CompStat.

An article from the Atlantic Monthly seemed to think that low-level crimes and misdemeanors were the breading ground for influencing serious crimes to move in by chasing off law-abiding citizens. To fix the situation the city worked to fix broken windows, cleaned up graffiti and took steps to discourage misdemeanor crimes.

In the 1990s a zero-tolerance movement gained support, “and the criminal justice system sent millions of mostly young minority men to prison, many of them for minor offenses” (p.88). These polices in New York became the stop and frisk anticrime policies. Over the next decade these stops increased by over 600% to nearly 700,000 incidents and about 85% of them were young African American or Latino men but only 0.1% were linked to a violent crime. This policy was ruled unconstitutional by judge Shira A. Scheindlin in August of 2013 because of the fourth and fourteenth amendments.

Each of these approaches represents a model, and each crime-fighting model calls for certain input data followed by a series of responses, and even calibrated to achieve an objective” (p.88). What we need to be looking at is how to reduce recidivism and looking at other factors such as if their time behind bars and what kinds of effects such as solitary confinement has on inmates while in prison. However, “all too often they use data to justify the workings of the system but not to question or improve the system” (p. 98).

Collateral Damage

Before the invention of the FICO score, consumers relied entirely on Bankers to approve or deny credit requests, often based on personal characteristics and activity rather than one’s ability to repay a loan. Earl Isaac and Bill Fair created a model, the FICO model, which would evaluate an individual’s risk of defaulting on a loan based solely on his/her finances. In addition to the FICO scoring system still used today, statistical data such as web browsing habits and purchasing patterns, is pulled together to provide insight on customers’ creditworthiness; and an e-score is generated. The problem with e-scores is that they are arbitrary, unregulated, and unfair. O’neil refers to them as the perfect WMDs. As companies are legally prohibited from using credit scores for marketing purposes, e-scores are used instead.

Many employers use credit scores in their determination of employment. A high credit score may indicate a person is more trustworthy and reliable, leading to more job opportunities. Using this model for employment determination can also have a negative effect on a person. If his/her credit score is low, he/she may be passed over for jobs. Without employment and income, he/she may be unable to pay bills, potentially leading to an even greater credit score drop.

Further concern with automated systems is the lack of human review; errors on credit reports aren’t considered or even noticed. Consumer profiles are sold to data brokers who profile people from different data sources, there is bound to be inaccuracies. Customers can be incorrectly profiled and have incorrect information in their files and can miss out of loans, jobs, housing, etc. due to the automation of the process.

While O’neil does not suggest we go back to the way banking was handled prior to the use of algorithms, she does recognize how e-scores are contaminating financing opportunities for those with lower credit scores.

Key Take-Aways:

• Predictive mathematical modeling includes readily available statistical information and is transparent. Dynamic modeling covers the use of memorizing, updating and adjusting your model based on feedback received.

• Level of Service Inventory-Revised (LSI-R) is one of the more popular models. This model gathers data from those convicted to asses their level of risk for recidivism. This involves hidden modeling and is not transparent.

• O’neil summarizes the three elements of a WMD as: Opacity, Scale, and Damage; and argues that all elements are included to a certain degree in the examples that are covered in the book Weapons of Math Destruction. They are ultimately unfair and universally damaging.

• Big Data and predictive programs are intended to help identify areas of high crime and assist police departments in addressing these crimes. However, historical data shows us that impoverished neighborhoods are targeted more often than other neighborhoods, leading to more arrests in these neighborhoods, thereby producing unfair and biased results.

• While automated data collection used for consumer profiling and customer creditworthiness has expanded greatly, it is not a perfect system and does not provide equal opportunity for everyone.

Discussion Questions:

• The Level of Service Inventory-Revised (LSI-R) model qualifies as a WMD while lacking fairness and transparency; this begs the question of how can this model be fair to anyone?

• “Innocent people surrounded by criminals get treated badly, and criminals surrounded by law-abiding public get a pass.” How does this cycle break when consindering data generated from these models correlates poverty and criminal behavior, and often times misses wealthy, white collar crimes?

• Where should fine-tuning the algorithms used in machine apps begin? Knowing there is inevitably inaccurate data, how do we get the statistical data to the best point possible?

References