What is the challenge? When we reach important decisions, such as who will be granted parole, we want the grounds of that decision to be transparent. First, because our ability to justify an action is a central element of its legitimacy. Second, because we are more likely to trust a system where the grounds of decision are clear for everyone to see. While some machine-learning algorithms can provide the reasons behind their recommendation in a much more clear and detailed way than humans, others are less interpretable (e.g. neural networks). Over and above generating reliable predictions, we strive to build systems that are interpretable.

How do we approach it? First, we speak to clients to understand their standards of interpretability. For some prediction problems, the standards are higher than for others. Second, we select a model that has an appropriate level of interpretability, and openly discuss whether there is a trade-off in terms of predictive power. While we generally prefer interpretable models with few inputs, a substantial increase in predictive power sometimes justifies lower interpretability. Third, we build features that actively increase interpretability, such as the list of features that led the model to make a prediction (this can be behavioral or demographic data for a beneficiary). 

What is the challenge? For any decision we take, we risk treating individuals or groups inequitably. For example, insurances will ask for higher premiums from young men, because their accident risks are higher. Sometimes, unequal treatment is justified, but sometimes, it reproduces historical injustices or places an unfair burden on certain groups in society. Predictive models inform decisions and thus face the same challenges: while they have the potential to reduce human bias and make decisions purely based on statistics, their recommendations can end up placing a high burden on individuals with risk characteristics, and sometimes the data used to train the algorithms reflects past injustices (e.g. higher levels of crimes in communities that were historically overpoliced).

How do we approach it? First, we engage in a discussion on fairness principles with our clients. Our employees are trained in philosophy and social sciences, and are familiar with discussions around ethics and social dynamics. Because data science is agnostic with respect to ethical norms, it is important to us to understand the values of our clients. Second, we evaluate the dataset used to train algorithms to understand potential sources of bias. Third, we consider options for building in ethical values into the design of predictive models. For example, models can be built to actively ignore gender, in order to create more equal outcomes for both genders. This involves honest discussions about limitations this imposes on predictive power. Third, we test what the implications of the models recommendations are: for example, what recommendations does the model generate for subgroups, and how accurate is the model for them.

What is the challenge? Predictive models learn from data that is often personal and sensitive, such as school records or health data. In addition to challenges regarding cybersecurity, there are often regulatory requirements for working with sensitive data that regulate legitimate uses of data and consent requirements. Our work is bound by three codes: our client’s internal rules on data security, regulatory requirements, and our own standards

How do we approach it? First, we minimize the use of sensitive data where possible. Second, we ensure that the owners of data (sometimes the beneficiaries themselves) have consented, where such consent is mandated. This sometimes means asking challenging questions to our clients. Third, we always encrypt data, whether at rest or in transit. Fourth, we insist on anonymisation of data (removing unique identifiers that can be traced to individuals and removing or altering data fields that lead back to individuals (e.g. precise location data).

What is the challenge? Machine-learning models make recommendations based on complex and multidimensional calculations. In some instances, those recommendations can directly be translated into decisions. Often, it is desirable to include an element of human decision-making, both as an extra layer of safety (particularly in the early stages), and to ensure that there are accountable individuals behind decisions

How do we approach it? First, we discuss our clients needs with respect to control for the particular context. Some decisions can be automated without the need for a layer of human control (e.g. predicting machine maintenance needs) while others are more sensitive (e.g. predicting whether a student will drop out of their program). Second, we identify the critical joints in operations where human decision-making is required. Third, we build features into our models that ensure automatic checks and balances (e.g. by requiring explicit permission of humans before implementing a recommendation).