HAVE YOU CONSIDERED?

1. Did you clearly articulate the problem and outcome you are optimizing for?
2. Is this tool adequate to obtain this outcome?
3. Do all involved and affected stakeholders recognize this as an important problem?
4. Did you consider the advantages and disadvantages of your AI system for each stakeholder?
5. How will you guarantee to keep the state of purpose of your AI system?

HOW NOT TO

A company introduced an AI system to speed up their production process, but as an indirect result, employees lost their bonuses. How could this have been avoided? Take the trade union as an involved stakeholder in your project and find a way to increase the speed without losing the bonus.

TOOLS & TRICKS

• Q 1&2: problem definition template
• Q 2: course on machine learning (Google)
• Q 3: translate other applications of your machine learning to your case: does it still makes sense?
• Q 3: stakeholder mapping and validation

HAVE YOU CONSIDERED?

1. What is the minimal viable data collection you need according to domain experts?
2. Who/What might be excluded in your data?
3. How will limitations in your data impact the representative nature of your model and the actions your model supports?
4. If your data is unbalanced, can you mitigate this limitation?
5. Considering your data, can you describe the case or person where your predictions will be most unreliable?

HOW NOT TO

After the release of the massively popular Pokémon Go, several users noted that there were fewer Pokémon locations in primarily black neighborhoods. This came to be because the creators of the algorithms failed to provide a diverse training set, and didn’t spend any time in these neighbourhoods.

TOOLS & TRICKS

• Q 1: interview with domain expert
• Q 2, 3 & 4: Data Collection Bias Assessment, Aequitas
• Q 5: create a persona of the invisible man/woman

Questions with regard to data governance and the impact on the privacy of the data subjects whose personal data will be processed by the AI system, are all part of the preparation of your AI project. Determining the level of access to data and describing the flow of information will help you with protecting your data subject’s rights.

HAVE YOU CONSIDERED?

1. Can you lawfully process or reuse the data?
   1. If you reuse the data, is the purpose the same?
   2. Are appropriate contractual arrangements in place?
   3. Can you process or reuse the data on the basis of consent or other grounds?
2. Do you gather sensitive data or not?
3. Are there special regimes to protect your data?
4. Who will have access to the (collected) data? (internally and externally)
5. Can you comply with the data subject’s rights of the GDPR?

HOW NOT TO

A UK hospital together working together with Deepmind on a AI application detection and diagnosis of kidney injury was fined for violating the rules on personal data. It had transferred personal data on 1,6 million patients without their adequately informing them about this.

TOOLS & TRICKS

• Data Protection Impact Assessment
• Data flow mapping
• Engage with Data Privacy Specialists

Know your team’s unknown knowns. It is difficult to be aware of possible (ethical) issues if you are not aware of prejudice within your team. To avoid such blindspots, it is necessary to unveil them.

HAVE YOU CONSIDERED?

1. Did you consider bias in your team?
2. Is your team diverse and multidisciplinary or in touch with the problem area you try to solve?
3. Who you should invite to myth bust this wrong idea?

HOW NOT TO

Google’s photo-categorization software has at times mistaken black people for gorillas. The chances of this occurring would decrease drastically if black team members tested the service.

TOOLS & TRICKS

• Q 1: implicit association test
• Q 2: site visit, empathy map, persona, ...

You may be an expert in machine learning but not in the field you apply machine learning to. This is fine if you have an expert to tell you what to look out for in terms of typical outliers, hugely important variables or common practices that may impact your data.

HAVE YOU CONSIDERED?

1. Discussing with domain experts what the minimal viable data collection is that you need in order to allow your AI system to fulfill its purpose?
2. Using an expert to understand what the impact should be from your algorithm?
3. Which variables are essential for your problem?
4. An expert to help you assess the results of your algorithm?

HOW NOT TO

A new algorithm would help with diagnosing who needs to be assessed for pneumonia ASAP in the ER. According to the algorithm, people with asthma do not require immediare care. Experts did not agree with this estimation as asthma cases are treated with urgency in the ER. The experts stated that this was based on faulty assumptions by the AI system. According to the training data, asthma patients spent the least time in the ER. Therefore, the AI system deemed them to be unimportant for reaching efficiency in the ER.

TOOLS & TRICKS

• Interview or focus group with expert(s)
• Workshop on technical and systems requirements

You want to create an AI system to improve something in the world. However, if you only focus on the good it does, you may overlook the ways in which it might cause harm. It is always better to prevent than to cure. So consider what a truly malevolent party could do to or with your application.

HAVE YOU CONSIDERED?

1. How the AI system might be used unethically?
2. What the consequences would be if your AI system was used unethically?
3. Who you have involved to understand the underlying social motivations and threat models?
4. What your mitigation strategy is if your AI system is used unethically?
5. What to do if your algorithm develops unethical behaviour?
6. What are the key ethical principles that your AI system should exhibit?

HOW NOT TO

In 2016 Microsoft introduced Tay, a Twitter chatbot, to the world. Within 24 hours Tay was changed as she had learned to be a racist Twitter user based on the tweets addressed to her. Microsoft therefore decided to retire her.

TOOLS & TRICKS

• Creating scenarios to grasp the malicious and unethical practices of your system, and map the consequences of these scenarios on innocent bystander personas
• Involve experts from social sciences and law
• Thing-Centered Design
  

(Personal) data must be processed with respect for the privacy of the data subjects (see also card on data governance & privacy). But sometimes, a trade-off between personal privacy and public interest must be made: think for example of the contact tracing apps for the COVID-19 crisis. Even though the amount of personal data that needed to be shared was strictly limited, people still shared some personal data in order to trace corona infections. Furthermore, a weighing exercise sometimes needs to be made between the social interests/costs of implementing an AI system and the economic interests of commercial partners involved in this process.

HAVE YOU CONSIDERED?

1. What are the benefits and risks of the AI system on (1) an individual level, and (2) on a collective level?
2. Are personal data really required by the AI system in order to be able to serve the public interest?
3. How long will the personal data be stored?
4. Are the data subjects adequately informed about the purposes for which their personal data will be processed?
5. Which interests do the commercial partners have in developing/implementing this AI system? How do these relate to the social interest of providing good health care for all people?

HOW NOT TO

A new disease is developing fast and kills many people. Doctors have noticed that the disease pattern is different for the group of children with Down syndrome between the age of 4 and 6. They want to use an AI system to detect which characteristics of the children are strongly correlated with this different disease pattern because it might help to find a cure for the disease. However, the data of these children can only be pseudonymised, not anonymised, and there is no time to ask for consent.

TOOLS & TRICKS

• Q1-Q3: A proportionality exercise
  
• Q4: Clearly communicate the purposes of data processing
• Q5: Make an explicit agreement with all the involved partners about the goals of the AI system
  

A patient in need of care may not be fully convinced of the advantages of the AI system that will be used during his/her care process. Be aware that a person (patient, healthcare staff, …) might feel to have no other option than to give his/her consent in order to receive care or to continue his/her work. The power structure between patient and caregiver or between management and staff can play a role as well in the freedom of consent. And what with patients who are physically or mentally unable to give their consent?

HAVE YOU CONSIDERED?

1. What does ‘giving consent’ entail? Is a patient giving consent for the purpose of (1) optimising his/her diagnosis or treatment, and/or (2) further processing of his/her personal data (e.g. training other AI systems)?
2. How will you explain to the affected stakeholders what ‘giving consent’ entails? Are they sufficiently informed about what they are consenting to?
3. Can stakeholders withdraw their consent?
4. Will you review the way(s) you are asking for consent?
5. When collecting data on other grounds than consent, the collection process may be legal but not necessarily ethical. Did you consider to not collect, store and analyse data, although you may be legally allowed to do so?
   

HOW NOT TO

As the performance of AI systems is in everyone’s interest, the hospital assumes consent from the patients for the use of their data to train other systems.

TOOLS & TRICKS

• Richtsnoeren 05/2020 inzake toestemming overeenkomstig Verordening 2016/679, Advice on the use of data without consent (Nationale Raad van de Orde der Artsen), Commentary (to art. 20 Code van Medische Deontologie) about the consent to (the use of data for) a treatment (cf. art. 8 Wet Patiëntenrechten), Guidelines on people-centric approaches to notice, consent and disclosure, consult the ethical committee of your institution
• Q2: Try explaining the terms of consent to a five year old child, does he/she understand? (see also card on explainability & transparency)

Not all information can be quantified. One can for example think of mental health information or information regarding a person's perception of pain. This information is extremely interesting but harder to measure, to datafy and to standardise. When trying to datafy this type of information, a choice has to be made between the parameters that will/can be measured. Some parameters may be more easily measured than others because they are easier to datafy.

HAVE YOU CONSIDERED?

1. Are there parameters on which the AI system is based that are more difficult to measure and to datafy than others? If so, how will you deal with this?
2. Which proxies can you use to gather data about the parameters that cannot be identified? What are the limits/disadvantages of these proxies?
   

HOW NOT TO

In order to assess the risk of burn-out, a company's HR department may decide to analyse facial micro-expressions when entering the building. However, this does not take into account differences between individuals’ facial structure. As a consequence, there is a risk of arbitrarily estimating certain facial structures as an indication of a high risk, while they are actually not.

TOOLS & TRICKS

• A group discussion with different stakeholders (IT department, management, care professionals) on the calculability of parameters.

It is important that an AI system is fed with accurate and qualitative data to ensure that the results and outcomes can be interpreted in a correct and adequate way. This is especially the case when the system is for example used to improve care or well-being, such as examining which type of treatment will be the most appropriate for a patient.

HAVE YOU CONSIDERED?

1. Are the data upon which the AI system is based accurate and representative for the time, space and population in which the AI system will be used?
2. Are the data carefully entered/integrated?
3. Is there a test phase to examine whether the AI system predicts the right symptoms/treatment?
   1. Are the appropriate performance estimates used for this test phase?
   2. Are the performance estimates unbiased?
4. Are there control mechanisms to monitor the analysis and the outcomes of the AI system?
5. Is there a strategy to detect and counteract users who give false information on purpose?

HOW NOT TO

A hospital uses an AI system to analyse the course of pain complaints of fibromyalgia patients. It collects data via an app in which patients have to give an estimation of their pain experience three times a day. However, these estimations widely vary depending on the patient, and patients also often forget to submit the data.

TOOLS & TRICKS

• Q1-Q3: Involve domain experts
• Q2 - Q8: Involve IT staff, check the system on statistical accuracy, Data Collection Bias Assessment, intermediate/prototype testing