AI Blindspots card set 2.0: development phase
The AI Blindspots cards are divided into three phases (planning, development and implementation). On this page, you can find the AI Blindspots of the development phase, the phase wherein you design, develop and test your data application or AI system.
Each AI Blindspots card contains:
- A set of questions to help you uncover this blindspot;
- A use case that illustrates the importance of considering the blindspot;
- A number of tools and tricks to help you detect and mitigate the blindspot.
Discrimination by proxy
You are not allowed to discriminate against people on the following data categories: gender, ethnicity, religion, race, .... Most organisations avoid this by not collecting this type of data or using this data in feature selection. But have you considered how proxy-data categories can lead to the same discrimination? Shoe size is for example a proxy for gender.
HAVE YOU CONSIDERED?
- Specific exceptions or practices in the context you are implementing your AI system?
- Inviting affected stakeholders to stress test your system against historical biases?
- Identifying and removing features that are correlated with vulnerable social groups?
HOW NOT TO
An AI system that predicts which patients would benefit from extra medical care prioritised healthier white patients instead of more at risk black patients. The algorithm was based upon how much a patient would cost to the healthcare system in the future, but did not consider that black patients spend less on medical care than white patients with the same chronic conditions.
TOOLS & TRICKS
- Involve domain experts
- Q 1 & 3: check for unintentional correlations that might impact vulnerable groups
- Q 2: contextmapping, workshop on participatory approaches to machine learning
- Q 3: Aequitas
Explainability
Why is explainability important? The predictions or recommendations generated by your AI system can be unclear and may be surprising. When creating an AI system, you have the responsibility to clearly inform users about the underlying technical logic of the system and how predictions or recommendations are generated.
HAVE YOU CONSIDERED?
- If people trust the choices made by your system?
- What the impact is of having an AI system generating a prediction versus a human?
- How you can interpret or explain the choices of your AI system?
HOW NOT TO
A medical authority in the US used an AI system to determine reimbursements for disabled people. However, the court stated that these reimbursements were not possible because the decisions of the AI system were not explained.
TOOLS & TRICKS
- Q 1 & 2: human-centered design methods
- Q 3: Lime, WhatIf
- Q 1, 2 & 3: explain your AI system to a random person and check if the results of your solution are clear and comprehensible, AI Explainability 360, Value Proposition Design
Performance balance
When determining an AI system’s metrics for success, trade-offs between optimal performance and negatively impacting vulnerable social groups must be made.
HAVE YOU CONSIDERED?
- If the chosen performance indicators will not stray the AI system from its original purpose?
- Which performance indicators are necessary and what the impact of these indicators will be on vulnerable social groups?
- How statistically accurate your AI system is?
HOW NOT TO
AI can help by screening for cancer. However, if it is optimized to detect all potential persons with cancer, this will result in a higher amount of false positives. These can cause unnecessary anxiety with those persons without cancer.
TOOLS & TRICKS
- Q 1 & 2: interviews with domain experts, IoT stress test (part of the Internet of Things Design Kit)
- Q 3: check your method for statistical accuracy, intermediate/prototype testing
Inclusion/ommission check
Your AI system might be beneficial to most people but have you considered how specific people might be worse off? Consider if your system is inclusive for economically vulnerable persons, people with lower digital literacy or people with a disability.
HAVE YOU CONSIDERED?
- How your system might exclude (vulnerable) people?
- How people might be digitally excluded with your system?
- How to minimize the number of affected people by your AI system?
HOW NOT TO
AI systems are often perceived as enablers for digital inclusion. They can for example detect atypical browsing behaviours and thus identify people’s difficulties when browsing the Internet. But what if things are the other way around and your AI system has a negative effect on (digitally) vulnerable people? How will you ensure your AI system is made for all and can be used by all?
TOOLS & TRICKS
- Q 1: inclusion by design tool of KCDM (coming soon)
- Q 2: 8 Profiles of Digital Inequalities: can all profiles make use of or benefit from your AI system?
- Q 3: involve UI designers, organize a co-creation workshop with targeted end-users
Dataset shift
A significant difference between your training datasets and testing datasets can result in what is called a ‘dataset shift’. This can heavily impact the performance of your algorithms.
HAVE YOU CONSIDERED?
- A systematic flaw in the data collection or labeling process that causes a nonuniform selection of training examples from a population and results in biases during the training of an AI system?
- If your data is (un)affected by shifts in time and location?
HOW NOT TO
If certain species are omitted in the training set of an image classification system for cats and dogs, the test set will reveal that not all images can be classified correctly.
TOOLS & TRICKS
Downloads
Below, you can find 2 downloads:
- A PDF of the AI Blindspot card set.
- A PDF with 2 templates to use the AI Blindspots card set. With the first template, you start from an ethical dilemma and use the AI Blindspots card set (workshop method 1 and 2). You can use the second template for the reversed brainstorm with the AI Blindspots card set (workshop method 4). A filled-in example of the templates is provided as an example. Visit the main page of the AI Blindspots card set for more information about the methods to use the card set.
