Introduction

Not so long ago, systems relying on machine learning (ML) were tailored to specific tasks, requiring experts in the field to set them up, tune and deploy them, and ultimately provide guidance to users. Nowadays, with deep learning ever on the rise and large general-purpose models readily available, this expert guidance is no longer needed. However, the wide availability and applicability of these models highlights several questions that have always been present, but perhaps not as obvious. Are these models fair to everyone? Do they abide by ethical principles? What makes an ML model fair in the first place? How can we make them so? Let’s try answering some of these questions and shed some light on the ethics in ML, because that is mostly what we are talking about when we mention ethics in the context of artificial intelligence (AI). We will try to give examples of common issues and the steps that can be taken to mitigate them.

What is ethics in AI?

According to the Cambridge dictionary, ethics represents a set of moral principles that govern a person’s behaviour or the conducting of an activity [1]. So, as ML models are used to make decisions, what are they really guided by? As it turns out, by the principles (and biases) ingrained in the acquired data, in the annotations, in the algorithms and even in the users’ interactions, which in certain cases might lead to systemic errors. In essence, everyone involved in a project with an ML related component, from the data annotators to the CEOs, leaves a footprint on the behaviour and outcomes of the system. Therefore, integrating ethical principles into ML products must be a responsibility shared by all the aforementioned. It should also be considered at every stage of the development process. In the context of ethics in AI, principles such as fairness, data responsibility, transparency, and accountability are outlined as the key ones that should govern its use [2].

Main issues

As given in [4], the most prominent potential issues include bias and discrimination against a particular group, denial of rights or autonomy, invasions of privacy, unexplainable and unjustified outcomes, while even unreliable or harmful ones are not unheard of. Let us then consider these categories with some examples to better illustrate the issues.

Explainability

Decisions made by ML systems have a major impact on human lives. The issue is that most of today’s systems do not disclose how they arrive at their decisions [5]. Some by design, as they are black-box models, others by choice of their creators. If the decisions cannot be explained, why should they be trusted? Furthermore, the decisions cannot be challenged by those afflicted. A student who attained the highest grade for an essay was dubbed a cheater by an automated, ML based, plagiarism detection tool developed by TurnItIn [4]. The system reported the essay as being written by a large language model and, unlike the case with plagiarism where original content is cited in the reports, here the teachers have to take the systems word for it. The solution for this case came down to trust between the teacher and the student, but might have, in different circumstances, been disruptive to the student. The European Union has attempted to tackle this issue in the General Data Protection Regulation (GDPR) by enforcing a right to explanation and in the more recent AI Act by considering such systems as high risk.

Bias

The idea that decision making computer systems are unbiased is widespread, but falls apart under deeper scrutiny. In order to make a decision, one group of values must be rated higher than another. This makes biases stemming from different sources, such as the developers’ beliefs or societal relationships, which are not easily discernible, seep into it. The main sources of bias are data, societal norms, model/algorithm and user interaction.

Data biases occur when the data used to train the ML model is incomplete or does not reflect the entirety of the problem due to the under- or over-representation of certain concepts or populations. For example, a system called Correctional Offender Management Profiling for Alternative Sanctions is used to predict the recidivism likelihood of a person which committed a crime. Black people were found to receive significantly higher risk scores [6]. 

Societal bias occurs when the prejudices that society has had throughout history are learned by the model and is also referred to as historical bias. Throughout history, different social groups (ethnic minorities, women) have been discriminated against and should such data be used to train an ML model, these biases become ingrained in it. Amazon, for example, has developed an ML based tool for job application screening to improve and automate the hiring process. Applicants were rated on a scale of 1-5. However, it turned out that it incorporated bias against women. You see, the tool was trained on resumes submitted to the company over a 10-year period primarily for technical positions, most of which came from male applicants, making the system penalize resumes that contained words such as “woman” or applicants coming from women-only universities [7].

Algorithmic biases arise from algorithms utilised to make decisions which either create rules for inherent biases due to training data or accentuate inherent issues with the data or the problem at hand. Examples include models built on biased assumptions, a single model created for different populations or models that discern proxies for a sensitive feature, such as age, sex etc. For example, an algorithm used to assess the risk factor of chronically ill people in order to provide them with pre-emptive care to achieve better outcomes and reduce costs was found to be biased. It used the past healthcare costs as a proxy measure of health status and medical needs. Although both black and white people had similar scores for comparable spending, for black people the money was spent on more significant interventions. This meant that severely chronically ill black people were less likely to receive pre-emptive care [8].

User bias arises from users interacting with the system and introducing their own biases into it. This is the case with popular social networks as they feed the users content that is similar to what they have previously consumed. A conspiracy theorist would be more likely to look at content denying climate change or claiming the earth is flat, etc., which causes the content promotion algorithm to suggest similar content, thereby creating an echo chamber.

Privacy

Every service and device connected to the Internet today is used to collect data about the user and their interaction with the service. Although EULAs detail data usage, outlining where, how and when the company collecting the data can use it, most users skim or skip over this content and agree to the terms. The issue is that data from multiple sources is aggregated, so that data containing information about your location, sleeping patterns, heart rate, spending habits, social interactions etc. can be linked to an identifier and used for anything the company sees fit, from serving “relevant” ads to curating content. As Hedlund from Lund University notes, a privacy paradox is being reached, where users are revealing personal information to not be excluded and simultaneously worrying about giving away their privacy. A good example of this was when the US company Clearview AI was accused of violating Canadian laws as it allowed law enforcement agencies and companies to match images against its database. The database contained several billion images, including those of Canadian adults and even children. The authorities considered this conduct to be monetized mass surveillance. They concluded that the facial recognition was carried out without people’s consent. However, the company claimed it was fair use [9]. Another example is the recall of Microsoft’s Copilot assisted computer programming tool due to a feature that allowed screenshots to be taken of the developer’s screen at regular intervals.  

Data ownership and IP

As already mentioned, we generate a lot of data, partly just by using the services available to us, and partly by publishing content on the Internet. In turn, modern ML algorithms require a lot of data to be useful (e.g. Meta’s LLaMA 3 large language model was pre-trained using 15 trillion publicly available data tokens, while Stable Diffusion 3 Medium was pre-trained by Stability AI on 1 billion images). This is a perfect storm. Two issues arise from this, firstly the issue of ownership and intellectual property rights (e.g. the rights to artwork used to train generative models) and secondly the ability to remove and delete data once ingested (as it is almost impossible to “untrain” the model, i.e. remove information once it has been encoded). Stability AI has been sued several times, most notably by Getty Images, a major provider of photography content, for the unauthorized use of images in the training data [10].

Accountability

The concept of accountability in AI is to build and use these systems in such a way that it is easy to assign responsibility for bad outcomes [11]. The guilty party should not be able to defer the responsibility to the ghost in the machine or wash their hands because of the black-box nature of these algorithms. To achieve accountability, systems must provide traceability throughout the design, implementation and deployment phases. A common example is autonomous driving. If such a vehicle is involved in an accident resulting in major damage or loss of life, who is to blame? The possibilities are many, the company that deployed it, the engineers who built it, the data collection team etc. A prominent case involving such a vehicle occurred in 2018 when a self-driving car with a control driver killed a pedestrian [12]. Although the control driver was found guilty, the system recognized the pedestrian but failed to react. Tesla has also been involved in several car accidents which occurred when their full self-driving car software was employed, and the driver was not paying attention to the road [13]. The software in use today has terms and conditions which state that the driver must pay attention to the road, but what happens when incidents occur with fully autonomous systems?

Shadow AI

So far, most of the focus has been on developing and deploying ML based systems with ethical principles in mind. But even the best of them can be misused. The term “shadow AI” has only recently been coined to describe the problem of using unauthorized AI tools in the workplace. Normally, they enable faster task completion, experimentation and automation. However, the lack of oversight and governance could lead to them being used against ethical guidelines and cause some of the issues described above. Examples of this include the leaking of private or sensitive information such as intellectual property into the model. This happened at Samsung, where employees pasted proprietary source code they were working on into a large language model service [14]. Such cases make even large companies such as Apple, Deutsche Bank, JPMorgan Chase and others to outright ban the use of services that rely on large language models that they themselves do not own and operate.

What can be done?

So, what can you do as a start-up, company or individual trying to develop a product which relies on ML? First, consider how you build your team and try to hire experts from different backgrounds. This includes people from different cultures, religions as well as fields of expertise [15]. The different opinions should aid in curtailing discrimination or at the very least react to it in the very early stages. In addition, the data which is stored and processed should be kept to a minimum needed for a viable product. The data collected should be reviewed by committees, and sensitive data removed while being mindful of proxy variables. A monitoring system should be devised to track accuracy, especially when deploying the solution in new environments, and to ensure result traceability. Users should be provided with safeguards such as human operators (who could take control and intervene if the decisions become unfavourable) and the possibility of opting-out of automated processing. If possible, the algorithm and data should be made available via open-source repositories and, where possible, explainable algorithms should be employed. This goes one step further for application areas where explainability is key, such as medicine or warfare. Finally, responsibilities should be divided within the team and risk management policies put in place to minimise risk.

Governance should stem from these guidelines provided either at the international or national level, while also incorporating the guidelines from non-governmental organisations of experts and even companies (e.g. IEEE, ACM, Google amongst others). There are currently several regulations on data protection. In the EU, the General Data Protection Regulation gives users the right to be forgotten. Similar regulations exist in some US states, while countries such as India, Canda or China have their own legislation to counter some of the issues mentioned in this post. A detailed list of legislation related to AI and the regulators can be found in [16]. The biggest change, however, to regulating AI comes from the EU where the AI Act [17] (Regulation (EU) 2024/1689) represents the first ever attempt to form a legal framework to mitigate the risks of AI and provide people with trustworthy AI. It came into force on August 1 and will be fully applicable in two years. The key point is the classification of projects and products that utilise AI into four risk categories, from unacceptable to minimal. Each category then has its own regulatory level. The products that fall into the “unacceptable” category will be banned completely (e.g. classifying people according to their socio-economic status). Products in the “high-risk” category (e.g. AI in critical areas such as health, transport and education) will be subject to assessment and must have risk management, monitoring and traceability, utilise highest quality data, be tested in a “real-life” scenario before release, and always have a human supervisor. Products in the “limited risk” category (deep fakes, chatbots) must comply with transparency rules (e.g. full disclosure if the content was AI generated), prevent illegal content, disclose summaries of training data etc. The category with the minimal risk (wherein most products available today fall into) will not require additional regulation. It should be noted that any major change to the product or service would also trigger an assessment procedure. The voluntary commitment from industry members is on the rise, as can be seen from the signees of the AI Pact [18], which encourages the implementation of AI Act measures ahead of time.

Conclusion

It is abundantly clear that AI is developing at a galloping speed, where even blinking for too long would make you lose track. And while this pace is great for bringing to life new ideas, automating tedious processes and tasks or creating new products, – the impact on lives of the people affected by this technology is somewhat neglected in lieu of profit. There is enough research and anecdotal evidence, some of it presented in this article, to suggest that we really need to rethink this view, and that ethical principles should be incorporated into every step of system design, development and deployment. While many sound guidelines are readily available, it is up to the individual market player to decide whether to adhere to them. The recent step taken by the EU is therefore interesting, as it essentially codifies these guidelines into regulations. The AI act is causing an uproar from both sides: Investors claim it will hinder development and drive-up cost while preventing the EU from participating in the AI race, whereas scientists and human rights groups see it as a welcome tool to combat the current behaviour of big tech companies. The truth lies somewhere in between. As with the GDPR, this law will not be a golden hammer, but it could very well help with this particular nail. Let’s just hope our thumbs remain intact.

If you want to learn more about this topic, I highly suggest checking out the “Artificial Intelligence: Ethics & Societal Challenges” or “Ethics of Artificial Intelligence” courses offered on e-learning platforms.

Literature

1. Cambridge Dictionary, Ethic, Cambridge University Press & Assessment,  https://dictionary.cambridge.org/dictionary/english/ethic
2. IBM, Ai Ethics, IBM, https://www.ibm.com/topics/ai-ethics
3. D. Leslie, Understanding artificial intelligence ethics and safety: A guide for the responsible design and implementation of AI systems in the public sector, The Alan Turing Institute, https://doi.org/10.5281/zenodo.3240529, 2019
4. G. A. Fowler, We tested a new ChatGPT-detector for teachers. It flagged an innocent student., Washington Post,  https://www.washingtonpost.com/technology/2023/04/01/chatgpt-cheating-detection-turnitin/
5. A. Adadi, M. Berrada, Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI), IEEE Access, vol. 6, pp. 52138-52160
6. J. Larson, S. Mattu, L. Kirchner, J. Angwin, How We Analyzed the COMPAS Recidivism Algorithm, Propublica, https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm
7. J. Dastin, Insight – Amazon scraps secret AI recruiting tool that showed bias against women, Reuters, https://www.reuters.com/article/us-amazon-com-jobs-automation-insight-idUSKCN1MK08G/
8. S. Vartan, Racial Bias Found in a Major Health Care Risk Algorithm, Scientific American, https://www.scientificamerican.com/article/racial-bias-found-in-a-major-health-care-risk-algorithm/
9. E. Thompson, U.S. technology company Clearview AI violated Canadian privacy law: report, CBC, https://www.cbc.ca/news/politics/technology-clearview-facial-recognition-1.5899008
10. TFL, From ChatGPT to deepfake creating apps – a running list of key AI lawsuits, TheFashionLaw, https://www.thefashionlaw.com/from-chatgpt-to-deepfake-creating-apps-a-running-list-of-key-ai-lawsuits/
11. Carnegie council, AI accountability, Carnegie council, https://www.carnegiecouncil.org/explore-engage/key-terms/ai-accountability
12. L. Smiley, The Legal Saga of Uber’s Fatal Self-Driving Car Crash Is Over, Wired, https://www.wired.com/story/ubers-fatal-self-driving-car-crash-saga-over-operator-avoids-prison/
13. Reuters, Tesla car that killed Seattle motorcyclist was in ‘Full Self-Driving’ mode, police say CNN, https://edition.cnn.com/2024/07/31/tech/tesla-full-self-driving-mode-seattle-motorcyclist-killed/index.html
14. Emily Dreibelbis, Samsung software engineers busted for pasting proprietary code into ChatGPT, PC mag, https://www.pcmag.com/news/samsung-software-engineers-busted-for-pasting-proprietary-code-into-chatgpt
15. E. Ferrara, Fairness and Bias in Artificial Intelligence: A Brief Survey of Sources, Impacts, and Mitigation Strategies, Sci, Vol. 6, No. 3
16. White & Case, AI watch – global regulatory tracker, White & Case, https://www.whitecase.com/insight-our-thinking/ai-watch-global-regulatory-tracker-canada
17. European commission, AI act, European commission, https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai
18. European commission, AI Pact, European commission, https://digital-strategy.ec.europa.eu/en/policies/ai-pact

Author
Associate prof. Bruno Zorić, PhD