AI output depends entirely on its input, which can be the prompt it is fed, the dataset used for training, or the engineers who create and develop it. This can result in explicit and implicit bias, both unintentional and intentional.
To “train” the system, generative AI ingests enormous amounts of training data from across the internet. Using the internet as training data means generative AI can replicate the biases, stereotypes, and hate speech on the web. According to Statista, 52% of the information on the internet is in English, which means that bias is built into the system through training data. About 70% of people working in AI are male (World Economic Forum, 2023 Global Gender Gap Report), and the majority are white (Georgetown University, The US AI Workforce: Analyzing Current Supply and Growth, January 2024). As a result, there have been numerous cases of algorithmic bias, when algorithms make decisions that systematically disadvantage certain groups in generative AI systems.
While this does not mean that AI-generated content has no value, users should be aware of the possibility of bias influencing AI output.
There are ongoing privacy concerns and uncertainties about how AI systems harvest personal data from users. The user voluntarily gives some of this personal information, like phone numbers. However, users may not realize that the system is also harvesting information like the user’s IP address and activity while using the service (OpenAI privacy). This is an important consideration when using AI in an educational context, as some students may not feel comfortable having their personal information tracked and saved.
Additionally, OpenAI may share aggregated personal information with third parties to analyze ChatGPT usage. While this information is only shared in aggregate after being de-identified (i.e., stripped of data that could identify users), users should be aware that they no longer have control of their personal information after it is provided to a system like ChatGPT.