The age of the self-driving bureaucracy? ChatGPT and the Public Sector

There has been a lot of recent media attention around ChatGPT, the chatbot launched by OpenAI in November 2022 that uses artificial intelligence (AI) techniques to generate human-like text responses to questions or prompts put to it by users.

Just last month, ChatGPT reportedly passed a US law school exam at the University of Minnesota, all three parts of the United States Medical Licensing Exam, and an MBA exam at Ivy League school the Wharton School of Business. It has even been used to draft a Bill regulating generative AI in Massachusetts.

For time-poor public sector workers returning from their summer holidays, it would be fair to wonder: “If ChatGPT can pass an Ivy League exam, then perhaps it can help me finish off that work product that is due next week!” And given the proliferation of news articles on ChatGPT and its use cases, senior managers in government may start to worry that their staff might start using ChatGPT to prepare Question Time briefs, or fret over whether the glossy 300-page report prepared by a consulting firm might have been written (at least in part) by an AI tool.

There is no doubt that generative AI tools like ChatGPT will change the way that governments carry out their work and the way that the bureaucracy functions, stirring excitement in some and fear in others. Indeed, David Thodey's 2019 independent review of the Australian Public Service, Our Public Service Future, identified that about 40% of APS employee time is spent on "highly automatable tasks".

As with all new technology, it is appropriate to exercise some caution. For government agencies and departments thinking about using (or at least experimenting with) generative AI tools like ChatGPT in connection with their work, it is critical to understand the legal risks and challenges of doing so.

How does ChatGPT work?

ChatGPT is a new breed of generative AI technology that uses a large language model and deep learning techniques to generate text outputs in response to prompts. Generative AI tools are trained on a huge corpus of text (such as books, articles and websites) to learn the patterns existing between words. Once a generative AI language tool has learnt enough, it can use its knowledge to create new text and content, answer questions, translate languages, and summarise text. Large language models also offer the potential for users to fine-tune the language model on new data specific to a particular task or subject matter to achieve more refined or tailored outputs.

The popularity of ChatGPT likely arises from its uncanny ability to mimic human language in the output it produces and to distil and respond to complex questions in an intelligent sounding manner (well, at least, seemingly so).

ChatGPT and the public sector

Generative AI tools like ChatGPT have enormous potential for the public sector. They can save time and resources. They also have the potential to enable governments to deliver better access to services. Conceivable use cases for government include (among others):

• automating and personalising customer support (via improved chatbots and virtual assistants);
• providing improved and automated translation services for non-English speakers and improved accessibility for the vision and hearing impaired;
• producing materials that better inform the population about what government is doing (such as on websites, and in social media and marketing content);
• improving written products by editing text for phrasing, spelling, punctuation and grammar;
• summarising legislation, policy papers and reports;
• preparing aspects of Ministerial speeches and correspondence; and
• developing computer code written by in-house development teams.

What legal risks arise from the use of ChatGPT for the public sector?

For government agencies and departments that are seeking to use or experiment with ChatGPT, there are some key legal risks that need to be considered.

1. The special context of government heightens the consequences of inaccurate, biased or inappropriate outputs

Tools like ChatGPT can make mistakes. This can be for a range of reasons – errors in the training data, limited training, lack of context, "hallucinations" (where the model just imagines responses) and where abstract reasoning is involved (eg. ChatGPT can be poor at maths).

If the inputs into ChatGPT contain biases in any way (for example, contain a disproportionate amount of information about a particular gender or race), then the output ChatGPT produces has the potential to reflect those biases. As a result, ChatGPT could operate unfairly and build bias into the way that services are delivered by governments, such as making services more accessible or available to one section of the community over another. Further, because generative AI tools can be set up so they are always learning, the outputs an AI bot produces may change over time, leading to different answers being given to the same question.

Governments generally have a duty to exercise reasonable care and diligence when providing information to the public to ensure that the information communicated is accurate. Additionally, governments can have exposure to legal liability for inaccurate or misleading information under negligence, via a breach of contractual warranties, and under consumer laws (where applicable to government).

For citizens accessing government services, it is crucial that they can trust that they are receiving accurate, impartial, and consistent information. This trust in government is imperative for a functioning and stable democracy. As aptly stated in the Australian Public Service Values and Code of Conduct in Practice: “trust in government and a nation’s democratic institutions is significantly influenced by the experience of… the public as clients of the public service”.

Mitigating the risks

Government agencies and departments seeking to use generative AI tools will need to understand the risks posed by the tool, fully understand the nature, reliability and consistency of the output, and also consider the fairness and human impact of the tool’s use. To support this, government agencies and departments should (to the extent not already in place) create or update AI governance policies to set clear parameters and controls about the use and application of AI tools, including in government projects.

An AI governance policy could incorporate a risk management framework that helps to determine the level of risk of using AI generative tools for public sector projects, as well as a compliance and review mechanism. The New South Wales Government's 2022 AI Assurance Framework is a useful resource in this respect, providing guidance on the safe, ethical and transparent use of AI and the risks of applying AI to particular use cases.

Additionally, AI tools should be continually tested and verified by humans. The outputs of ChatGPT and other AI systems should not be taken for granted.

2. Intellectual property infringement risks need to be managed

As with a range of emerging technologies, ChatGPT raises numerous challenges from an intellectual property perspective, including how copyright law concepts such as authorship and infringement interact with ChatGPT, which you can learn more about in our Generative AI vodcast.

The output of ChatGPT (which is not a human author) is in most cases unlikely to be protected by copyright or other intellectual property laws in Australia (although the position varies in other countries). ChatGPT raises the potential for intellectual property infringement given that works that are protected by copyright were used to train the model. An infringement could occur if a user causes ChatGPT to produce an output that reproduces a part of an existing work the subject of copyright (either intentionally – by asking it to do so – or inadvertently). Also, using third party data to fine-tune a model like ChatGPT could infringe rights in that data.

Mitigating the risks

Governments using ChatGPT for work purposes, or managing projects where service providers might use ChatGPT, can mitigate intellectual property risks by ensuring that:

• they are authorised (or can rely on an exception like fair dealing for research or study) to use any content prompting ChatGPT or in training data used for fine-tuning a model;
• any ChatGPT outputs used by government do not infringe a third party’s intellectual property rights (though appropriate policy, audit and technology measures) or trigger crown use obligations;
• government considers whether it is better not to use generative AI where it's critical to own IP in content;
• they do not falsely claim ownership of IP in AI generated works; and
• the government’s contracts with content generators (such as report writers and consultants) include the standard warranties that deliverables created under the contract will not infringe a third party’s intellectual property rights.

3. Incorporating AI tools into administrative decision-making requires considerable caution

In looking for ways to make administrative decision-making more efficient, there may be appeal in the idea that a generative AI tool could be directed to make a decision, or even part of a decision, in a fraction of the time that a human might take. However, without being able to expose the reasoning process applied by a tool like ChatGPT or, at a deeper level, the data from which those reasons were drawn, it is unlikely that generative AI could replace human decision makers any time soon.

Fundamental administrative law concepts such as affording procedural fairness, acting within the scope of decision-making power and only taking into account relevant considerations present a challenge to the use of a tool like ChatGPT, which gives answers that sound convincing, but with limited detail about how it came to them. As outlined above, there is a real risk that generative AI tools could be trained on data that has a particular bias which is not easily able to be identified or otherwise addressed. OpenAI, the developer of ChatGPT states that it has attempted to reduce the risk of bias, but that it may still produce biased content. Whether a generative AI tool could ever comprehend (and then limit itself to) the scope and basis of a decision-making power and all the case law, regulations and directions that guide its use is an open question. Identifying relevant and irrelevant considerations and only taking account of the former in making a decision is a task that human decision makers can find difficult. Training a generative AI tool to perform this task (and being able to review and test the resulting decisions) to the standard required by administrative law is a complex problem that will likely take some time to solve.

Further, not all decisions are legally able to be made by a computer. While it is increasingly common to see provisions in legislation that allow a decision-maker to "arrange for use of a computer program to make decisions", this power is not yet ubiquitous to administrative decision-making. All of these challenges (among others) mean that a tool like ChatGPT is unlikely to be able to produce a clear, defensible administrative decision that complies with the fundamental requirements of administrative law.

Mitigating the risks

Where a generative AI tool might potentially be useful is in its capacity to summarise information from a particular document or data to assist a human in the decision-making process. However, there should continue to be human checks and balances in place, including to ensure that the summary produced is correct and does not infect the decision with error. While ChatGPT can mimic human language and sound convincing in the answers it gives, this is not an indication of the accuracy of those answers. Further, manually checking any summaries generated by a generative AI tool against the source material may ultimately take as much time as was saved by using the tool in the first place.

For these and other reasons, the risks of generative AI tools – at least in their current form – producing inaccurate administrative decisions that are inconsistent with established principles of law mean that extreme caution should be exercised when considering whether to use or incorporate AI tools in any aspect of the administrative decision-making process. While AI technology may get to the point where it is sophisticated enough to replace a human decision-maker, it is not there yet. Before using generative AI, legal advice should be sought to ensure that relevant legislative requirements and administrative law principles will be adhered to.

4. The government’s privacy, security and confidentiality obligations must be observed

AI presents heightened privacy and data security risks because it uses large volumes of data for its deep learning algorithmic models that generate insights and outputs. If that data is drawn from publicly available resources (as is the case with ChatGPT), the data will likely contain personal information (and potentially significant amounts of it). As such, before ChatGPT is used by government, government users must ensure that the use of ChatGPT will not give rise to privacy harms or result in privacy law breaches in connection with personal information produced in an output.

The use of ChatGPT raises a number of challenges from a privacy law compliance perspective. Once personal information is inputted into the ChatGPT system in a prompt, control is lost over how that information is used, stored and protected. There is even the potential risk that personal information fed into generative AI tools will turn up in an output created for another user. The current terms of use that apply to ChatGPT make the user and not OpenAI responsible for providing privacy notices and obtaining consents in connection with any personal information processed using ChatGPT. There are also no current direct contractual obligations imposed on OpenAI in relation to how personal information is secured, protected and controlled. Further, there are no indemnities or other contractual avenues of redress that governments could rely on in the event of a privacy breach by OpenAI (although the extraterritorial reach of the Privacy Act 1988 (Cth) may facilitate enforcement action in response to a privacy breach by OpenAI).

Mitigating the risks

As a result, we do not recommend that governments input any personal information into ChatGPT because it will be difficult for government agencies and departments to meet their obligations under all relevant privacy laws and associated privacy principles. Rather, government agencies should ensure that data entered into ChatGPT is truly de-identified data and that there is no risk that any data could potentially reasonably identify a person (including, when used in combination with other datasets that may have been entered into ChatGPT).

Government agencies and departments should carefully assess the potential privacy impact of any proposal to incorporate ChatGPT into existing or new government systems or processes.

In addition to privacy considerations, governments should ensure that any use of ChatGPT does not breach their confidentiality or security obligations. No data inputted into ChatGPT should contain any confidential information or other sensitive government data. Any data provided to ChatGPT is expressly on a non-confidential basis.