5th August, 2026
5 min read
High Stakes Design
Most IT roles now come with a pre-requisite: do you use AI in your daily routine? AI adopters already have it streamlined in their processes, ranging from using it as a search engine to building deliverable prototypes for upcoming product releases - but is it actually safe to do so?
AI heavily impacts product design. It’s changing how designers work day-to-day, as prompting supplements prototyping and traditional market research, which in turn reduces timelines so companies can ship products faster, and possibly cheaper. However, when designing applications and software that influence their health, rights, or money, the scenario requires oversight. It is often easy to forget that any data an employee inputs in a chatbot carries legal liability, even if the company has acquired an AI license. This can carry real risk for the employee, their company, or the AI developer.
Courts treat AI systems as tools, and the operator of the tool is responsible for how they use it. [1] If a doctor recommends an AI’s hallucination as a diagnosis, then there is risk that follows along with it that affects not only the patient, but the doctor’s credibility. Similarly, when a product designer inputs sensitive data to create designs for a healthcare app, if a data breach occurs, they can be held liable as the operators in this case.
Why is AI not responsible - it gave me the recommendation?
AI models are inherently black boxes. It is difficult to understand why a model provides a recommendation; however, tools exist that provide some explanation- though they have real limitations. [4]
In a court of law, this lack of transparency introduces a real block in litigation; defendants can argue that harm caused by AI cannot be deemed “reasonably foreseeable”, as it is difficult to pinpoint why an AI gave a specific recommendation. Even if the liability gaps were identified, it may be difficult to differentiate if the instance is an AI problem or unique to a specific AI. [3]
From a legal standpoint, AI does not qualify as a “legal person”. Legal personhood [1] can be defined as an entity that can hold rights, property, and/or obligations, enter contracts, and face legal consequences for any harm they cause. As of 2026, AI has no legal personhood under any jurisdiction worldwide; it cannot be sued or penalized, as it lacks intentions, awareness, or legal standing.
If an AI hallucinates, provides false information, misrepresents data, or misinforms, the liability currently falls on the legal entities involved: the developers who made the AI, the company currently using it, and the employee who used the misinformation to cause potential harm to another person or entity.[2]
Should we give legal rights to AI?
Several discussions weigh the pros and cons of granting AI legal rights. The strongest contender for granting it is rooted in AI’s growing autonomy. [6] Modern AI systems can autonomously make decisions, provide recommendations, train older models, and are even being refined to govern base models.
As we move towards AI sentience, the question arises: will recommendations provided by AI be based on its model training and company policies, or will it have gained self-awareness? In such a case, should AI hold part of accountability alongside humans?
To mitigate such risks, the EU's AI Act in 2017 proposed the concept of introducing “Electronic persons”, with specific liabilities as quoted:
“...Creating a specific legal status for robots in the long run, so that at least the most sophisticated autonomous robots could be established as having the status of electronic persons responsible for making good any damage they may cause, and possibly applying electronic personality to cases where robots make autonomous decisions or otherwise interact with third parties independently;.” [7]
Discussions about granting AI legal rights are yet to cover the scope of AI’s impact on society and the economy, in systems, platforms and institutions, as well as biases introduced during training along with its inequitable power dynamics. [6]
What does this mean for design?
If you are creating AI products within categories that range from Healthcare, Law, Finance, Education, Biometrics, Cybersecurity, and Public safety, your product may fall under the “high-risk” category defined by the EU’s AI Act [3].
There are specific guidelines under this act that safeguard public interest, which highlight the fundamentals required for a product to enter the market: transparency, risk assessment and mitigation, traceability and human oversight, to name a few.
In current AI design/code tools, you can upload whole design systems, brainstorm as you prompt, and fetch MCP APIs all within select AI tools, which deliver ready-to-ship products within weeks instead of months. There are several legal IP at play here:
The design system uploaded to the AI’s directory
UX Artifacts such as personas, heuristic analyses, journey maps, wireframes
Organizational Language and Jargon
Research and brainstorming data - competitive, market research, internal research
Third-party assets - stock photos, icon sets, fonts, etc.
Proprietary design system components and tokens
Internal process documentation
Potential trade secrets
As an employee, any sensitive data not pre-approved by your company for AI use and exploration may be considered a breach of employment confidentiality clauses and implicate the employee for sharing trade secrets, regardless of whether AI did anything wrong. In such a case, the employee may be held liable as the “operator” responsible for use of AI, if uploaded data becomes exposed by way of prompt injection, data leakage, or model-training exfiltration.
AI is already narrowing which roles get to design - junior designers are the highest impacted because it's easier and faster to train a model as opposed to a person. That logic flips when it comes to designing for high-stakes; it comes with an added layer of accountability and direct impact on human lives, making speed the wrong thing to optimize for.
Until there are laws that help guide humans in dividing or sharing accountability with AI as it becomes more autonomous, we need humans to supervise and actively own risk mitigation when delivering products that impact lives at scale - a responsibility that grows more urgent now that AI is an essential aspect of the software development life cycle.
References:
WCR.LEGAL, "Can AI Be Legally Liable for False Statements" — https://wcr.legal/ai-liability-false-statements/
UK Law Commission, "AI and the Law" (discussion paper) — lawcom.gov.uk https://cdn.websitebuilder.service.justice.gov.uk/uploads/sites/54/2025/07/AI-paper-PDF.pdf
European Commission, "AI Act" https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai#1720699867912-2
Rathor, H. (2025, November 12). AI Can Explain - But To What End? | The Promise and Pitfalls of Explainable AI through LIME. Heena Talks Design. https://heenarathor.framer.website/blog/ai-can-explain---but-to-what-end-the-promise-and-pitfalls-of-explainable-ai-through-lime
"No legal personhood for AI" — https://pmc.ncbi.nlm.nih.gov/articles/PMC10682746/
Eduwik, "Should AI Be Granted Legal Personhood?" — https://eduwik.com/should-ai-be-granted-legal-personhood/
European Parliament Resolution 2017/2103(INL), Civil Law Rules on Robotics — europarl.europa.eu [Section Liability Principle 59] https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=oj:JOC_2018_252_R_0026
