Visibility of system status and Match between System and real world [1] are two of the most important pillars in designing successful and usable products. People should have full transparency of system status and exit routes, and familiarity with real-life scenarios helps bridge the gap of service to software.
With the increase of AI integration in healthcare, it is crucial now more than ever, that AI safety protocols be put into question for applications that directly deal with human health, finance and justice. In this article, I will share some observable trends in AI-first Symptom checker app “ADA health” and “AI overview” by Google.

How ADA Health works

ADA [3] is a symptom checker app where an AI asks the user a series of questions regarding any symptoms they may be having, and provide possible explanations of those symptoms. The questionnaire typically includes 20–40 questions, most of which are binary (Yes/No), with an additional “I don’t know” option to account for uncertain or vague responses.
ADA is inherently a feedback-based app, where a user's input drives the reliability of the recommendation received. For this test, I have used ADA version release v3.136 on iOS.

To evaluate ADA's efficiency, I created a fake persona by the name of “Jane Lovely”. Jane is a native english speaker, belonging to Collingwood, Ontario (smaller town) and is generally concerned about her health, and prefers seeking medical assistance if required. In the following test, "Jane" is suffering from the following symptoms for the time period 1 week to 1 month:

1. Cough with Blood

2. Difficulty breathing

3. Difficulty sleeping due to cough

I ran three simulations based on the symptoms listed above, with the first two being similar aside of one detail, and the third one, where I purposefully chose severe inputs to test the scope of simulatability. The questions spanned a range of 20-35, building upon Jane's inputs each time.

Test 1 and 2 – Regular cough with Blood, no other symptoms

I answered the questions ADA provided as "Jane" to the best of my capabilities based on the symptoms above, which I will refer to henceforth as "discovery stage". When you are a new ADA health user, the series of questions seem to be higher in volume, as ADA is "learning" about you as the patient, and some details such as history of smoking, diabetes, blood pressure and pregnancy seem to be foundational. However, fundamental questions never included potential allergies, family history of disease such as heart conditions, which are often standard in healthcare practice.
Typos were also accounted for, as the top search result only accounts for correct words input, and also there was no indication for privacy transparency of data collection at any stage. (diagram 1)

The questions do however, seem to account for medical uncertainties, as several times through the discovery process "Jane" was asked, "Do you have any other symptoms?". This is crucial for healthcare, as the smallest of symptom can point to a plethora of diseases, and it is important to understand if there are any other underlying symptoms. ADA also reminds you, that the app only provides predictions based on your inputs, and the diagnosis should not be mistaken for a medical diagnosis from a professional. (diagram 1)

Upon completion of the discovery stage in test 1, ADA diagnosed "Jane" with either Asthma, Acute Bronchitis, Chronic Obstructive Pulmonary Disease, Mitral Stenosis or Occupational Asthma, in that order. (Diagram 2) In healthcare, diagnosis requires patient collaboration and trust with the healthcare provider to arrive to an accurate conclusion. [2] .

Medical practicioners often come across a wide variety of personalities; ranging from people proactive about their healthcare to outright dismissive, and use their experience and expertise to mitigate and guide covering grounds for accurate diagnoses for all individuals. As all of ADA's inputs are user-directed, so it is difficult to take into consideration the biases and ethos a user espouses in seeking medical care.

In test 2, all inputs were similar except "Difficulty breathing at rest" being an "I don't know" instead of a "No". (Diagram 3) Following suit, two additonal questions were added to the original roster:

1. Are your lips blue?

2. Has any one in your family been diagnosed with Asthma?

We know that many AI models provide predictions over-confidently, unless measures are applied to mitigate this during training. [4][5] How ADA computes confidence scores for possible diagnoses is unknown; they may be derived from an internal training database of user inputs and do not account for new entries, as I could not verify the correctness of any diagnosis. Users have no practical way to verify whether the probabilities or ranked diagnoses reflect real outcomes; at best, the app may let you review or rerun assessments, but clinical correctness cannot be confirmed. This raises questions about the recency of the symptom profiles and how the diagnosis is ranked for any discovery stage. It also creates a disconnect in model interpretability, making it difficult to weigh the numbers against a predicted diagnosis

User Experience professionals often account for missteps and provide corrective guidance, which is difficult to achieve with ADA's binary approach. The app can miss details in symptom discovery that might otherwise be captured in free-text responses.

Test 3 - Severity Increased, Cough with Blood and Severe Chest Pain

In this test, I wanted to understand how ADA flags potentially life-threatening conditions to seek emergency care, so the discovery stage was framed in a worsening trajectory, with initial symptoms being fundamental.

In real-world scenarios, classification systems such as AI should be accurately caliberated to provide predictions based on high confidence scores. Any predictions ranking low in confidence should ideally be passed on to a human SME, especially in automated healthcare systems. [5]

I answered questions that point to myocardial infarction, such as Arm, Jaw and Chest pain (Left side) [6]. To my surprise, even upon registering the severe symptoms, there were no call-to-actions provided in a potentially life-threatening situation, and the discovery stage continued as usual.

The diagnosis provided included Pulmonary Embolism, Acute Bronchitis, Intercostal muscle strain, Broken rib or Acute respiratory Failure, with Pulmonary Embolism and Acute Respiratory Failure marked as "Seek emergency care". (Diagram 4) Another surprising observation being myocardial infarction not being included on the list of possible suspects.

AI Overview - Linguistic Barrier Test

While AI promises faster symptom recognition in clinical practice, it can overlook nuances affecting end-users. In this round, I tested language barriers by emulating a person searching for a pulmonologist with Hindi as their primary language instead of English. I used Google, since most people habitually search for symptom matches there.

Linguistic barrier is a common occurrence in the healthcare industry, and professionals often have measures in place to address them. Health queries on Google often arise from people anxious about possible serious health conditions. In my queries, I emulated various inputs for linguistic barrier for "shortness of breath", ranging from "breath bad" to "not breath", and ultimately looking for pulmonary doctor with "breath doctor".

AI overview often responded by either explaining the query, such as foul breath for "bad breath" and grammar for "breath not breath". The useful query appeared to be "Not breath", but as AI overview occupies a significant portion of the screen real-estate, finding a doctor requires scrolling through options at the bottom of the screen. (Diagram 5)

I also searched for "pulmonoligist" in Hindi, where Google directed me to information about the role of a pulmonologist or general guidance on when to consult one.
An article [7] by The Guardian covers this very phenomenon, where incorrect queries may lead to serious health complications, such as AI overview suggesting people with pancreatic cancer to avoid high-fat foods, which was the exact opposite of factual medical advice, possibly leading to death, among many other cases.

UX Takeaways

Applications involving Healthcare should consider the highest priority in screening, understanding patient vulnerabilities, and safety in guidance. Some UX considerations to take away from this experiment include:

1. Understanding Patient Anxiety: People searching for symptoms arrive from a place of concern, often expecting severity. It is important to guide them to the appropriate healthcare professionals whenever there is reasonable doubt.

2. Accounting for Patient Differences: Applications should account for mental fortitude of potential patients; anticipate non-compliant users as well as misinformed users and potential linguistic barriers.

3. Mapping Red Flag Pathways: Systems should provide clear actionables, including escalation to authorities, for potentially life-threatening scenarios.

4. Transparency for AI Limitations: When an AI system reports high confidence in a serious diagnosis, it should clearly communicate that the result is based on algorithmic inference and may be prone to error. Users should be made aware of potential oversight, and guidance to consult a human professional should be included.

5. Data Privacy: Users should be made aware how their data is being used, especially critical in Healthcare context.

6. Recency of Symptom Profile: Systems should use up-to-date information and indicate whether symptom data is current.

As more of these applications move to mainstream, it is essential that we design products responsibly and ethically. Safe AI isn’t just about predictions; it’s about responsible UX that protects users.


References

1. Nielsen, J. (2024, January 30). 10 Usability Heuristics for User Interface Design. Nielsen Norman Group. https://www.nngroup.com/articles/ten-usability-heuristics/

2. Rathor, H. (2025, November 24). Beyond Accuracy: UX Spotlight on AI in healthcare. Heena Talks Design. https://heenarathor.framer.website/blog/ux-ai-in-healthcare

3. https://ADA.com/app/

4. Filho, T. S., Song, H., Perello-Nieto, M., Santos-Rodriguez, R., Kull, M., & Flach, P. (2021). Classifier Calibration: A survey on how to assess and improve predicted class probabilities. ArXiv. https://doi.org/10.1007/s10994-023-06336-7

5. Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On Calibration of Modern Neural Networks. ArXiv. https://arxiv.org/abs/1706.04599

6. Cleveland Clinic. (2025, December 22). Heart Attack (Myocardial Infarction). Cleveland Clinic. https://my.clevelandclinic.org/health/diseases/16818-heart-attack-myocardial-infarction

7. Gregory, A. (2026, January 2). Google AI Overviews put people at risk of harm with misleading health information. The Guardian. https://www.theguardian.com/technology/2026/jan/02/google-ai-overviews-risk-harm-misleading-health-information