Google has quietly pulled back some AI-generated health summaries at the same time that the industry is seemingly racing to embed AI into healthcare. The move came after The Guardian found some of the information provided by the feature was misleading.

The Guardian reported Sunday that Google had removed AI-generated search summaries for the queries “what is the normal range for liver blood tests” and “what is the normal range for liver function tests.” As of Tuesday morning, those searches no longer provide an AI Overview at the top of the page. Instead, users are shown short excerpts pulled from traditional search results.

According to The Guardian, the AI Overviews for those liver-related searches and others had “served up inaccurate health information and put people at risk of harm.” For example, the summaries reportedly provided for the liver test searchers included “masses of numbers, little context and no accounting for nationality, sex, ethnicity or age of patients.”

The Guardian reported that some experts said these results could be dangerous. For instance, someone with liver disease might delay follow-up care if they rely on an AI-generated definition of what’s normal.

“We invest significantly in the quality of AI Overviews, particularly for topics like health, and the vast majority provide accurate information,” a Google spokesperson told Gizmodo in an emailed statement. “Our internal team of clinicians reviewed what’s been shared with us and found that in many instances, the information was not inaccurate and was also supported by high quality websites. In cases where AI Overviews miss some context, we work to make broad improvements, and we also take action under our policies where appropriate.”

The removals come at a moment when more people than ever are turning to AI for answers about their health, and the industry is taking notice.

OpenAI, the company behind ChatGPT, said last week that roughly a quarter of its 800 million regular users submit a healthcare-related prompt every week, with more than 40 million doing so daily.

Days later, OpenAI launched ChatGPT Health, a health-focused experience that can connect with users’ medical records, wellness apps, and wearable devices. And shortly after that, the company announced it had acquired the healthcare startup Torch, which tracks medical records including lab results, recordings from doctor visits, and medications.

Meanwhile, rival AI company Anthropic is not far behind. On Monday, the company announced a new set of AI tools that allow healthcare providers, insurers, and patients to use its Claude chatbot for medical purposes.

For hospitals and insurance companies, Anthropic says the tools can help streamline tasks such as prior authorization requests and patient communications. For patients, Claude can be given access to lab results and health records to generate summaries and explanations in plain language.

As AI advances in healthcare, even minor errors or missing context can have significant consequences for patients. Whether these companies are ready for that remains to be seen.