The bacteria that live inside our digestive tract undoubtedly play a vital part in our health. But buyer beware of companies that claim to have deciphered the gut microbiome. Research out today shows that no two at-home tests will tell you the same thing.

Government scientists sent standardized fecal samples to seven different gut health testing companies. The companies returned results that varied from one another, sometimes dramatically, while one company’s tests couldn’t conclusively decide if the same samples belonged to a healthy microbiome or not. The findings indicate that customers shouldn’t put too much stock in these tests, at least right now, the researchers say.

“Our results demonstrate the need for standards to ensure analytical validity and consumer confidence,” the authors wrote in their paper, published Thursday in Communications Biology.

Not quite there yet

Exciting as the field of gut health is, it’s very much in its infancy. We’re still not quite sure exactly what makes for a healthy mix of bacteria in our guts, much less how to reliably fix an unhealthy microbiome (it’s likely there are many different combinations of bacteria that could be “healthy”). And we’re still trying to untangle the complex interactions between our gut bacteria and various health conditions.

This uncertainty hasn’t stopped several companies from entering the direct-to-consumer industry, however. While some may be cautious in their advertising, others have claimed their tests can tell whether a person’s microbiome is healthy, and they might even sell products that will supposedly restore a dysfunctional one. Many scientists have already called for tighter regulation of these tests. Researchers at the National Institute of Standards and Technology, a division of the U.S. Department of Commerce, and others sought to gauge the reliability of these tests across different companies.

Our poop can be used as a marker for the gut microbiome, but it has its flaws. There’s lots of inherent variability in the samples (aka, turds) you use for testing, such that two samples given a day apart can yield noticeably different compositions of bacteria. And even the bacteria found at the beginning of a sample can differ significantly from the bacteria found at the end of one. Companies can also have important differences in how they collect, analyze, and interpret the samples.

To get around some of these limitations, NIST scientists developed and released a human stool reference material last year. This material has a consistent mix of bacteria and metabolic byproducts, which have been extensively categorized by the NIST. It’s important to note that this stool isn’t meant to be the gold standard of a healthy microbiome, only a baseline that scientists can use for research and testing. Though the final version contains a blend of samples from eight healthy donors, the NIST also created mixtures from individual donors.

In this latest study, the researchers used a single donor’s standardized poop to vet the accuracy of testing kits from seven DTC gut health companies (none of the companies are named in the study). For each company, they provided three samples of the same poop.

The companies’ tests routinely disagreed with one another, they found. While one company stated that a sample was filled with Clostridium bacteria (a common group of gut bacteria, some of which can cause disease), for instance, three others failed to detect traces of it in one or more samples. One company also offered contradictory assessments of the same poop. It declared one sample to be “unhealthy,” yet the other two were found to be “healthy.” All in all, they determined that the variability between companies was about as wide as the variability you’re likely to find between different donors.

These varying test results are likely due to differences in methods for sample collection and analysis used by the companies. And it suggests that these tests can’t be relied on to tell you much of anything currently. That lack of reliability could be harmful, too.

While some people might take these tests out of simple curiosity, others will use them to guide their future health decisions. At a press briefing Wednesday, co-author Diane Hoffman, a professor of health care law at the University of Maryland, detailed having surveyed doctors on how their patients have used these tests. In some cases, people radically changed their diets (or the diets of their children) in possibly dangerous ways based on test results; in one case, a person even underwent a fecal microbiota transplant, a promising but still developing treatment for changing the gut microbiome.

Other consumers could end up getting swindled by less scrupulous companies.

“Some of these companies also offer probiotics and other dietary supplements that they say will correct your microbiome,” said co-author Scott Jackson, a now-retired NIST researcher, in a response to a question from Gizmodo at the press briefing. “To correct your microbiome would presume that they’re seeing something wrong with your microbiome. More times than not, it just so happens that the thing that’s wrong with your microbiome happens to complement the product that they also sell.”

Should you take these tests?

There have been genuinely important advances in understanding the gut microbiome lately. And down the line, doctors could possibly offer treatments and interventions that can consistently help improve the gut health of many people. But that potential will depend on scientists and companies developing consistent standards across the field, both for testing and research, the authors say.

For now, these DTC gut health tests simply aren’t ready for primetime. And much like consumer genetic tests, the average person is probably better off staying far away from any tests claiming to have figured out the gut.