Forensics on research and non-commutative property of science

We learn early on that when adding numbers, order doesn't matter.

2 + 3 = 5

3 + 2 = 5

Like addition, science should be commutative.

Any time new data are collected, the whole of the evidence needs to be reexamined with no favor given to hypotheses that were favored in the past.

Yet, science (and its application) is often not commutative. Order does matter.

The bar for disproving an idea is higher than proving it initially.

Here's an example.

Gene Towne and I just published a new study on burning of grasslands. The study examines 20 years of data from watersheds burned at different times of year--fall, winter, and late spring. It's the most comprehensive study of the importance of the season of burning in North American grasslands.

In short, the research shows that burning grasslands in the fall or winter compared to the late spring has little effect on grasses productivity, while favoring cool-season grasses and forbs. The details of the study can be found here or here.

To the rest of society, it's an important study because it has implications for how much of the last remaining tallgrass prairie is managed. This affects the grasslands that support a million or so cattle each year. But it also affects the millions of people that are downwind of the region when the grasslands burn.

Currently, in the Flint Hills, ranchers (and conservationists) burn prairie frequently every year. Based on recommendations, burning is concentrated in the late spring. When that happens, air quality standards are often exceeded in major cities downwind of the fires.

What is interesting here is to dig into where the recommendations came from. Scientific forensics.

When you examine the research upon which the recommendations are based, almost all of the research was conducted over 40 years ago. Moreover, it seems evident that the findings on when to burn are fairly equivocal.

And probably wouldn't pass rigorous scientific scrutiny today.

For example, the research from the 1960's and early 70's showed that plant biomass was found to be lower when grasslands were burned in early spring vs. late spring. Yet, these means were generated for just one small experimental plot per treatment.

The old data also showed that when grasslands were burned earlier soil moisture was lower, which presumably caused the lower productivity. Yet, in the particular study conducted in the 1960's, soil moisture data are reported as being lower in January in the early-season burn than the late-spring burn, two months before the burning has occurred. This result was likely a site effect.

Another major line of evidence to recommend late-spring burning was long-term data on cattle weight gain. The data on weight gain showed a trend of 6% greater steer weight gain from May to September over 16 years when burning on average on April 10 vs. March 20. Yet, as with other data, the weight gain of steers was again measured in just one pasture with no control for site differences that might be confounded with treatments. As for the data collected, although the weight gain of steers was significantly higher in burned than unburned pastures across 16 years, there was no significant difference in monthly weight gain with timing of burning (P > 0.1).

Based on the data circa 1970, if you had to choose a time to burn, late spring made sense. Yet, when all the past and current evidence are taken together, there is little evidence to support burning exclusively in the late spring.

Burning Flint Hills grasslands earlier is unlikely to have any major negative consequence for grass or cattle production, and may actually be beneficial (see the paper).

Unfortunately, more than likely, unlodging the idea that burning in late spring is necessary will be a lot harder than lodging it in the first place.

Unlike elementary addition, order is likely to matter here.