Norms of Validation and the Agile Community

Had a great dinner with Mike Cottmeyer tonight in Bean Town.  At one point we got on the subject of the nature of research and validation expectations.  I was reminded that someone told me that different sciences have different validation and correlation expectations because they will reject the null hypothesis.

In high energy physics, the expectation is for the coefficient of determination to be at least 99.9999%.
Depending on which engineering community, 95-99% or greater is expected.
In the social sciences, it is not unusual to accept R2 values of 30-50% as "good enough".  Human interactions are considered so complex as that when R2 values are >90% are one has to validate you aren't just measuring the same construct in different ways.

These communities have other norms.  A couple norms in these communities are that research must be falsifiable and replicable.  Another typical norm is that another minimum level of validation is peer review by people with PhD's and publication in a journal associated with the discipline.

These are the norms these communities.  We can argue about them, we can even argue if these communities hold themselves to these norms all the time.

It struck that my observation is that the agile community has their own set of norms for considering work valid and something to be built upon.  I would argue that the agile community currently has validation norms well below those of the physical sciences, engineering, and social sciences.  Sometimes a poorly done case study (by the standards of the social sciences) or a claim that is published in book form (but not peered reviewed) is sufficient for validation in this community.  Sometimes it is just a blog post by a well paid consultant.

As agile software development enters these other communities (such as the engineering systems community) the agile community shouldn't be surprised if they are expected to reach new levels of validation before their findings are accepted.

This is a great opportunity for research.  Agile is relatively new and now there should be enough cases out their to start and build serious research upon.