The outline below is what happened in my brain when Public Lab people started discussing what a research note is and how to make the information in research notes more useful. There was some of this discussion at the Barnraising last month (photo above, photographer unknown) and more in the last couple of days. Below I am specifically responding to attempts to introduce more reproducibility into the research presented by the Public Lab community. I thought making an outline would prevent me from rambling, but it didn't work. I hope a lot more gets said about this topic, including refutation of all the points below that deserve it. Some other documents about this have been posted, but I don't know how public they are supposed to be, so I did not link to them here. Feel free to do so in a comment. Also for comments: Does it matter that most research in Public Lab research notes is not reproducible?
Liz, good points. It might be useful to draw a general dividing line between research and data collection. Research often requires data collection, but research seeks to test a hypothesis through evidence and new measurements whereas data collection is generally limited to just collecting the results within existing tool limits. I can profile water temperature and observe a pattern (data collection) but it isn't research unless there is a hypothesis about what causes that temperature profile and an objective of using the data to discover or prove a correlation (cause and effect not already known).
Yes, both development of measurement methods / tools / techniques require research and analyzing environmental cause-effect relationships using those tools can require research. (...and yes, not to appear too narrow, a high school student can still be doing research on something even though some scientist has already figured it out -- the methods, the discovery, the hypothesis, the data collection and analysis can all still be there. Research is also required to figure out a new, clever way to build or perform some measurement even when the root fundamentals have already been discovered.)
I'd agree, the two prevalent reproducibility examples are devices and results and both are needed. The devices and methods have to be reproducible -- which also means they could, in theory, be tested against some standard to validate they are working within spec. (This means that PLab devices should all have a set of specs which have been measured, verified and reproducible.) This is the best way to assure that measurement results can be reproduce. If a temperature probe has a 5% accuracy when calibrated but it is built by 10 people and used w/o taking a reference, then 1% resolution measurements will be meaningless.