The outline below is what happened in my brain when Public Lab people started discussing what a research note is and how to make the information in research notes more useful. There was some of this discussion at the Barnraising last month (photo above, photographer unknown) and more in the last couple of days. Below I am specifically responding to attempts to introduce more reproducibility into the research presented by the Public Lab community. I thought making an outline would prevent me from rambling, but it didn't work. I hope a lot more gets said about this topic, including refutation of all the points below that deserve it. Some other documents about this have been posted, but I don't know how public they are supposed to be, so I did not link to them here. Feel free to do so in a comment. Also for comments: Does it matter that most research in Public Lab research notes is not reproducible?
Reproducibility
This is a good topic and, IMHO, points to the heart of entire PLab hypothesis -- that reproducible scientific measurements are possible by a broad spectrum of users.
A few thoughts: - The existing spectrometers can be mechanically reproduced - yet the mechanical results are variable (eg. alignment, ambient light, etc.) - They have reproducible results (they show a spectrum) but results are not yet sufficiently reproducible beyond simple observations. The important point is to investigate, identify, measure and then publish the real-world limits so that interpretation of collected data has a quantifiable value. This has yet to be done.
PLab published research notes also have a wide range of 'reproducibility' questions related to their content. For example: - Reporting measurement results using "accepted" devices is largely just data collection so the larger issue is technique and the limits of the device as used. - Notes about investigating a measurement limit, method, technique, etc. does increase the documentation requirement of the author and results and claims can beg review and more tests of reproducibility.
I agree that deciding about what needs further validation is not simple. However, I'd suggest that any research which proposes to change a boundary of what has, "in fact" (i.e. has been verified) been widely accepted as "known", needs verification. When one sets a spec for a device, it is necessary to sufficiently test so as to assure all users will get the same degree of measurement capability within that spec. If a new technique improves the results, that pushes the boundary so must be validated again -- reproducibility.
Cheers, Dave
Reply to this comment...
check this out, @p_silva_82
Reply to this comment...
I've noticed that we use the word reproducibility/replicability towards (at least) two ends: 1) can you build a tool according to instructions that succeeds at reaching a "Hello World" benchmark; and 2) can you reproduce the findings of a research project.
It may be confusing that our community uses the word "research" to apply to both tool development AND to environmental research using such tools.
Thanks Phil @p_silva_82 for chatting about this.
Reply to this comment...
Liz, good points. It might be useful to draw a general dividing line between research and data collection. Research often requires data collection, but research seeks to test a hypothesis through evidence and new measurements whereas data collection is generally limited to just collecting the results within existing tool limits. I can profile water temperature and observe a pattern (data collection) but it isn't research unless there is a hypothesis about what causes that temperature profile and an objective of using the data to discover or prove a correlation (cause and effect not already known).
Yes, both development of measurement methods / tools / techniques require research and analyzing environmental cause-effect relationships using those tools can require research. (...and yes, not to appear too narrow, a high school student can still be doing research on something even though some scientist has already figured it out -- the methods, the discovery, the hypothesis, the data collection and analysis can all still be there. Research is also required to figure out a new, clever way to build or perform some measurement even when the root fundamentals have already been discovered.)
I'd agree, the two prevalent reproducibility examples are devices and results and both are needed. The devices and methods have to be reproducible -- which also means they could, in theory, be tested against some standard to validate they are working within spec. (This means that PLab devices should all have a set of specs which have been measured, verified and reproducible.) This is the best way to assure that measurement results can be reproduce. If a temperature probe has a 5% accuracy when calibrated but it is built by 10 people and used w/o taking a reference, then 1% resolution measurements will be meaningless.
Reply to this comment...
Does it matter that most research in Public Lab research notes is not reproducible?
Insofar as reproducibility builds trust and trust binds and commits a community I think it does matter. Reproducibility across devices seems the more important and will lead PLabbers to more reproducible results. Reproducible results are not always correct, but at least we know they're accurate.
I agree with singling out certain reports as targets for replication though in practice I think when a note is of high caliber you're sort of drawn in and naturally want to try and replicate it. Does mere emulation of good behavior in others constitute replication?
Many research notes are part of a [often incomplete] series, and don't purport to be the equivalent of an entire research writeup.
The amount of leeway allowed for notes that are vague and rambling as often as they are rigorous and detailed is what I most enjoy about reading PL research. I like the idea of encouraging readers to continue along lines of a thread of work that someone else initiates. Having an ability to search and group along arbitrary lines of interest/level of detail would be nice too. Maybe I should figure out how to implement that.
Chris, I'll see your Karl Popper and raise you a Charles Sanders Peirce. Your 'ramblings' always enlighten and as you point out this is an an issue for pro science too.
Is this a question? Click here to post it to the Questions page.
Reply to this comment...
7 Comments
Love this. One thing I remember from the discussion was that someone pointed out that many research notes are part of a [often incomplete] series, and don't purport to be the equivalent of an entire research writeup. Maybe one way to encourage including methods, or "taking the next step" towards a fully documented research publication could be that if one tagged one's note as one phase of research, it could prompt you (or anyone else?) to take the next step; something like:
And it could have information and/or a template (header prompts in the note body field) with the appropriate material for the next "phase". This might have the added benefit that it doesn't add much complexity to a given note posting form -- but breaks up the process gradually over a series of notes. Eases people into it.
Is this a question? Click here to post it to the Questions page.
Reply to this comment...
Log in to comment
Community members could try to help tag new notes as appropriate to generate such prompts for others...?
Is this a question? Click here to post it to the Questions page.
Reply to this comment...
Log in to comment