Public Lab Research note


Environmental data management & analysis tools

by warren | February 17, 2017 01:13 17 Feb 01:13 | #13937 | #13937

A few fellow staff members asked me to do a survey of different platforms available for storing, managing, and sharing environmental data, as well as for doing analysis; after a lot of storyboarding to develop different in-depth specifications, I began organizing a list of existing software to see how well it met these needs.

While this is a really broad set of questions, I tried to develop a set of features which folks at Public Lab might need for specific cases. I also had a lot of great input from other staff in compiling this set of needs, so thanks!

I also only looked at programs that would (ideally) not take programming skills to use, although one exception was Phant -- because the (unfinished) Grapherate prototype was designed as a front-end for it.

The software I looked are:

Software

Here they are! Please leave a comment if there are others you'd like to add, and any help in actually filling in the details, or in correcting mistakes, is welcome. For some of these it wasn't easy to say yes or no, but I took notes on my best attempt. Also, for some of these the answers may change as the software continues to be improved!

Software Photosynq iSense Codap Phant Phant + Grapherate (as spec'ed) Plotly
Notes Some difficulty assessing as it's designed more narrowly for leaf measurements, though with programming it can be more flexible. $396/y but open source libraries
Audience Programmers, field collection, scientists, community groups ...? Programmers, scientists Programmers, scientists, students, citizen scientists Students, teachers Programmers Community groups, programmers, scientists Scientists
Easy to use? Hard to set up, easier to use Hard to set up, easier to use Hard to set up, easier to use No Yes Good web UI
Data type CSV upload API only Yes Yes Programmatic only, no UI Yes Yes
Live from sensors No Maybe with extra work Maybe with extra work Yes Yes Yes
Photo data Upload, no analysis Upload, no analysis Maybe with extra work No No Programmatic only, no UI
Bulk photo upload No No No No No No
Location data Yes Yes Yes Possible with extra work Yes Yes, but not from photos
Lab results: csv, pdf (hard to answer without sample data) No No No No No No
Form UI customization for data entry with JavaScript Yes Exploratory, educational No Yes No
Support Documentation Good for leaf analysis Great video series Thorough + legible docs API docs, for programmers No Yes
Long-term support infra/community Unclear Institutional Institutional Sparkfun, healthy community Nothing hosted, but debug/maintenance Probably, but commercial
Independent of single host Open source Open source Open source Open source Open source Significant portions are open source, but not UI
Discursive peer support Forums + commenting, some not fully implemented No No No No Some, via StackOverflow
Display Graphing Yes Yes Yes Via relatively easy integrations Yes Yes
Multiple sensors in a graph Yes Yes Yes Via relatively easy integrations Yes Yes
Map Yes Yes Yes Via integrations Yes Yes
Embeddable No Yes Yes Via integrations Yes Yes
Multiple sensors on a map Yes Yes Yes Via integrations Yes Yes
Display by project/user Yes Yes Yes Yes Yes Yes
Associate photo with metadata/notes fields Yes Yes No No No No
Comparison to historical/existing data No No No No Via data importing No
Contextual weather/wind/temp/humidity Programmatically via macros No Via integrations No Possible with extra work No
Heatmap Yes Yes Yes No Possible with extra work Yes
Threshold alerts No No No No Possible with extra work Possible with extra work
Evidential Changes log Not combined but each data entry listed Not combined but each data entry listed No No Possible with extra work No
User accounts / collaborative Yes Accounts but no discussion Accounts but no discussion No Embeddable on PL Yes
Checksum validation No No No No Possible with extra work No
Compatibility Online Yes Yes Yes Yes Yes Yes
Offline Yes No No No Possible with extra work No
Mac/Windows Web-based analysis, but not input Yes Yes Yes Yes Yes
Smartphone Android Browser, responsive Browser No Yes Probably limited interactive
Privacy Selective visibility If complete open sourcing, could run our own instance Could run our own instance Could run our own instance Could run our own instance Possible with extra work Yes

4 Comments

Awesome! Adding this open source data viz platform "Workbench" built by stalwart allies Columbia Journalism School with support from the Knight FDN -- could be useful~! https://medium.com/@Workbench/data-journalism-made-easier-faster-and-more-collaborative-e33081bf0080 and http://cjworkbench.org/signup-for-beta/

Reply to this comment...


Hello, @warren!

I'm at a startup, qri (https://qri.io/), where we're building an open source dataset version control tool. I've been working with Liz Barry to identify a Public Lab project where that might particularly helpful, so I wanted to reach out.

If you'd be interested, I'd be happy to explain where we are in developing the tool and whether / how it might be useful to you. Here are a few ways to learn more about what we're building in the meantime:

github: https://github.com/qri-io twitter: https://twitter.com/qri_io

Let me know if you'd like to discuss in more detail. Thanks!

Reply to this comment...


More listed here: https://forum.openhardware.science/t/server-software/1233/2

I'd love to know what people think about these options you've mentioned; we didn't review them. Always interested in thinking about long-term sustainability of these services.

https://thingspeak.com/

https://www.influxdata.com/time-series-platform/

https://www.oreilly.com/ideas/understanding-the-elk-stack

https://prometheus.io/docs/concepts/data_model/

Reply to this comment...


Oh hi @rgardaphe - sorry, missed your comment! Is there a plan for hosted or web-based QRI options? We are looking for ways to enable folks to upload data without having to run their own server -- for example folks collecting sensor data who aren't programmers -- as easily as possible, and for that to be archived. I understand the interest in decentralized storage, but is there a use case where people would be able to find an accessible home for community-collected sensor data?

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


Login to comment.