Public Lab Research note


SOC 2019: A small proposal for global environmental monitoring

by MaggPi | March 30, 2019 04:51 | 190 views | 9 comments | #18907 | 190 views | 9 comments | #18907 30 Mar 04:51

Read more: publiclab.org/n/18907


Title: A small proposal for global environmental monitoring with Image Sequencer

About me

I am a 19 year old student at The George Washington University in Washington D.C.. I am just finishing my first year and majoring in computer science. My main interests are computer vision and computer graphics. Outside of computers I enjoy drawing, math, and hiking. I participated in Google Summer of Code last year and while it was a lot of work, I really enjoyed it!

Project description

Proposal submitted in response to ‘Public Labs Idea List’ : Image Sequencer (multiple possible projects)'

  • The project will increase Image Sequencer capability while simultaneously demonstrating the ability to process satellite images. The general approach is to develop Image Sequencer as a user interface for openCV.js and openCV.js as computer vision processing engine for Image Sequencer. Technical objectives include: 1) streamlining satellite processing capability via image sequencer functions, 2) enabling/extending opencv applications for image sequencer and 3) demonstrate daily satellite environmental analysis over a 3 month period.

Abstract/summary (<20 words):

OpenCV.js and Image Sequencer will be integrated to develop a user friendly workflow for analysis of satellite images.

Problem and Proposed Solutions

Increasing Image sequencer applications While Image sequencer provides a baseline capability to display and process images, applications are limited by the lack of analysis tools. The addition on openCV.js will extend the range of possible applications.
Making satellite data accessible While there are several commercial and open source satellite analysis tools available, the tools require hours of users training and/or programming experience. This project’s goal is to upgrade Image Sequencer processing modules to enable ‘easy’ processing of satellite image data.

This proposal considers four main problems related to processing satellite data:

  • large file size: Satellite file sizes are typically greater than 100 MBs and this often results in data storage problems. The proposed solution is to develop an Image Sequencer module that restrict file size (10MB) and is centered on a region of interest identified by the user. While this may provide only a small portion of the remote sensing data, the assumption is that most Public Lab users are primarily interested in monitoring their local environment that has a footprint of several kilometers.
  • raster/geoTIFF: Satellite data requires reformatting for display and processing. This proposal will develop routines to process the data in formats acceptable to image sequencer (jpg). The initial approach will be to use OpenCV and GDAL to read geospatial files (assuming appropriate javascript bindings and features are available).

  • Image analysis: While Image Sequencer has several analysis tool such as NDVI, this proposal will develop more advanced processing features. One application under consideration is the measuring of water levels by comparing different image contours.

  • geo-referencing: Satellite data needs to be geo-referenced to permit comparison with other data. The proposed solution is to develop an Image Sequencer transfer module that provides the coordinates and image for the MapKnitter mapping application.

Public engagement in environmental monitoring This proposal seeks to increase environmental awareness by demonstrating the different ways to collect, process and combine neighborhood environmental data. As part of the proposal, one or two sites nominated by the Public Lab community will be selected for daily satellite imagery collection. Ideally, the same locations will also be observed by Public Lab Balloon/Kite cameras and other ground sensors. The objective is to develop ground truth monitoring to assist satellite measurements and for satellite sensing to guide ‘near earth’ based measurements.

Planet Satellite Imagery Info

(selected from https://assets.planet.com/docs/Combined-Imagery-Product-Spec-Dec-2018.pdf):

  • This proposal will use Planet satellite ‘Analytic’ Imagery - 3-band natural color (red, green, blue) or 4-band multispectral image (blue, green, red, near-infrared)
  • Planet Image Tiles are in GeoTIFF format with size 25 km (8000 lines) by 25 km (8000 columns). Data size is up to 500 Mbytes per Tile for 4 bands at 3.125 m pixel size.
  • Planet.com satellite software overview is available at: https://github.com/kscottz/PythonFromSpace/blob/master/Pycon2017.pdf
  • Planet.com software environment uses python based open standard tools GeoJSN, GDAL, QGIS, Rasterio , and GRASS GIS to manipulate geoTiffs. The tools enable (Application Processing Interface) APIs that allow listing, filtering, and downloading of data to anyone using a valid API key.
  • Planet.com has a software development environment for advanced processing functions such as image mosaicing and change detection. While these tools are not currently available for web-based platforms such as Image Sequencer, this proposal can be adapted to develop specialized programs specific to public lab applications.

Timeline/milestones

Community Bonding period (May 6 - 27, 2019) * Familiarize with Planet software/processing pipeline * Explore all image sequence functions * Submit Public Lab questionnaire for satellite analysis locations * Identify procedures for porting opencv.js to image sequencer * Connect with mentor(s)

Phase 1 Adapting openCV for Image sequencer: (May 27 to June 28)

  • Week 1 - First draft of Image Sequencer User manual/establish local environment of opencv.js/begin daily satellite downloads
  • Week 2 - Establish opencv porting /upload procedures/begin satellite collections
  • Week 3 - Opencv/Image sequencer interface and testing
  • Week 4 - Prepare documentation/final evaluation

Phase 2 Processing satellite data with openCV/Image sequencer: (June 29 to July26)

  • Week 5 – Segment/ crop geoTIFF data based on lat/long radius
  • Week 6 - convert geoTIFF to R,G,B,IR
  • Week 7 - optimize histogram correction and color balance routines
  • Week 8 - Prepare documentation/final evaluation

Phase 3 Processing satellite data with openCV/Image sequencer (cont) (July 27 to Aug 26)

  • Week 9 - advanced processing functions TBD
  • Week 10 - Geo reference w/mapKnitter module
  • Week 11 - End satellite date downloads/Process satellite data/update Image sequencer user manual
  • Week 12 -Prepare documentation/final evaluation

Needs

Satellite data: To gain access to satellite imagery, I have enrolled in Planet satellite student development program. Student developers can access up to 10,000sq km or about 30 satellite images per month.

Links below provide examples of 3m satellite imagery available from Planet.com: -Boston -New Orleans -This MapKnitter link provides an example of how Image Sequencer and MapKnitter can be combined to process Planet satellite imagery. The GIF displays portions of the processing workflow (RGB, R, IR , NDVI (grayscale), NDVI (Fastie), NDVI (stretched)) for a scene near the Mississippi River (Oakville, LA).

Adding OpenCV.js and GDAL to Public Labs may require additional support. Porting OpenCV.js and GDAL-js (GDAL can be considered a kind of OpenCV for geoTiffs) may require emscripten to compile OpenCV functions and a supporting server environment.

Contribution

New colormaps added to image sequencer. Work was done in javascript and provides an example of processed satellite data.
https://publiclab.org/notes/MaggPi/01-14-2019/updated-image-sequencer-colormaps

In addition, I have developed a microscope image stitching prototype designed for public lab hardware and software.


Experience

Examples of prior work and github files

Other areas of experience include a familiarity with Public Lab equipment (microscope, spectrometer) and have used Public Lab software (MapKnitter, Image Sequencer) to process imagery.

Teamwork

  • GSOC 2018 participation with Public Lab
  • Participation is several University team assignments ranging from robotics, computer programming and French.
  • My work for a NASA program called VASTS (Virginia Aerospace Science and Technology Scholars). The program selects 30 high school students to work with NASA engineers to design a hypothetical human Mission to Mars. I lead the strategic communications team that developed a consolidated system design across the different engineering teams. You can see the final design in a short video at: https://www.youtube.com/watch?v=gOZS4kw82cE&feature=youtu.be

Passion

I am hoping to use the Public Lab software developed under the project to analyze remote sensing data from environmentally stressed areas. I can’t think of a better opportunity (or community) to help develop the tools to monitor the environment and develop a better understanding of our planet’s future.

Audience

The availability of new satellite systems such as Planet.com creates opportunity to review three meter resolution environmental data more frequently. I’m hoping my project will help anyone interested in their local environment download, display and analyze satellite imagery. Possible public lab applications include monitoring of algae blooms, oil spills and water pollution.

Many people, like myself, are visual learners and need to see the entire picture in order to fully grasp a concept. In computer programming one rarely comes across visual representations since topics are usually extremely abstract. I hope that my project can help people who are still learning about microscopes, satellites, and image processing to see the bigger picture and gain a greater understanding of several abstract concepts.

Commitment

Strong commitment because I think I can make a difference. If we can make it easier to understand our local neighborhood ecology maybe we can determine better ways to shape the planet’s future.


9 Comments

Note there is some type of bug in the note editor that prevents making format changes. Everytime a change is made the entire post is wiped clean from the display.

Reply to this comment...


Oof, sorry about that bug, @MaggPi. Possibly legacy mode can help. Reading this now - very interesting!!!! Thank you!!!

Reply to this comment...


OK, first I really like your overall summary -- Increasing Image sequencer applications... especially! Making satellite data accessible is also a great goal for the project.

large file size: Satellite file sizes are typically greater than 100 MBs and this often results in data storage problems. The proposed solution is to develop an Image Sequencer module that restrict file size (10MB) and is centered on a region of interest identified by the user.

To consider here -- a lot of satellite data is stored as (giant) GeoTiffs by default, and a number of systems have been developed to serve sub-sections of a big GeoTiff over HTTP -- as a way around the need to reprocess GeoTiffs into another format like TMS tiles. (and because if you reformat into TMS, you are kind of "stuck" in the available tile grid, png/jpg image compression, and zoom levels of the TMS format you choose, and it's hard to work back). Some background that's good to know -- https://www.gdal.org/gdal_utilities.html GDAL is perhaps the most authoritative standard toolset for satellite data, but it's not a web-based application, it's a commandline system. It could be interesting (though not directly related here) to look at how the new MapKnitter Exporter dual-track project is creating a web-based toolchain both based on GDAL/ImageMagick, and in parallel, a clone based on Image Sequencer: https://github.com/publiclab/mapknitter/issues/298 -- it will also include a GeoTiff converter for Image Sequencer, which could be useful to you, and in general may be an interesting way to look at how a GDAL-type system can be adapted for web, or replaced by a more optimal JS/Image Sequencer alternative!)

Anyways, i think the "serve subsections of GeoTiffs" idea is called HTTP Get Range requests -- more here: http://www.cogeo.org/in-depth.html which... i dunno, could then be piped into Image Sequencer?

Also, I'm guessing you've seen these, which are really minimal, but provide a model for using ImageSequencer in Leaflet:

They're both using a TMS tile model for data, not GeoTiff, but as part of that I found a LOT of tile sources for satellite imagery here: https://github.com/publiclab/leaflet-tile-filter#multispectral-tms-tile-sources

So using only tiles is a viable path, even if it loses some of the original source data in Geotiffs.

One idea could be to adapt some of the tile-based approach in leaflet-tile-filter or leaflet-multispectral to filtering subselections of GeoTiffs -- maybe building on https://github.com/stuartmatthews/leaflet-geotiff

Here's an interesting mini-problem: how to preserve GeoTiff georeferencing data in non-Tiff file formats? Is there a standard for JPG or PNG that could essentially mirror GeoTiff (GeoJPEG or GeoPNG -- i bet something like this exists) and is it possible to convert lossessly between these? That could be really relevant for Leaflet.DistortableImage as well!

Some UI and logistical issues which could be great to solve to make the overall goal easier:

  • a better UI for piping remote imagery (via URL? via a draggable map?) into ImageSequencer's sequencer UI -- maybe related somehow to the multiple-inputs plan in https://github.com/publiclab/image-sequencer/issues/212
  • I think Image Sequencer has matured a lot, but it still lacks sufficient test coverage to be very stable. There are bugs in the crop module, and other modules, both in the UI and in the core module functions. Defaults are inconsistently provided. So I think a lot of work on test coverage might be a prerequisite to building a more complex stable library on top of IS. But the good news is that testing of the modules is pretty easy, and a lot of issues have been ironed out to open a pathway to near-100% coverage.

Finally, I'm interested in what the UI for a more advanced Image Sequencer based system might look like, for example within a Leaflet UI. This may or may not be the direction you're going, but could it be possible to select an imageOverlay or collection of tiles, and to open IS in a sidebar of a Leaflet map? Or would it be better to have a popup menu toolbar, as provided in https://github.com/publiclab/Leaflet.DistortableImage/, with https://github.com/leaflet/Leaflet.Toolbar ?

This is a really interesting and awesome project, @MaggPi, thanks so much! I think OpenCV.js will likely open up a huge set of possibilities, and this is a really interesting direction to go in. I think it's also possible that it may link up with @rexagod's proposal in some places, if both of you are able to work in well-documented standard ways -- for example, a number of OpenCV abilities may be relevant to feature point matching, for example.

Small question - what formats does Planet Labs provide?

Finally, i wonder what some specific workflows and abilities might be that you could describe that are /not/ possible from a web-based platform currently that you would be opening up and making easier for people through this project? Are there some applications using Planet Labs imagery that are currently only doable using complex or command-line software, or other use cases for NASA imagery that would be made more possible here? Some more examples could be great to outline.

Thanks a lot - i left a lot of thoughts here, you don't have to adopt them all but I hope they're interesting.

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


Hi, please upload your proposal at the Google Summer of Code website at the earliest. Please ignore this comment if already done.

Reply to this comment...


Hi everyone, Just want to say thank you to my mentors (@warren , @gauravano , @sagarpreet ) for agreeing to help in GSOC 2019! Also, would like to introduce myself to the other Image sequencer developers ( @aashnaaashna , @lit2017001 ) and their mentors (@Harshith and @jainaman224 ).

A project comparison spreadsheet is available at , https://github.com/MargaretAN9/GSOC-2019

or copy and past the direct link https://github.com/MargaretAN9/GSOC-2019/blob/master/Comparison%20matrix%20Public%20Lab%20GSoC%20Outreachy%202019%20v1%20(1).xlsx .

I did this to see if there are common trends across the different programs. If you think there are other program connections please feel free to update the matrix.

One common theme is opencv. Asshna’s and my project directly depend on opencv and Divy’s project may also benefit from colorspace conversion typically used in color comparisons.

So, does anyone have ideas on the best way to add opencv to Image sequencer? My first thought would to try to add opencv.js as a new module to Image Sequencer? Since this would be a very big module I suspect that another approach would work better.

There is also the issue that some image sequencer modules may duplicate open. cv functions (see https://github.com/opencv/opencv/blob/master/modules/js/src/embindgen.py#L82-L171 for list of opencv functions). Not sure if this is a bad thing but it may mean that we don’t need to include all opencv functions or some of the inconsistent IS modules (see Warren’s ‘crop’ comments above) would be replaced by opencv?

I am very interested in how opencv is added to IS since I assume a similar approach can also be used for GDAL. (GDAL is a sort of open.cv for GEOTIFF satellite images).

Is this a question? Click here to post it to the Questions page.

Hey, congratulations! I see we have project similarity and also the spreadsheet is some great work! I agree that adding opencv entirely is heavy and would cause duplicacy for many of our modules! I have the opencv addition to our nodejs project as the approach but am open to new options which are better. Also I would like to inform that I am pre-occupied with my semester exams till 22May as I mentioned in my proposal. I would love to share new ideas and work together to add the best approach. Thanks


Hi all! Just circling back here to sync up a bit. Some new things exist! Some plans have evolved! Let's stay synced, esp. @aashnaaashna and @MaggPi !

Just a few to note:

Any other updates people want to share? Thank you!

Is this a question? Click here to post it to the Questions page.


Heyy! I believe this is in reference to the commonality regarding praticle detection, I would like to share my progress as is seen in the pr, that currently blobs can be identified and I am trying if we can use opencv using npm.


Reply to this comment...


Note that in the Note Editor some sort of bug avoids modifications in size. The full article is washed off the screen every time a shift is created.

Reply to this comment...


Login to comment.

Public Lab is open for anyone and will always be free. By signing up you'll join a diverse group of community researchers and tap into a lot of grassroots expertise.

Sign up