Public Lab Research note


GSoC proposal: Computer Vision enhancements for Raspberry Pi based Public Lab Science Projects

by MaggPi |

Read more: publiclab.org/n/15978


About me

18 year old aspiring computer programmer; interested in computer science and how computer vision can be applied to automate spectroscopic analysis.

Affiliation
-Currently enrolled in high school/ Northern Virginia Community College with goal to transfer to a University (TBD) in 2018 Location: Northern Virginia, United States

Project description

The project would provide a series of modular python programs that could be used to support different Public Lab imaging and spectrum measurement toolkits. The programs enable near real time OpenCV computer vision (CV) measurements of images or spectra. The CV measurements can be used to improve instrument performance (feedback that controls lighting amplitude or camera array exposure times) or assist calibration by subtracting reference images/spectra.
General objectives include: • increasing the scope and usability of Public lab Spectral Workbench and spectrometer kits • collecting software techniques from Raspberry PI, computer vision and Adafruit blogs into a single source that can be applied to science projects • provide code that would automate data collection and present data in a visually appealing manner • optimize the use of Raspberry PI visible and NOIR camera in Public lab spectrometer kits • provide a foundation for new spectroscopic applications using python/opencv image processing libraries

Abstract/summary (<20 words):

Computer Vision enhancements for Raspberry Pi based Public Lab Science Projects

Problem

The problem I would like to solve is make camera integration easier into spectroscopic systems. Current systems are faced with multiple problems ranging from poor dynamic range, limited annotation and manual processing.

Timeline/milestones

Community Bonding period (April 23 - May 14)

• Familiarize with published software on PublicLab site/check operation.
• Configure Raspberry Pi 3 B+ with latest software • Identify project camera(s) and spectrometer(s) • Determine documentation requirements • Connect with mentor(s)

Phase 1 Code deliverables:basic camera functions: (May 14 to June 15 Evaluation)

• Week 1 -How to annotate picture/video • Week 2 -Camera resolution options and memory implications • Week 3 -Optimization of camera settings • Week 4 -Prepare documentation/final evaluation

Phase 2 Code deliverables:GPIO and Computer Vision(CV): (June 16 to July13 Evaluation) • Week 5 -Controlling light sources for imaging or spectroscopy • Week 6 -Manipulating RGB images/Histogram (masked) measurements • Week 7 -Time Lapse CV/Frame data annotations • Week 8 -Prepare documentation/final evaluation

Phase 3 Code deliverables:Spectroscopic Applications (July 14 to Aug 14 Evaluation) • Week 9 -Controlling dynamic range with feedback from CV processed spectrographs • Week 10 -Spectroscopic calibration for (near) real time spectroscopy • Week 11 -Transmission spectroscopy experiment – Is it possible to use a transmission grating and CV for a low cost imaging spectrometer? • Week 12 -Prepare documentation/final evaluation

Needs

-Open CV
-Raspberry Pi 3 B+ w/camera
-Transmission grating
-Public lab spectrometry kits
-Documentation requirements

First-time contribution

An example of prior work is available at the public lab website(s): https://publiclab.org/notes/MaggPi/03-15-2018/computer-vision-led-plant-measurement-system https://github.com/MargaretAN9/Peggy link text

Experience

Almost all my experience is from Raspberry Pi programming. I have taken one JAVA course on programming at a Community College. I have also taken 4 years of computer graphics/art in high school and become proficient in Adobe Illustrator/Lightroom/Photoshop.


Teamwork

An example of past team contributions is my work for a NASA program called VASTS (Virginia Aerospace Science and Technology Scholars). The program selects 30 high school students to work with NASA engineers to design a hypothetical human Mission to Mars. I lead the strategic communications team that developed a consolidated system design across the different engineering teams. You can see the final design in a short video at: https://www.youtube.com/watch?v=gOZS4kw82cE&feature=youtu.be


Passion

I understand that this proposal is not listed as a Public Lab GSOC topic. I just can’t pretend to be interested in topics such as ‘email or editor enhancements’. But, if you’re talking about the chance to help create new ways for software to automate scientific measurements then I am all in. I have been using Raspberry Pi for my science fair projects since sophomore year. My current interest is OpenCV computer vision measurements that can be used to enhance imaging and spectroscopic applications. With Public lab guidance (and equipment) I hope to continue my passion and develop a better understanding of software algorithms and scientific instrumentation

Audience

I would like to provide open source code that I wish was available when I first started working Raspberry PI based science projects. I believe that code would be useful to Public lab and Raspberry PI communities.

Commitment

-High motivation given the alternative is a summer job at the mall. What’s could be more fun than tinkering with computers, cameras and gratings all day!



software gsoc soc gsoc-2018 soc-2018 soc-2018-proposals gsoc-2018-final

response:13975


20 Comments

Hi @Maggpi, I guess we should talk about GSOC, when/how do you want to connect? @Warren, is there another way of communicating I should be using? Amir

Is this a question? Click here to post it to the Questions page.


Hi, I am not really sure how this works bit I think GSOC discussions moves over to github, https://github.com/publiclab/plots2/issues/2653, which is replicated to Gitter. https://gitter.im/publiclab/soc. In any case, really impressed with your opencv work, https://publiclab.org/notes/amirberAgain/01-12-2018/python-and-opencv-to-analyze-microscope-slide-images-of-airborne-particles and wondering if you could recommend any opencv teaching guides or ‘how to’ manuals?

Is this a question? Click here to post it to the Questions page.


Thank you for the kind words! I was very impressed with some of your posts on microscopy! The two resources I personally like best are the openCV tutorials and PyImageSearch. https://docs.opencv.org/3.4.0/d6/d00/tutorial_py_root.html

https://www.pyimagesearch.com/ [Try looking at the older content to begin with, as he now works a lot on ML/AI].


@amirberAgain we have a chat room on gitter and also on slack. You people can communicate to each other or us on these platforms. and if you people are interested i will send you the invitation link for the same. Thanks


Hi @MaggPi! Actually the easiest way to reach the PublicLab chatroom is thru https://chat.publiclab.org/ . While we support bridges to other chat networks, our "home" chatroom is hosted by matrix.org and reachable with Riot https://about.riot.im/ - a libre messaging system. I will be helping with co-mentoring - also I am PublicLab.Org's sysadmin. My strengths are systems programming, web, and Python. I hope you will have a great summer project! Please ping us if you feel stuck at any time. Any community channel is good!


Hi, @maggpi - could you paste in your GitHub project link here? Have you thought of a name for your project, even provisionally?

Looking through your materials now! Welcome!

Is this a question? Click here to post it to the Questions page.


One change seems to be the desire to collect images by microscope. I have the equipment for this but need advice on what should be test images? I assume a calibration slide is appropriate https://publiclab.org/wiki/basic-microscope but it’s also not a very fun reference and has no color. It may be a good time to try to measure something readily available (like a dollar bill) so that anyone could have a reference without paying $10 for a calibration slide.

Great question -- there are some ideas here; https://publiclab.org/notes/partsandcrafts/02-26-2018/6-looking-at-dust-and-other-things -- i like looking at things that move, but they're hard to keep alive :-)

One concern is what is the best spectrometer for phase 3. After reviewing the different Public Lab kits, the lego spectrometer seems the best choice. Lego was selected since it was compatible with the Raspberry Pi camera and I could possibly share results with cedarlodge(from Public Lab), https://publiclab.org/notes/cedarlodge/03-07-2018/raspberry-pi-spectrometer-gui#comments, So is this OK?

That's great, yes. And if @programmer1200 is willing, maybe we could send you some of the attachments he's working on? ( #16042) @programmer1200 if you're still interested in swapping some with us, and could include a second set, we'd love to forward it along. But either way - we can just forward ours along too.

What is the policy for getting access to public lab kits and parts? Do I need to purchase or can Public lab provide? I would like to use 2 lego kits (one for visible and one for vis-IR ). Note I plan to use my own v2 cameras and Raspberry Pi. (Please tell me if there is there some reason why the lego kit spectrometer comes with v1 instead of higher resolution v2?)

Sorry, we actually already sent you things! I hope there's enough there - we can send you a second lego set. The camera thing is just a cost issue - we know people who want higher quality cameras can get them on their own... they're expensive for us to carry in quantity. But we'll probably start carrying both versions!

Hope this is helpful! Reading your revisions above now.

Is this a question? Click here to post it to the Questions page.


OK, revisions look good! As @amirberagain mentioned, maybe expanding your timeline/checklist a bit could help. I'd like to see an outline of how you plan to organize your code; maybe pointing at a well-organized library as an example?

In JavaScript, we're trying for relatively short files, which are require()'ed together, and named consistently -- this tends to be good for other people to be able to quickly read through, find relevant parts, etc. Here's an example: https://github.com/publiclab/image-sequencer/tree/master/src/

I'm not as familiar with Python but I imagine similar coding style could be achieved! If there's an example project you'd like to aspire to, structurally, please share!

Once your GitHub repository is set up, I think a good step will be to make a planning issue there with the tasks you have in mind. See these examples: https://github.com/publiclab/plots2/labels/planning

I hope this helps. Thank you very much @MaggPi!

Is this a question? Click here to post it to the Questions page.


I think you are just right @warren. possibly one of the outputs of the initial planning phase should be a list of functions (and objects?) which could then be mapped out and divided in a logical way, maybe even some state machine flows? Calling functions and objects from other Python libraries is not a problem and should be desired, basically, you just write a .py file and import it to another one.

Is this a question? Click here to post it to the Questions page.


Hi, this is first test of the chat in chatroom. I was wondering if there is any preference for Open CV python virtual environment vs a non virtual environment. I understand the virtual environment helps multiple programs use the same libraries but it also seems like another rthing to go wrong. Also, AmirberAgain is not listed in the chat name

HI @MaggPi With regard to Python virtual environments you shouldn't worry about the they are a convenience for the developer but make no difference on the actual code. They exist so that you can mix and match library versions without messing with your operating system's Python. For our purposes it's fine to use the system's Python interpreter and libraries. You can install libraries as your current user with pip install --user library.

If your program uses external libraries be sure to create a requirements.txt file.

Also on IRC - some of us leave bots but are not at the keyboard. It's a way to get a backlog when we come back. A similar functionality is provided by Matrix.Org/Riot.im - it's good to leave your question and stick around until somebody answers! Github issues are also good for open discussion.


Hey @maggPi and @icarito, let's try to sync up and try to have a call in the coming days. I was thinking Thursday 15:00 EST (the same time as the open call starts), how does that sound to you?

Is this a question? Click here to post it to the Questions page.


icarito amirberAgain

My IB French exam is Thus and Fri. Is is possible to set something up Friday afternoon? I know you are looking for more details so please read below.

This post reviews GSOC first month activities and goals/programs to be developed for the second month.

First month activities:

-Reviewed public lab/spectral workbench web site. – Have tried to review all sections related to microscopes /spectrometry and multispectral imaging.
-started building lego spectrometry, multispectral imaging – having trouble with consistent measurements. Tried different designs to see what works best. Working on adjustable camera mount for spectrometer.
-prepared visible/IR table top camera
-prepared several notes/questions, see: https://publiclab.org/profile/MaggPi -began learning spectral workbench calibration software -purchased 2 raspberry pi 3B+/installed opencv. -setbacks- New raspberry pi gave blinking light power supply error on start-up and had to be returned. Damaged a pi camera while trying to mount in spectrometer.
-established contact with mentors -received public lab equipment

Goals:

The second month will develop a series of programs/functions that will adjust cameral settings and display images with opencv. The programs will be tested with imaging (microscope), multispectral and spectroscopic applications.

-Programs will be based on the Raspberry Pi visible, NoIR cameras. This will permit controlling camera functions by picamera. The general goal is to establish a two way link between the measurement, camera settings and computer vision analysis so the programs can be used for the next phase.

-Programs will be not be written for video analysis. This will provide time for Raspberry Pi to process opencv tasks at high resolution. Video will be used for camera set-up and spectrometer alignment.

Program Checklist:

Phase 1 tasks concentrate on controlling and recording different camera settings. -annotation of jpeg picture (time and file name) from pi camera -collection of RAW image data from pi camera (this should provide 12 bit range, so far all I have been able to collect is 8 bit jpeg) -annotation of picture with camera settings (displays auto exposure settings) -collect and process RGB or other split channel images (for example: https://publiclab.org/notes/MaggPi/05-14-2018/is-there-a-way-to-use-the-rgb-spectral-overlap-to-make-better-spectral-measurements) -prepare routines for side-by side display of multiple images (permits comparison) -cycle routines that captures and display side by side annotated images at different camera settings

General Comments:

Biggest challenge so far is working with opencv. I worked with simplecv before and opencv is ---- not so simple. One example is simple tasks such as displaying side by side images often get errors. I know the opencv sidebyside command is hconcat but it only works if the arrays are formatted the same size. So this means you need to think how to scale all the images beforehand.

Realize now that getting this stuff to behave isn’t easy.

Questions: What type of format is required for the June 11 evaluation? Is there some type of grading criteria like how many lines of code were written?

Is this a question? Click here to post it to the Questions page.


hey @Maggpi, I was sure the exam you mentioned previously was last Thursday - sorry for that. Best of luck with your exams, Let's sync start of next week? As for OpenCV - this is not a hard thing to do, use the resize as explained here: https://docs.opencv.org/3.4/da/d6e/tutorial_py_geometric_transformations.html

I guess @warren will be able to share experience on grading. If it's based on the number of lines of code we will need to change the criteria...

Is this a question? Click here to post it to the Questions page.


Hi @maggpi, when would you like to talk?

Is this a question? Click here to post it to the Questions page.


Hi, @amiberAgain How about next Monday or Wednesday 9AM US time. Hopefully, I will have camera comparison results by then. What's the the best way to set up a phone call? Thx

Is this a question? Click here to post it to the Questions page.


The coming Monday doesn't work for me, how about Tuesday, June 5th at 13:00 EST? Maybe a bit later and just before the openhour? The esiest option for me is to use the Hangouts, but we could also set a zoom meeting. The main subject I want to discuss is the plan and goals for the rest of the summer and see if there are any questions you might have that I could help with. @icarito does that work for you as well?

Is this a question? Click here to post it to the Questions page.


1300 EST would work for me too actually, and I'd love to join!


@amirberAgain, @warren, @icarito

This note is a follow up to the June 6 meeting, a past work summary and outline of future goals.
a. Code organization b. Milestones and future goals c. Raspberry Pi

Code organization: There was a good discussion of different ways to organize Raspberry Pi software for different Public Lab users and applications at the June 6 meeting. The approach selected considers three categories (Picamera, USB and Image Processing) that are based on different ways the image is acquired. In addition, multiple software routines have been posted to accommodate different levels of experience. For example, a beginner will probably just want to capture an image, a slightly more experienced user will want to capture and annotate the image, and an experienced programmer will want to capture, annotate and process the image. (with opencv). All code is available at: https://github.com/MargaretAN9/Peggy. Demonstration of code output can be seen at https://publiclab.org/profile/MaggPi

Postings on github as of June 11, 2018 Picamera: Designed for the Raspberry PI with PI v2 camera/picamera software.

--PiCameraEffectsShow.py Display several processing modes available with picamera --PiCameraEffectsVid.py Same as PiCameraEffectsShow.py but also records video in h264 format --picameravidwithimagecapture.py
Displays video and captures image
--picamerawithimagecaptureopencv.py Displays video captures image (with open cv)

USB: Designed for Raspberry PI with a USB webcamera.

--USBTakesPic+Annotates.py -uses fswebcam for picture --USBVidWithImageCapture.py -uses pigame for video and image capture

Image Processing: This category is for users who want to process a stored image.

--AddsAxisToImage.py
-Adds resolution scale to image --RGBquad.py
-Displays RGB pictures in quad format --YUVQuad.py -Displays YUV pictures in quad format

Milestones and future goals: The third month will continue past work and concentrate on interacting with the camera/object versus just using the auto settings. The programs will be tested with imaging (microscope), multispectral and spectroscopic applications.

Phase 2 Code deliverables: Exposure control programs: (June 16 to July13 Evaluation) Week 5 - Optimization for longer exposure times, requires multiple settings to be adjusted and looping thru different settings. Week 6 -Manipulating RGB/HSV image, masked measurements and selection program. Region of Interest selection is useful for spectral data since you can record a slice of the 3240 x 2464 image. Week 7 -Intensity and histogram maps
Week 8 -Prepare documentation/final evaluation

Phase 3 Code deliverables: Spectroscopic Applications (July 14 to Aug 14 Evaluation) The last phase will concentrate on applying some of the general processing options developed earlier to specific applications. Applications to be considered are NIR/G/B microscopy and spectral calibration. The general approach for spectral calibration is to see if multiple views at different exposure settings and analyzing data from the higher spectral orders could be used for spectral analysis. I thought I would work on this since most everyone in computer vision works stuff like face recognition but few are working spectral recognition. A major challenge is overcoming spectral overlap issues in higher orders.
Week 9 -Controlling dynamic range with feedback from CV processed spectrographs Week 10 -Spectroscopic calibration for (near) real time spectroscopy Week 11 -Transmission spectroscopy experiment – objective is to extend the free spectral range Week 12 -Prepare documentation/final evaluation

Raspberry PI:

I know Public Lab already provides a pre-load image for streaming applications ( https://publiclab.org/notes/partsandcrafts/12-03-2017/setting-up-the-raspberry-pi-camera-for-wireless-streaming ) but it may be worthwhile to consider an image for non-streaming applications as well. This would permit work without WiFi and allow camera preprocessing capability. The image does not necessarily need to include opencv since python/picamera/ matplotlib also provide many image processing/display features.


@amirberAgain, @warren, @icarito This note is a GSOC phase 2 summary and updates phase 3 goals.

Phase 2: Phase 2 research notes are posted at https://publiclab.org/profile/MaggPi. Comments on software programs (posted at https://github.com/MargaretAN9) are listed below :

-Color detection – generates mask/ calculates HSV colorspace range. Primary contribution was the use of trackbars for the user to easily select color range(s). (HSVcolortrackbar.py) -Exposure control matrix - Developed to get a systematic way to adjust Raspberry PI manual camera settings. My real first experience with the multiple tunings necessary for long exposures (up to 10 sec) and low frame rate. (exposuremosaic.py) -Microscope grid overlay with calibration. Simple CV geometry functions permit scale reference. Amazed that this works as well as it did. (videooverlayfinal.py) -Insect infrared microscopy – Simple contrast adjustments permit enhanced detail. Effort demonstrates need for better focusing options and real time histogram optimization.

In addition to the research notes, additional ‘test’; code was tested on Fourier masks , real time measurement, histogram equalization and region of interest selection .
On setback is that I have been unable to use 10bit RAW Raspberry Pi camera data. I have tried different processing options but they all need color and lens corrections. I have started work on HDR techniques as an alternative. My initial review is that HDR may operate over a larger dynamic range than a 10 bit RAW image.

Phase 3 Code deliverables: Spectroscopy and Microscope Applications (July 14 to Aug 14) Goals were outlined in prior posts but are summarized below for convenience: ‘The last phase will concentrate on applying some of the general processing options developed earlier to specific applications. Applications to be considered are NIR/G/B microscopy and spectral calibration. The general approach for spectral calibration is to see if multiple exposure settings and analyzing data from the higher spectral orders could be used for spectral analysis’.

Week 9 –Real time spectrographs Week 10 -Spectroscopic calibration for (near) real time spectroscopy Week 11 -Transmission spectroscopy experiment – objective is to extend the free spectral range Week 12 -Prepare documentation/final evaluation


Thanks, this is very helpful for the eval writing. Much appreciated!


You must be logged in to comment.