18 year old aspiring computer programmer; interested in computer science and how comput...
Public Lab is an open community which collaboratively develops accessible, open source, Do-It-Yourself technologies for investigating local environmental health and justice issues.
Public Lab Research note
As an open source community, we believe in open licensing of content so that other members of the community can leverage your work legally -- with attribution, of course. By joining the Public Lab site, you agree to release the content you post here under a Creative Commons Attribution Sharealike license, and the hardware designs you post under the CERN Open Hardware License 1.1 (full text). This has the added benefit that others must share their improvements in turn with you.
GSoC proposal: Computer Vision enhancements for Raspberry Pi based Public Lab Science Projects
18 year old aspiring computer programmer; interested in computer science and how computer vision can be applied to automate spectroscopic analysis.
-Currently enrolled in high school/ Northern Virginia Community College with goal to transfer to a University (TBD) in 2018
Location: Northern Virginia, United States
The project would provide a series of modular python programs that could be used to support different Public Lab imaging and spectrum measurement toolkits. The programs enable near real time OpenCV computer vision (CV) measurements of images or spectra. The CV measurements can be used to improve instrument performance (feedback that controls lighting amplitude or camera array exposure times) or assist calibration by subtracting reference images/spectra.
General objectives include:
• increasing the scope and usability of Public lab Spectral Workbench and spectrometer kits
• collecting software techniques from Raspberry PI, computer vision and Adafruit blogs into a single source that can be applied to science projects
• provide code that would automate data collection and present data in a visually appealing manner
• optimize the use of Raspberry PI visible and NOIR camera in Public lab spectrometer kits
• provide a foundation for new spectroscopic applications using python/opencv image processing libraries
Abstract/summary (<20 words):
Computer Vision enhancements for Raspberry Pi based Public Lab Science Projects
The problem I would like to solve is make camera integration easier into spectroscopic systems. Current systems are faced with multiple problems ranging from poor dynamic range, limited annotation and manual processing.
Community Bonding period (April 23 - May 14)
• Familiarize with published software on PublicLab site/check operation.
• Configure Raspberry Pi 3 B+ with latest software
• Identify project camera(s) and spectrometer(s)
• Determine documentation requirements
• Connect with mentor(s)
Phase 1 Code deliverables:basic camera functions: (May 14 to June 15 Evaluation)
• Week 1 -How to annotate picture/video
• Week 2 -Camera resolution options and memory implications
• Week 3 -Optimization of camera settings
• Week 4 -Prepare documentation/final evaluation
Phase 2 Code deliverables:GPIO and Computer Vision(CV): (June 16 to July13 Evaluation)
• Week 5 -Controlling light sources for imaging or spectroscopy
• Week 6 -Manipulating RGB images/Histogram (masked) measurements
• Week 7 -Time Lapse CV/Frame data annotations
• Week 8 -Prepare documentation/final evaluation
Phase 3 Code deliverables:Spectroscopic Applications (July 14 to Aug 14 Evaluation)
• Week 9 -Controlling dynamic range with feedback from CV processed spectrographs
• Week 10 -Spectroscopic calibration for (near) real time spectroscopy
• Week 11 -Transmission spectroscopy experiment – Is it possible to use a transmission grating and CV for a low cost imaging spectrometer?
• Week 12 -Prepare documentation/final evaluation
Almost all my experience is from Raspberry Pi programming. I have taken one JAVA course on programming at a Community College. I have also taken 4 years of computer graphics/art in high school and become proficient in Adobe Illustrator/Lightroom/Photoshop.
An example of past team contributions is my work for a NASA program called VASTS (Virginia Aerospace Science and Technology Scholars). The program selects 30 high school students to work with NASA engineers to design a hypothetical human Mission to Mars. I lead the strategic communications team that developed a consolidated system design across the different engineering teams. You can see the final design in a short video at: https://www.youtube.com/watch?v=gOZS4kw82cE&feature=youtu.be
I understand that this proposal is not listed as a Public Lab GSOC topic. I just can’t pretend to be interested in topics such as ‘email or editor enhancements’. But, if you’re talking about the chance to help create new ways for software to automate scientific measurements then I am all in.
I have been using Raspberry Pi for my science fair projects since sophomore year. My current interest is OpenCV computer vision measurements that can be used to enhance imaging and spectroscopic applications. With Public lab guidance (and equipment) I hope to continue my passion and develop a better understanding of software algorithms and scientific instrumentation
I would like to provide open source code that I wish was available when I first started working Raspberry PI based science projects. I believe that code would be useful to Public lab and Raspberry PI communities.
-High motivation given the alternative is a summer job at the mall. What’s could be more fun than tinkering with computers, cameras and gratings all day!
@amirberAgain we have a chat room on gitter and also on slack. You people can communicate to each other or us on these platforms. and if you people are interested i will send you the invitation link for the same.
Reply to this comment...
Actually the easiest way to reach the PublicLab chatroom is thru https://chat.publiclab.org/ .
While we support bridges to other chat networks, our "home" chatroom is hosted by matrix.org and reachable with Riot https://about.riot.im/ - a libre messaging system.
I will be helping with co-mentoring - also I am PublicLab.Org's sysadmin. My strengths are systems programming, web, and Python.
I hope you will have a great summer project! Please ping us if you feel stuck at any time. Any community channel is good!
Reply to this comment...
Hi, @maggpi - could you paste in your GitHub project link here? Have you thought of a name for your project, even provisionally?
One change seems to be the desire to collect images by microscope. I have the equipment for this but need advice on what should be test images? I assume a calibration slide is appropriate https://publiclab.org/wiki/basic-microscope but it’s also not a very fun reference and has no color. It may be a good time to try to measure something readily available (like a dollar bill) so that anyone could have a reference without paying $10 for a calibration slide.
That's great, yes. And if @programmer1200 is willing, maybe we could send you some of the attachments he's working on? ( #16042) @programmer1200 if you're still interested in swapping some with us, and could include a second set, we'd love to forward it along. But either way - we can just forward ours along too.
What is the policy for getting access to public lab kits and parts? Do I need to purchase or can Public lab provide? I would like to use 2 lego kits (one for visible and one for vis-IR ). Note I plan to use my own v2 cameras and Raspberry Pi. (Please tell me if there is there some reason why the lego kit spectrometer comes with v1 instead of higher resolution v2?)
Sorry, we actually already sent you things! I hope there's enough there - we can send you a second lego set. The camera thing is just a cost issue - we know people who want higher quality cameras can get them on their own... they're expensive for us to carry in quantity. But we'll probably start carrying both versions!
Hope this is helpful! Reading your revisions above now.
OK, revisions look good! As @amirberagain mentioned, maybe expanding your timeline/checklist a bit could help. I'd like to see an outline of how you plan to organize your code; maybe pointing at a well-organized library as an example?
I think you are just right @warren. possibly one of the outputs of the initial planning phase should be a list of functions (and objects?) which could then be mapped out and divided in a logical way, maybe even some state machine flows?
Calling functions and objects from other Python libraries is not a problem and should be desired, basically, you just write a .py file and import it to another one.
Hi, this is first test of the chat in chatroom. I was wondering if there is any preference for Open CV python virtual environment vs a non virtual environment. I understand the virtual environment helps multiple programs use the same libraries but it also seems like another rthing to go wrong. Also, AmirberAgain is not listed in the chat name
With regard to Python virtual environments you shouldn't worry about the they are a convenience for the developer but make no difference on the actual code. They exist so that you can mix and match library versions without messing with your operating system's Python. For our purposes it's fine to use the system's Python interpreter and libraries. You can install libraries as your current user with pip install --user library.
If your program uses external libraries be sure to create a requirements.txt file.
Also on IRC - some of us leave bots but are not at the keyboard. It's a way to get a backlog when we come back. A similar functionality is provided by Matrix.Org/Riot.im - it's good to leave your question and stick around until somebody answers! Github issues are also good for open discussion.
Reply to this comment...
Hey @maggPi and @icarito, let's try to sync up and try to have a call in the coming days. I was thinking Thursday 15:00 EST (the same time as the open call starts), how does that sound to you?
My IB French exam is Thus and Fri. Is is possible to set something up Friday afternoon?
I know you are looking for more details so please read below.
This post reviews GSOC first month activities and goals/programs to be developed for the second month.
First month activities:
-Reviewed public lab/spectral workbench web site. – Have tried to review all sections related to microscopes /spectrometry and multispectral imaging.
-started building lego spectrometry, multispectral imaging – having trouble with consistent measurements. Tried different designs to see what works best. Working on adjustable camera mount for spectrometer.
-prepared visible/IR table top camera
-prepared several notes/questions, see: https://publiclab.org/profile/MaggPi
-began learning spectral workbench calibration software
-purchased 2 raspberry pi 3B+/installed opencv.
-setbacks- New raspberry pi gave blinking light power supply error on start-up and had to be returned. Damaged a pi camera while trying to mount in spectrometer.
-established contact with mentors
-received public lab equipment
The second month will develop a series of programs/functions that will adjust cameral settings and display images with opencv. The programs will be tested with imaging (microscope), multispectral and spectroscopic applications.
-Programs will be based on the Raspberry Pi visible, NoIR cameras. This will permit controlling camera functions by picamera. The general goal is to establish a two way link between the measurement, camera settings and computer vision analysis so the programs can be used for the next phase.
-Programs will be not be written for video analysis. This will provide time for Raspberry Pi to process opencv tasks at high resolution. Video will be used for camera set-up and spectrometer alignment.
Phase 1 tasks concentrate on controlling and recording different camera settings.
-annotation of jpeg picture (time and file name) from pi camera
-collection of RAW image data from pi camera (this should provide 12 bit range, so far all I have been able to collect is 8 bit jpeg)
-annotation of picture with camera settings (displays auto exposure settings)
-collect and process RGB or other split channel images (for example: https://publiclab.org/notes/MaggPi/05-14-2018/is-there-a-way-to-use-the-rgb-spectral-overlap-to-make-better-spectral-measurements)
-prepare routines for side-by side display of multiple images (permits comparison)
-cycle routines that captures and display side by side annotated images at different camera settings
Biggest challenge so far is working with opencv. I worked with simplecv before and opencv is ---- not so simple. One example is simple tasks such as displaying side by side images often get errors. I know the opencv sidebyside command is hconcat but it only works if the arrays are formatted the same size. So this means you need to think how to scale all the images beforehand.
Realize now that getting this stuff to behave isn’t easy.
What type of format is required for the June 11 evaluation? Is there some type of grading criteria like how many lines of code were written?
The coming Monday doesn't work for me, how about Tuesday, June 5th at 13:00 EST? Maybe a bit later and just before the openhour? The esiest option for me is to use the Hangouts, but we could also set a zoom meeting.
The main subject I want to discuss is the plan and goals for the rest of the summer and see if there are any questions you might have that I could help with. @icarito does that work for you as well?
This note is a follow up to the June 6 meeting, a past work summary and outline of future goals.
a. Code organization
b. Milestones and future goals
c. Raspberry Pi
Code organization: There was a good discussion of different ways to organize Raspberry Pi software for different Public Lab users and applications at the June 6 meeting. The approach selected considers three categories (Picamera, USB and Image Processing) that are based on different ways the image is acquired. In addition, multiple software routines have been posted to accommodate different levels of experience. For example, a beginner will probably just want to capture an image, a slightly more experienced user will want to capture and annotate the image, and an experienced programmer will want to capture, annotate and process the image. (with opencv). All code is available at: https://github.com/MargaretAN9/Peggy. Demonstration of code output can be seen at https://publiclab.org/profile/MaggPi
Postings on github as of June 11, 2018
Designed for the Raspberry PI with PI v2 camera/picamera software.
Display several processing modes available with picamera
Same as PiCameraEffectsShow.py but also records video in h264 format
Displays video and captures image
Displays video captures image (with open cv)
Designed for Raspberry PI with a USB webcamera.
-uses fswebcam for picture
-uses pigame for video and image capture
This category is for users who want to process a stored image.
-Adds resolution scale to image
-Displays RGB pictures in quad format
-Displays YUV pictures in quad format
Milestones and future goals:
The third month will continue past work and concentrate on interacting with the camera/object versus just using the auto settings. The programs will be tested with imaging (microscope), multispectral and spectroscopic applications.
Phase 2 Code deliverables: Exposure control programs: (June 16 to July13 Evaluation)
Week 5 - Optimization for longer exposure times, requires multiple settings to be adjusted and looping thru different settings.
Week 6 -Manipulating RGB/HSV image, masked measurements and selection program. Region of Interest selection is useful for spectral data since you can record a slice of the 3240 x 2464 image.
Week 7 -Intensity and histogram maps
Week 8 -Prepare documentation/final evaluation
Phase 3 Code deliverables: Spectroscopic Applications (July 14 to Aug 14 Evaluation)
The last phase will concentrate on applying some of the general processing options developed earlier to specific applications. Applications to be considered are NIR/G/B microscopy and spectral calibration.
The general approach for spectral calibration is to see if multiple views at different exposure settings and analyzing data from the higher spectral orders could be used for spectral analysis. I thought I would work on this since most everyone in computer vision works stuff like face recognition but few are working spectral recognition. A major challenge is overcoming spectral overlap issues in higher orders.
Week 9 -Controlling dynamic range with feedback from CV processed spectrographs
Week 10 -Spectroscopic calibration for (near) real time spectroscopy
Week 11 -Transmission spectroscopy experiment – objective is to extend the free spectral range
Week 12 -Prepare documentation/final evaluation
-Color detection – generates mask/ calculates HSV colorspace range. Primary contribution was the use of trackbars for the user to easily select color range(s). (HSVcolortrackbar.py)
-Exposure control matrix - Developed to get a systematic way to adjust Raspberry PI manual camera settings. My real first experience with the multiple tunings necessary for long exposures (up to 10 sec) and low frame rate. (exposuremosaic.py)
-Microscope grid overlay with calibration. Simple CV geometry functions permit scale reference. Amazed that this works as well as it did. (videooverlayfinal.py)
-Insect infrared microscopy – Simple contrast adjustments permit enhanced detail. Effort demonstrates need for better focusing options and real time histogram optimization.
In addition to the research notes, additional ‘test’; code was tested on Fourier masks , real time measurement, histogram equalization and region of interest selection .
On setback is that I have been unable to use 10bit RAW Raspberry Pi camera data. I have tried different processing options but they all need color and lens corrections. I have started work on HDR techniques as an alternative. My initial review is that HDR may operate over a larger dynamic range than a 10 bit RAW image.
Phase 3 Code deliverables: Spectroscopy and Microscope Applications (July 14 to Aug 14)
Goals were outlined in prior posts but are summarized below for convenience:
‘The last phase will concentrate on applying some of the general processing options developed earlier to specific applications. Applications to be considered are NIR/G/B microscopy and spectral calibration. The general approach for spectral calibration is to see if multiple exposure settings and analyzing data from the higher spectral orders could be used for spectral analysis’.
Week 9 –Real time spectrographs Week 10 -Spectroscopic calibration for (near) real time spectroscopy Week 11 -Transmission spectroscopy experiment – objective is to extend the free spectral range Week 12 -Prepare documentation/final evaluation
Reply to this comment...
Thanks, this is very helpful for the eval writing. Much appreciated!
HDR imaging, manual camera controls and Elodea microscope observations look at different ways to extend dynamic range and display images.
NDVI was explored in NDVI microscopy and NGB-NDVI video optimization. In addition, a NDVI LED Simulator was proposed as a potential method to calibrate NDVI data.
Software also was developed to display real time spectral data output. Work to separate orders and conduct calibration by colormaps were frustrated by signal crossover between RGB channels.
In general, the programs demonstrate a transition from ‘snapshot’ imaging to processing data at video rates with manual control of camera settings. The ability to adjust image properties in real time provides a quick way to control image quality and optimize for different computer vision tasks. While resolution is limited in real time processing, the same settings can be used for higher resolution images which can then be processed by image sequencer (https://publiclab.github.io/image-sequencer/examples/#steps=).
My only regret is that I didn’t have the time to look at problems discovered along the way. Issues like real time flat field correction and keeping focus from visible to near infrared are two annoying areas that deserve a second look. It may be worthwhile to look at how computer vision could compensate for optical problems as well as provide different lens options.
My last comment is I would like to thank all my mentors @amirberAgain, @warren, @icarito as well as those who provided help, comments, and encouragement @cfastie@liz. I learned so much but I also know I have much more to learn. While the programs provide a way to get started with real time computer vision techniques for Public Lab kits, possible future applications are unlimited.
Reply to this comment...
Hi, @maggpi - i'm just going through this now, thanks! Perhaps the .docx file could be uploaded as a .txt or .markdown file -- it's displaying oddly when I click the link.
Looking great so far, thank you!
Reply to this comment...
In fact, as you've provided this link for your final work product, I think reformatting your .docx file as README.md (markdown) and including it in the root folder of your GitHub repository would be great, as that will appear as the default on https://github.com/MargaretAN9/Peggy/ -- would you be able to do that in the next day? Thanks!