Public Lab Research note

GSoC proposal: Computer Vision enhancements for Raspberry Pi based Public Lab Science Projects

by MaggPi |

Read more:

About me

18 year old aspiring computer programmer; interested in computer science and how computer vision can be applied to automate spectroscopic analysis.

-Currently enrolled in high school/ Northern Virginia Community College with goal to transfer to a University (TBD) in 2018 Location: Northern Virginia, United States

Project description

The project would provide a series of modular python programs that could be used to support different Public Lab imaging and spectrum measurement toolkits. The programs enable near real time OpenCV computer vision (CV) measurements of images or spectra. The CV measurements can be used to improve instrument performance (feedback that controls lighting amplitude or camera array exposure times) or assist calibration by subtracting reference images/spectra.
General objectives include: • increasing the scope and usability of Public lab Spectral Workbench and spectrometer kits • collecting software techniques from Raspberry PI, computer vision and Adafruit blogs into a single source that can be applied to science projects • provide code that would automate data collection and present data in a visually appealing manner • optimize the use of Raspberry PI visible and NOIR camera in Public lab spectrometer kits • provide a foundation for new spectroscopic applications using python/opencv image processing libraries

Abstract/summary (<20 words):

Computer Vision enhancements for Raspberry Pi based Public Lab Science Projects


The problem I would like to solve is make camera integration easier into spectroscopic systems. Current systems are faced with multiple problems ranging from poor dynamic range, limited annotation and manual processing.


Community Bonding period (April 23 - May 14)

• Familiarize with published software on PublicLab site/check operation.
• Configure Raspberry Pi 3 B+ with latest software • Identify project camera(s) and spectrometer(s) • Determine documentation requirements • Connect with mentor(s)

Phase 1 Code deliverables:basic camera functions: (May 14 to June 15 Evaluation)

• Week 1 -How to annotate picture/video • Week 2 -Camera resolution options and memory implications • Week 3 -Optimization of camera settings • Week 4 -Prepare documentation/final evaluation

Phase 2 Code deliverables:GPIO and Computer Vision(CV): (June 16 to July13 Evaluation) • Week 5 -Controlling light sources for imaging or spectroscopy • Week 6 -Manipulating RGB images/Histogram (masked) measurements • Week 7 -Time Lapse CV/Frame data annotations • Week 8 -Prepare documentation/final evaluation

Phase 3 Code deliverables:Spectroscopic Applications (July 14 to Aug 14 Evaluation) • Week 9 -Controlling dynamic range with feedback from CV processed spectrographs • Week 10 -Spectroscopic calibration for (near) real time spectroscopy • Week 11 -Transmission spectroscopy experiment – Is it possible to use a transmission grating and CV for a low cost imaging spectrometer? • Week 12 -Prepare documentation/final evaluation


-Open CV
-Raspberry Pi 3 B+ w/camera
-Transmission grating
-Public lab spectrometry kits
-Documentation requirements

First-time contribution

An example of prior work is available at the public lab website(s): link text


Almost all my experience is from Raspberry Pi programming. I have taken one JAVA course on programming at a Community College. I have also taken 4 years of computer graphics/art in high school and become proficient in Adobe Illustrator/Lightroom/Photoshop.


An example of past team contributions is my work for a NASA program called VASTS (Virginia Aerospace Science and Technology Scholars). The program selects 30 high school students to work with NASA engineers to design a hypothetical human Mission to Mars. I lead the strategic communications team that developed a consolidated system design across the different engineering teams. You can see the final design in a short video at:


I understand that this proposal is not listed as a Public Lab GSOC topic. I just can’t pretend to be interested in topics such as ‘email or editor enhancements’. But, if you’re talking about the chance to help create new ways for software to automate scientific measurements then I am all in. I have been using Raspberry Pi for my science fair projects since sophomore year. My current interest is OpenCV computer vision measurements that can be used to enhance imaging and spectroscopic applications. With Public lab guidance (and equipment) I hope to continue my passion and develop a better understanding of software algorithms and scientific instrumentation


I would like to provide open source code that I wish was available when I first started working Raspberry PI based science projects. I believe that code would be useful to Public lab and Raspberry PI communities.


-High motivation given the alternative is a summer job at the mall. What’s could be more fun than tinkering with computers, cameras and gratings all day!

software gsoc soc gsoc-2018 soc-2018 soc-2018-proposals gsoc-2018-final



Hi @Maggpi, I guess we should talk about GSOC, when/how do you want to connect? @Warren, is there another way of communicating I should be using? Amir

Is this a question? Click here to post it to the Questions page.

Hi, I am not really sure how this works bit I think GSOC discussions moves over to github,, which is replicated to Gitter. In any case, really impressed with your opencv work, and wondering if you could recommend any opencv teaching guides or ‘how to’ manuals?

Is this a question? Click here to post it to the Questions page.

Thank you for the kind words! I was very impressed with some of your posts on microscopy! The two resources I personally like best are the openCV tutorials and PyImageSearch. [Try looking at the older content to begin with, as he now works a lot on ML/AI].

@amirberAgain we have a chat room on gitter and also on slack. You people can communicate to each other or us on these platforms. and if you people are interested i will send you the invitation link for the same. Thanks

Hi @MaggPi! Actually the easiest way to reach the PublicLab chatroom is thru . While we support bridges to other chat networks, our "home" chatroom is hosted by and reachable with Riot - a libre messaging system. I will be helping with co-mentoring - also I am PublicLab.Org's sysadmin. My strengths are systems programming, web, and Python. I hope you will have a great summer project! Please ping us if you feel stuck at any time. Any community channel is good!

Hi, @maggpi - could you paste in your GitHub project link here? Have you thought of a name for your project, even provisionally?

Looking through your materials now! Welcome!

Is this a question? Click here to post it to the Questions page.

One change seems to be the desire to collect images by microscope. I have the equipment for this but need advice on what should be test images? I assume a calibration slide is appropriate but it’s also not a very fun reference and has no color. It may be a good time to try to measure something readily available (like a dollar bill) so that anyone could have a reference without paying $10 for a calibration slide.

Great question -- there are some ideas here; -- i like looking at things that move, but they're hard to keep alive :-)

One concern is what is the best spectrometer for phase 3. After reviewing the different Public Lab kits, the lego spectrometer seems the best choice. Lego was selected since it was compatible with the Raspberry Pi camera and I could possibly share results with cedarlodge(from Public Lab),, So is this OK?

That's great, yes. And if @programmer1200 is willing, maybe we could send you some of the attachments he's working on? ( #16042) @programmer1200 if you're still interested in swapping some with us, and could include a second set, we'd love to forward it along. But either way - we can just forward ours along too.

What is the policy for getting access to public lab kits and parts? Do I need to purchase or can Public lab provide? I would like to use 2 lego kits (one for visible and one for vis-IR ). Note I plan to use my own v2 cameras and Raspberry Pi. (Please tell me if there is there some reason why the lego kit spectrometer comes with v1 instead of higher resolution v2?)

Sorry, we actually already sent you things! I hope there's enough there - we can send you a second lego set. The camera thing is just a cost issue - we know people who want higher quality cameras can get them on their own... they're expensive for us to carry in quantity. But we'll probably start carrying both versions!

Hope this is helpful! Reading your revisions above now.

Is this a question? Click here to post it to the Questions page.

OK, revisions look good! As @amirberagain mentioned, maybe expanding your timeline/checklist a bit could help. I'd like to see an outline of how you plan to organize your code; maybe pointing at a well-organized library as an example?

In JavaScript, we're trying for relatively short files, which are require()'ed together, and named consistently -- this tends to be good for other people to be able to quickly read through, find relevant parts, etc. Here's an example:

I'm not as familiar with Python but I imagine similar coding style could be achieved! If there's an example project you'd like to aspire to, structurally, please share!

Once your GitHub repository is set up, I think a good step will be to make a planning issue there with the tasks you have in mind. See these examples:

I hope this helps. Thank you very much @MaggPi!

Is this a question? Click here to post it to the Questions page.

I think you are just right @warren. possibly one of the outputs of the initial planning phase should be a list of functions (and objects?) which could then be mapped out and divided in a logical way, maybe even some state machine flows? Calling functions and objects from other Python libraries is not a problem and should be desired, basically, you just write a .py file and import it to another one.

Is this a question? Click here to post it to the Questions page.

Hi, this is first test of the chat in chatroom. I was wondering if there is any preference for Open CV python virtual environment vs a non virtual environment. I understand the virtual environment helps multiple programs use the same libraries but it also seems like another rthing to go wrong. Also, AmirberAgain is not listed in the chat name

HI @MaggPi With regard to Python virtual environments you shouldn't worry about the they are a convenience for the developer but make no difference on the actual code. They exist so that you can mix and match library versions without messing with your operating system's Python. For our purposes it's fine to use the system's Python interpreter and libraries. You can install libraries as your current user with pip install --user library.

If your program uses external libraries be sure to create a requirements.txt file.

Also on IRC - some of us leave bots but are not at the keyboard. It's a way to get a backlog when we come back. A similar functionality is provided by Matrix.Org/ - it's good to leave your question and stick around until somebody answers! Github issues are also good for open discussion.

Hey @maggPi and @icarito, let's try to sync up and try to have a call in the coming days. I was thinking Thursday 15:00 EST (the same time as the open call starts), how does that sound to you?

Is this a question? Click here to post it to the Questions page.

icarito amirberAgain

My IB French exam is Thus and Fri. Is is possible to set something up Friday afternoon? I know you are looking for more details so please read below.

This post reviews GSOC first month activities and goals/programs to be developed for the second month.

First month activities:

-Reviewed public lab/spectral workbench web site. – Have tried to review all sections related to microscopes /spectrometry and multispectral imaging.
-started building lego spectrometry, multispectral imaging – having trouble with consistent measurements. Tried different designs to see what works best. Working on adjustable camera mount for spectrometer.
-prepared visible/IR table top camera
-prepared several notes/questions, see: -began learning spectral workbench calibration software -purchased 2 raspberry pi 3B+/installed opencv. -setbacks- New raspberry pi gave blinking light power supply error on start-up and had to be returned. Damaged a pi camera while trying to mount in spectrometer.
-established contact with mentors -received public lab equipment


The second month will develop a series of programs/functions that will adjust cameral settings and display images with opencv. The programs will be tested with imaging (microscope), multispectral and spectroscopic applications.

-Programs will be based on the Raspberry Pi visible, NoIR cameras. This will permit controlling camera functions by picamera. The general goal is to establish a two way link between the measurement, camera settings and computer vision analysis so the programs can be used for the next phase.

-Programs will be not be written for video analysis. This will provide time for Raspberry Pi to process opencv tasks at high resolution. Video will be used for camera set-up and spectrometer alignment.

Program Checklist:

Phase 1 tasks concentrate on controlling and recording different camera settings. -annotation of jpeg picture (time and file name) from pi camera -collection of RAW image data from pi camera (this should provide 12 bit range, so far all I have been able to collect is 8 bit jpeg) -annotation of picture with camera settings (displays auto exposure settings) -collect and process RGB or other split channel images (for example: -prepare routines for side-by side display of multiple images (permits comparison) -cycle routines that captures and display side by side annotated images at different camera settings

General Comments:

Biggest challenge so far is working with opencv. I worked with simplecv before and opencv is ---- not so simple. One example is simple tasks such as displaying side by side images often get errors. I know the opencv sidebyside command is hconcat but it only works if the arrays are formatted the same size. So this means you need to think how to scale all the images beforehand.

Realize now that getting this stuff to behave isn’t easy.

Questions: What type of format is required for the June 11 evaluation? Is there some type of grading criteria like how many lines of code were written?

Is this a question? Click here to post it to the Questions page.

hey @Maggpi, I was sure the exam you mentioned previously was last Thursday - sorry for that. Best of luck with your exams, Let's sync start of next week? As for OpenCV - this is not a hard thing to do, use the resize as explained here:

I guess @warren will be able to share experience on grading. If it's based on the number of lines of code we will need to change the criteria...

Is this a question? Click here to post it to the Questions page.

You must be logged in to comment.