Hi Sreyanth,

I really like the idea of combining information from the intensity graph of a spectrum with information about which colors the peaks are in order to recognize spectra. That should have great power to identify CFL spectra. If that procedure works well, it might be possible to include a few other types of common emission spectra, like mercury and neon. In fact, the two peaks now used to calibrate CFL spectra are both peaks of mercury, so it might be important to distinguish pure mercury from a CFL which also has distinct peaks from terbium and europium. If your algorithm detects those two mercury peaks, it could look for the big red europium peak at 611 nm to confirm that it is a CFL. In fact, calibration will be much stronger using that red peak (611 nm) and the blue peak (436 nm) because they are farther apart than the two mercury peaks used by default at Spectral Workbench. Then you will have to decide, if you have identified three peaks, whether to use all three in your calibration. This might require non-linear calibration, but most spectrometers probably need that, and its just math, right? Once you have pinned down three peaks in a putative CFL spectrum, you could even look for more and really know how much massaging each spectrum required to calibrate it. Then you could report to the user "Congratulations, this spectrum required only a second order polynomial to achieve excellent calibration." And then maybe "As long as you don't touch your spectrometer, subsequent spectra will be spot on!"

Chris

Hi Chris,

Amazing. Thank you for detailing out what can be done. So, as far as I can understand, instead of finding our 2 peaks (which always makes the calibration a linear one), we now need to find one more (the big red line at 611nm).

Now, we will have 3 peaks with us. We can use these to calibrate the spectrum with some non-linear calibration (mostly a 2nd order polynomial as we have only 3 peaks). The higher the number of peaks we consider, higher would be the order of this polynomial.

Also, could you throw some light on few must-have features for offline SWB? I am planning for a stable release and it would be better if we included some really required features in both online and offline SWB.

Sreyanth

Sreyanth,

I don't know the issues associated with using SWB offline. I always work offline -- taking still photos of spectra and later uploading them to SWB. So my workflow might be different from most.

I think you noted elsewhere a problem with using color to identify spectra. In many spectra, the highest peaks are overexposed in at least one color channel. So the channel that primarily determines which color the peak is has only the value 255 for a range of wavelengths and provides poor information about the location of the peak. You still know the general color of the peak (even though it might look white) because you know which channel is overexposed, but you have to determine the location of the peak from another channel that is not overexposed. So this is doable, but tedious. When an algorithm has worked through this process, you know a lot about the quality of the spectral image. It might be good to report this to the user and in some cases maybe encourage a repeat attempt with better exposure. Maybe each spectrum could be given a quality grade based on exposure and dynamic range to educate people about what makes more useful spectral data.

Chris

Thanks for your informative comment Chris.

Its interesting to know that you take still photos and later upload them to SWB. It would be great if we get to hear more about this, so even others can try it out!

Yes, there are chances that one or more channels may be overexposed. If only one channel is overexposed, say the value is 255, we can use the other two channels for peak detection purpose (as we know the pattern of how these blue, red and green bright lines should look in those particular non-overexposed channels). If more than one channel is overexposed, then we can alert the user to re-attempt with better exposure. I would like you to help me out to gather these overexposed spectrums from SWB database, so that I can come up with a better algorithm.

Also, how shall we go about giving a quality grade to the spectrum? Which dynamic range are we looking at for this purpose?

Sreyanth

It is easy to find CFL spectra that are overexposed. A quick survey suggests that maybe > 3/4 of the CFL spectra at SWB have at least one peak with the primary channel flooded (select More_Tools/Toggle RGB). This is not surprising because the diffraction pattern of CFL has very bright peaks with black areas between, so it takes a good camera with high dynamic range and very careful exposure to avoid overexposure while still capturing the fainter peaks. So this will be a persistent problem, but in most cases a secondary channel has good peak information.

It might be possible to analyze a spectral image of CFL and determine how many bright peaks and faint peaks have acceptable information. If three or four of the brightest peaks and four or five of the fainter peaks can be discerned, the spectrum might deserve a high grade. Maybe just reporting how many of the 16 major CFL peaks are recognizable will be a good metric.

The diffraction pattern produced by the Public Lab spectrometers is very detailed and contains a lot of information. The obstacle is making a good photographic image of that pattern. The beauty of the spectral patterns captured me, and I just use a Canon Powershot to take photos of them. This allows higher resolution but most importantly greater dynamic range and easier control of exposure. Even so, all of my CFL spectra have overexposed channels.

I think I will have to learn more about the spectrometers and what factors lead to over exposure and the patterns to determine those reasons automatically. This way, we can try correcting the spectrum, may be?

Sreyanth - can we or shall we send you at least a foldable spectrometer so you can test this stuff out? Do you have a webcam you could use with it?

Sorry, i haven't had more time to respond to this post; just trying to get through all the GSOC postings and am traveling and working on the road this past week. Forgive me! I'll see if I can jump in and give more substantive feedback soon.

Sreyanth - Yes, off-line is a valuable capability as it frees the user from the internet and it allows independent capture and retention of spectral data (i.e. capture 50 and upload 3 good ones.)

In searching for peaks (for frequency cal), there are at least 3 approaches: 1) center = max peak value, 2) center = mid-point between sides of peak at -3dB from max peak, and 3) the centroid of the peak. Based on the variety of the spectra uploaded, I'd suggest #2. Remember, image processing doesn't really apply; this is just a 1D waveform.

I'd suggest that with just 2 data points (436nm and 611nm) will not allow a non-linear calculation. Even 3 points is insufficient. Remember that with most of the spectra uploaded, there is clipping which will distort at least one peak. That alone will cause more error to a simple linear approximation that what might be corrected for with a non-linear factor. However, that said, if the non-linearity could be observed using many peaks with a very careful setup, that non-linear correction should apply to every device -- i.e. there would be no need to do non-linear correction on every spectrum.

A much bigger factor is the general failure to capture valid spectrum data to start. I would recommend you take a look at my HDR2 note: http://alpha.publiclaboratory.org/notes/stoft/03-09-2014/hdr2-using-over-exposure-to-your-advantage and think about the source data as everything else comes from that. The source data is of little value if it contains gross errors due to poor SNR and heavily clipped RGB channels. Throwing away most of the data from the full spectral image band and then expecting to extract it later using some alternate technique just isn't possible. Once you throw data away, it's gone. Period.

How about a project with 3 goals: 1) Full-band data extraction and clipping correction, 2) Linear-Only Auto-Cal (from the results of #1) of only CFL spectra and 3) Off-line data collection to view #1 (and maybe #2) but only upload either the full image or an auto-cropped 'spectral band' segment of the image. Admittedly ambitious, but at least each element is self-contained.

Cheers, Dave

OK, my thoughts -- very lengthy! I had a while on an airplane to really dig into this. Also Dave posted since I did, so maybe there's some overlap with what he said. Take a deep breath, here we go!

Auto digitization: along with auto-calibration, what about auto-detection of the clearest-lit region of the video output to use -- perhaps relating to Dave Stoft's recent work on "HDR" spectra, and/or reading/averaging various lines to get a clearer, less noisy, and well-exposed spectrum. It's worth looking at if existing overexposed spectra do have enough data above and below the chosen cross-section to make a better-exposed calibration. But often spectra images are curved, so it may not be a simple issue of moving up or down... but i guess if the sampling line is adjusted in the capture interface (and recorded in the device profile) then it would be OK.

While using some advanced techniques to get well-exposed spectra would be great, it could get quite messy unless we have a good systematic way to record how these digitizations of images were performed. This could relate to the idea of "recipes" which is written in some Github issue (remind me to look for the issue) where the manner in which the graph is generated from the spectrum is recorded in the device profile as a sequence of coordinates and such: "x1,y1,x2,y2,spread" where spread is the # of pixels above and below the line to average in.

Re: #4: will including manual calibrations necessarily improve the automated calibration? could it make an auto-recognition more likely without allowing poor manual calibrations to make the actual scaling of the autocalibration less accurate?

Offline SWB: use localStorage (a local key:value sqlite-backed database in HTML5) instead of local files to store spectra. You can dump the dataUrl into the value, in a JSON object. Chris is talking about the ability to upload images, which is not the same as the live capture interface running offline. People have asked for the ability to live-analyze, calibrate, and use the whole Capture interface offline. Obviously they can't run matching, but you might be able to design your auto-calibration system to run offline. Don't let this distract you from designing a great auto-calibration tool though -- if it works better do to it server-side, that's more important. Key features that don't already exist for offline include mainly:

• ability to store spectra with their calibration, for later easy upload
• some kind of prompt when you reconnect that says "You have 23 offline stored spectra. (Upload these now)"
• maybe... a way to download the spectra images with the calibration data etc. stored in exif tags? or something? Or encoded along the bottom edge of the image as pixels or something crazy, for later upload... but this seems like a waste of time compared to the other challenging - and very important - features we're discussing.

Secure uploading: do you think requiring a user login is not secure enough?

Using color instead of just ratio of distances for recognition: I worry that overexposure (sometimes in all 3 color channels!) or camera variation might make color a poor indicator, but I'm not confident about my doubts either... it could work! My hope was that my original approach of ratio of distances between major peaks could be improved by making it the ratio of distances between each peak and some absolute location. Consider that sometimes people miss near-UV peaks or aren't recording infrared peaks. Also consider that infrared peaks can show up as all kinds of weird colors depending on the Bayer filters. So this is a complex question, deserving of some empirical attempts. Maybe also just the ratios between all possible peak pairs, or something. Surely there's good literature on this already out there, perhaps for general pattern matching, or for matching bar codes.

Non-linear calibration: I think this is a lower priority, at least insofar as it should not stop fast development of a linear calibration first. This could be related to the recipes idea, like "x1,y1,x2,y2,x3,y3,spread" in terms of how it's recorded and displayed

Jeff, Specific to data/analysis, post-processing can only as good as the source data. This is why I'm focused on first getting that data; other things can, and should, follow. It is important to not throw away image data; at least not until the most accurate spectral response curve has been extracted.

Frequency calibration is based on identifying peaks, so if those peaks are skewed because of RGB channel clipping the calibration will be affected. Remember also that not all peaks are simple, symmetrical, smooth shapes so clipping will shift the peak center, dependent on the shape of the peak -- clipping distorts this.

True, some images have curves which are a result of the diffraction grating / slit. However, that can easily be detected -- IF you look at the entire 'band' of data. However, where there is some curvature, it will be uniform (the physical area of the grating is very small) so a first-order approximation should not be difficult to generate; and used to compensate the image data.

The effect on the final spectral curve is primarily on the sensitivity curve -- which is primarily an intensity factor, not a frequency effect. ie. variability in RGB detection sensitivity won't have much impact on frequency cal -- but RGB clipping will.

So, whether on-line or off-line, if the entire spectral band image area is used to extract the final spectral response, you will go a long way toward making posted response data much more independent of device / user / setup. It is always important to put the source data handling as far toward the front-end as possible for the best SNR and minimize error propagation.

I used a random sample 640x480 full-image spectrum from the original PLab webcam but admittedly, it was my setup and my own slit. If you want to point to or send me an RGB clipped, curved full-image spectrum which you think is a good test case, I'll tweak my algorithm and post the results to my HDR2 research note.

Cheers, Dave

Hi Dave and Jeff

Thanks for your feedback. I really liked the idea of using over-exposure to our advantage to create a high dynamic range spectrum. Though I personally love to work on these features, a recent proposal drew my attention, which overlaps to a large extent with my proposal.

Pascal's proposal ( http://publiclab.org/notes/PascalW/03-11-2014/gsoc-proposal-spectralworkbench ) is well researched and he started working on the problems we are discussing here, in this thread. I found this proposal to be promising and I don't want his work to go waste.

This leaves me with only Offline SWB to work on ( :-O ) and some other less-important features. Seems like I will have to explore other ideas and propose a much useful contribution to any of the Public Lab projects.

Sreyanth

I don't think there is too much overlap in proposals, and I'm going to encourage both of you to widen your scope to make your proposals unique but helpful counterparts.

(repeating what I wrote to Pascal) There's plenty of work to be done in SWB! My initial thought is that Pascal's interest is in exposure, clipping, and sample row selection, whereas you are interested in calibration. Perhaps if each of you affords the other space and coordinates where your code "touches", this can work... if you each take on another sub-project as well, we can get an awful lot done this summer!

One area is Macros. I think the macro system could make it much easier for people to add new functions to SWB, and some better "plumbing" of that system, as well as the ability to access/run macros both at capture time and analysis time, would make a huge difference. Other interesting things macros could offer:

• more powerful "matching" features
• more informative tools for displaying results -- easy ways to highlight a region of the spectrum display, to add a % indicator at a given wavelength, to indicate quality of match to a given reference.
• related to the above -- a way to easily create a macro for a given purpose, like "matching a type of oil from a set of references"
• a macro which helps "troubleshooting initial spectrometer setup" (which might make recommendations about exposure, calibration, intensity, light leaks).

Another feature which could be interesting is one which automates collection of spectra periodically (every 5 minutes, every minute) and helps you search for events over time, like heavy metal spectra in a smokestack flare, or something. This means of collecting data 24hrs a day could easily overwhelm the current system, but some smart adaptation of SWB could make this a powerful analytic technique. Maybe it would only save a spectrum if a certain condition is met. Maybe it would save everything locally but only prompt you to upload unique events -- rapid changes in the spectrum or something. Lots of possibilities related to http://publiclab.org/wiki/refinery-watching

After all, Sreyanth, you did propose quite a few projects for this summer -- even if you let Pascal work on sample-row-selection (digitization) you'd still have calibration, offline SWB, and potentially macros or 24-hour monitoring to work on. And, you two could help each other problem-solve.

Sreyanth,

Don't give up so easily; I think Pascal has described an enormous amount of work. I'd suggest some collaboration is warranted and everyone would benefit as a result. I'll add a few comments to his note.

Cheers, Dave

Hi Jeff and Dave!

Thanks for your timely feedback. I actually cloned, setup Infragram and started looking at what can be done (as the deadline is too close!). Since, I realize that there are still some great features I can work upon, I would like to modify and cleanup the proposal.

I will better work on the following:

1. automatic calibration (Lets try developing a functional linear calibration module first. And then extend it following what Chris and Dave suggested -- the non linear model)

2. Macros for SWB.

3. I also strongly believe that the refinery watching feature can be implemented using the macros. Can't we, Jeff? Lets put the macro system developed in #2 to work!

4. Offline version of SWB. Lets leverage the HTML5 additions as Jeff rightly pointed out. But I was wondering if that would slow down the system (a huge dataset right? -- so proposed using local files). Will do some reading and get back to you!

My thoughts (which I didn't post previously): I think we need to provide an option for manual calibrations too, at least in the initial phases of development. This way, we can analyze such manual calibrations and find in which cases our module is failing.

Also, I don't think only ensuring an user is logged in is sufficient for uploading the offline spectra. I am afraid there may be some robot attacks which may result in Denial of Service. So, lets use the CSRF tokens to make sure only valid spectrums are uploaded. To make sure this system is secure, we will need to develop a new, faster and secure way to upload the spectrums.

Offline SWB: I have a feeling that we need to use the SWB API to the fullest for this project. So, for this, I would like to refactor a part of the API code and document it where necessary. This makes our approach much easier. Also, the entire macro system can be used here! Isnt it?

So, this should actually let us use offline SWB for refinery watching, once developed completely.

Thanks Sreyanth

UPDATED the proposal. Please have a look and let me know if this is Okay!

Thanks

Sreyanth

Sreyanth, Jeff,

I may not be up on the latest strategy for the code projects, but I'm still sensing that some concepts are diffuse. Performing frequency calibration can only be done AFTER there is a quality spectral response from the RGB data. Therefore, extraction of this response is part of the front-end. This also help the freq cal process since it can assume there is a valid response to observe -- and neither need nor want to look back at RGB data. Having a quality spectral response, then allows easier searching for peaks to decide if it might be CFL (..or other types could be added later).

I think that any off-line user would expect automatic RGB to spectral response and freq cal because anything less is near useless for off-line work. This also encourages users to experiment and only upload when they have something they think is useful and tested enough to share -- which reduces the amount of low-quality or useless spectra for sorting.

This then suggests that the algorithms need to implement the same concepts on and off line -- though the coding can be different. However, it seems likely that if both are browser based, then they should be quite similar. I'd suggest that this summer's work might therefore consist of 4 elements:

1) Modularize SWB for both on and off line use with common modules as possible,

2) re-write the front-end to auto-locate the spectral band, extract the full-band RGB, correct for curvature, HDR to extract data from clupped regions and produce a quality spectral response,

3) Using the results of #2, auto-recognize CFLs and auto-cal (linear),

4) Provide both on and off line use of the freq-cal data ( a device only needs one calibration so only one need be stored on/off line) and provide a way to upload calibrated spectra.

I'd guess that this could keep two motivated students very active and, if finished, would provide a huge step forward in usability and accessibility.

Cheers, Dave

Hi Dave

Sorry. I somehow missed this comment.

Yes, You are right with your steps to be taken. And, I think I planned to take the same steps (along with designing modules) so that independent and effective development can take place.

I am looking into the Macros module. I think we should be exploit it to port most of the required functionalities for both off and on line use.

As usual, you are great with your insights.

Thanks, Sreyanth

I echo Dave's thought about ordering, and agree that it means we'd prefer a client-side implementation of both auto-extraction of spectral response and auto-calibration. I left a question on Pascal's proposal to that effect too.

the refinery watching feature can be implemented using the macros

Yes!!!! I agree. Also about your CSRF token idea. Another thought I had was that we could use the time dimension differently for long-term monitoring... we could have each row of pixels represent a single scan, at 5-minute intervals, or whatever. I dunno... just thinking of how to reduce the need to store whole images every N minutes. And Dave is right about this too -- if we have spectrum extraction (of unclipped data) client-side, on-the-fly in javascript, we don't need to store the whole image for long-term monitoring. We can store just the good data we collect every 5 minutes. I think that's an acceptable amount of raw data to store... Dave?