That's a wonderful interface. I calibrated many times because it was fun to do.

The interface choices you made work really well, but you might be interested to know that I would have taken a different approach:

1. I would have allowed the user to stretch the new spectrum to match the calibrated reference, not the other way around.
2. I would have used the spectral graph trace of the reference instead of the photo of the diffraction pattern.

But it certainly works well the way it is now.

The Snowy Sky spectrum is very clear, but not as accurately calibrated as a calibration reference should be. It was calibrated with a macro by using the 405 nm and the 650 nm lines, so those are very accurate, but most of the other lines are not. For example, the 436 and 546 nm lines used in your calibration procedure are both more than 5 nm off. This is the spectrum that I recalibrated with a third order polynomial using 14 peaks. The final calibrated result exists in Excel but could be transferred to Spectral Workbench. Of course that is just the data, not the photo of the diffraction pattern. Somebody with MatLab or something could probably apply the polynomial to the image.

SW 2.0 looks very awesome.

Thanks, Chris, for the encouraging feedback. A lot of thought went into this, and although in the first version you stretched your spectrum, it was easier to do it this way because:

• here we don't need to move both the graph and the image, so it's smoother (less CPU)
• here the thing you stretch is at the top, and there's no inclination to put the example reference between the spectrum and its graph
• the eye more easily distinguishes color order and balance in the image than in the graph, in my opinion
• a graph line would've overlapped other graph lines, rather than been placed alongside, which I thought would be visually more confusing (i.e. what's "on top"? which line is which?)
• this is also a good opportunity to show folks what a really nice image should look like, so they can aim for that

I'd like to use your polynomial calibration, but without having an actual spectrum image which has been corrected, we can't show the user such an example. Do you know of one? Eventually, we should write a calibration function that can do that too -- actually irregularly stretch the image and corresponding data with as many control points as needed -- kind of like image warping with GCPs. I've modularized the calibration code to accommodate this in the future, but it's a big project. Actually, the whole ToolPane system is something designed so that other folks can publish their own ToolPanes (like different calibration methods) as user-generated utilities. That'll be covered in the upcoming API reference in the wiki.

The RMSE 'fitness' function, however, can be used with numerical data, so if you can publish your polynomial calibration as a CSV, I'd love to use it.

For starters, I'd like to use a more precisely calibrated image reference. I need an image with the following dimensions:

• overall image pixel width (1818px in the current image)
• distance in pixels to (middle) 436.6nm blue line from left edge of image (586px in current)
• distance in pixels between blue and (second) 546.5nm green line (538px in current)

These are used in the procedure here: https://github.com/publiclab/spectral-workbench/blob/master/app/assets/javascripts/spectralworkbench/SpectralWorkbench.UI.ToolPaneTypes.js#L310

How was the interface speed on your computer? I am hoping to optimize it to speed things up, but was it usable?

Here is a text file with the 14 peaks I used to calibrate the CFL spectrum (in nm):
14CFLpeaks.txt

Here is a csv file of the intensity at each wavelength of the calibrated (3rd order polynomial) CFL spectrum:

3rdOrderSnowySkyCFL.txt

Here is a jpg image of the calibrated diffraction pattern of the Snowy Sky CFL spectrum (the emission lines are where they should be according to this source):

I think the jpeg has the dimensions you requested.

The slider is definitely a little jerky, but it does not affect the function very much.

I did a check on the files I attached to the last comment and they are not as precise as I thought they would be.

A graph of the data file of intensity x calibrated wavelength (pink) overlain on the corrected diffraction pattern. Both data sources are from the Snowy Sky CFL spectrum.

The reason for the imprecision is that the image of the diffraction pattern was corrected differently than the data series of emission peak intensities. The diffraction image was corrected manually in Photoshop by moving individual emission lines or groups of lines so they matched an overlay of the known emission lines of a CFL. So the lines in the image should be within a nm or two of where they should be. The data file is the result of applying a 3rd order polynomial correction (based on the same known emission line data) to the intensity series. This should also be accurate to within a nm or two. But the error for the two results is not necessarily in the same direction for any particular line, so there is a discrepancy of a nm or two for some lines.

This is probably good enough for your current application, but be aware that these data sets have their limitations.

OK, so we have two different needs:

First, a list of known spectral peaks:

• from some known trusted
• maybe an external source, like NIST?
• we only need wavelengths and intensities
• we should use this long-term as a means of assessing how good a calibration spectrum is

I'm going to use the 14CFLpeaks.txt file for this.

Second, an example reference spectrum:

• which we can upgrade later
• which should be viewable alongside existing spectra
• which has as good precision/accuracy as we are hoping users will achieve

For this, I agree that we don't need something to the same standard as the first case. I'm curious though --

corrected manually in Photoshop by moving individual emission lines or groups of lines

How did you do this? With the Warp tool, somehow? Or did you just marquee select lines and translate them left and right?

Also, in the jpg of "the calibrated diffraction pattern of the Snowy Sky CFL spectrum" are the peaks at the same positions, i.e. 586px from left, and then 538px from there, as in the image I used? Or should I re-measure that?

One more thought. Although this would only work for a monochrome image (although we could colorize it for clarity, in theory), we could generate an image directly from the graph data, so it's 1:1. We could even do so from some external source, like NIST or something. Would that be of interest?

Oh, and is the 14CFLpeaks.txt from the Wikipedia reference? If so, it says:

Spectrum with peaks labelled taken with an Ocean Optics HR2000 spectrometer [1] of ambient light provided by fluorescent lamps. Spectrum taken by me (apparently en:user:Deglr6328). The spectrometer appears to be about ~.6 to .8 nm off judging from the location of known peaks. Interpretation of spectral peaks has been done using the NIST database of spectra for mercury [2] and an article on fluorescent light phosphors [3]. This spectrum is not calibrated for intensity.

Which should still be OK as a reference. Although if all 14 are available at http://physics.nist.gov/PhysRefData/ASD/lines_form.html, we could just use those.

OK, I've filtered out the highest NIST peaks, and exported them in plain text. But these are in a vacuum, and for example, the 436.6nm line in the Wikipedia reference shows as 435.8335nm. Is this because it's in a vacuum? Or is 435.8335nm the more accurate number? @stoft, any input on this?

Also, I'm not clear what Hg II means -- an ion? Would we only be seeing Hg I, or would we see both? I'm wondering about the Hg I line at 546.0750nm vs. the Hg II line at 542.5253nm, which seems MUCH brighter, but that's not the case in the CFL spectra we see, or in Wikipedia's Ocean Optics example. Does that mean we aren't seeing any Hg II lines and I could filter those out of the NIST results?

I see that the 14 peaks you've chosen are sourced from the NIST data, not the Wikipedia/Ocean Optics data.

OK, one last issue -- for intensities on the RMSE, I'm wondering if I can just use NISTs relative intensities, stretched to 0-255, or if enough of the peaks are not mercury, and so I should just use the intensities from the snowy sky spectrum. Thoughts?

Correcting the diffraction pattern was done this way:

marquee select lines and translate them left and right?

This was done in the Photoshop file that produced the lead image on the original note http://publiclab.org/notes/cfastie/3-6-2013/polynomial in which the dashed lines are the 14 emission lines I used for the polynomial calibration. These 14 lines are from the right hand column in the Wikipedia source (https://commons.wikimedia.org/wiki/File:Fluorescent_lighting_spectrum_peaks_labelled.svg). There is no guarantee that these known lines are the same ones produced by the particular CFL bulb I used, but the 14 I chose seemed to be pretty unambiguous.

Most of these 14 lines are not from mercury, so knowing the relative intensity of the emissions from all three elements might be hard. The intensity probably varies among bulbs manufactured with different recipes for the phosphor coating. Using the intensities from Snowy Sky is probably the way to go. However, the intensity is also a function of the camera's sensitivity at different wavelengths and will therefore vary among cameras. The PowerShot S95 used for Snowy Sky likely differs from the webcams.

I have never been able to parse the NIST database results. I don't know what Hg II is. The Wikipedia source seems to match what I get from a CFL bulb, so I just rely on that.

I measured the pixels to scale the jpeg to your specifications but there is error involved in deciding exactly where each peak is. You should confirm that it is good enough for your purposes.

we could generate an image directly from the graph data, so it's 1:1

It would much easier to use Spectral Workbench to do the reverse. Submit the jpeg of the corrected diffraction pattern and get a trace that is a perfect match. Calibrate it with a macro using the two endpoint peaks and it becomes the new NIST.

Now there is a new set of references.

A newly corrected image (background) of the diffraction pattern of Snowy Sky CFL with emission lines shifted more carefully to match the 14 emission lines from Wikipedia (dashed lines), and the intensity trace (pink) derived in Spectral Workbench from the new image of the diffraction pattern.

Definitely not traceable to NBS.

https://spectralworkbench.org/spectrums/58584

Hg (II), Hg (I), and Hg (0) are oxidation states of mercury. Hg (II) and Hg (0) are by far the most common forms of Hg in the environment -- Hg (I) is quite rare. However, I'm not sure if it would be prominent in compound fluorescent lights. I was under the impression that CFLs used Hg (0), since it is fairly inert whereas Hg (II) is reactive and will form compounds, but I'm not 100% sure. The Hg atomic absorptions spectrometers monitor at 253.7 nm, but alas, that is outside the range of our spectrometer.

Oh, and clarification for my comment above: oxidation states are simply different common ions. Hg (0) is neutral, Hg (II) is short two electrons, so has a +2 charge.

Hmm, is there a reason Hg (0) is not listed in the NIST spectra? Perhaps Hg (0) does not have any intense peaks? My list is filtered for most intense peaks only. Are you sure about Hg (0)? Wikipedia (which could be wrong of course) states:

Mercury exists in two main oxidation states, I and II. (citing Wang et al, 2007)

There was also some good information here: https://en.wikipedia.org/wiki/Fluorescent_lamp

And finally the mercury-vapor lamp article includes a listing of emission lines, including the ones we use for linear calibration, listed as 435.8nm and 546.1nm.

It notably does not specify which oxidation state, or list a second green peak, which makes me think that perhaps the 2nd green peak is indeed another element fluorescing? The article also provides this image of a mercury vapor lamp with no coatings, so it should be just pure mercury:

I am absolutely positive that the most common forms of Hg are Hg(0) and Hg(II) -- my whole PhD is on mercury, specifically on Hg isotopes. The only thing I can think of for why they might say that in Wikipedia is that oxidation states are officially described for atoms that are in molecules or forming molecules, and Hg(0) is monatomic and inert (mostly), so maybe it is not an official oxidation state? Regardless, everyone discusses Hg(0) as an oxidation state.

Ah, it makes sense why Hg(0) won't have a fluorescence line on compound fluorescent bulbs though: the liquid Hg(0) in the tube needs to be ionized in order conduct electricity!

Oh, cool -- can you link us to your dissertation, as a source? I'm hoping this note can be a sort of bibliography for the calibration code decisions.

OK, so does Hg (0) have any emission lines at all? If not, could that be why it's not listed in the NIST database?

OK - I'm adapting the calibration slider to use the photoshop-corrected version of the Snow Sky spectrum.

I read the blue 436 peak at 210px from left, and the green 546 peak at 743px from left. The new zooming tool is great for this!

Does this look right?

I'm going to revise: 211px and 742px, out of a total of 1390px.

I'm happy to post my dissertation, but it has nothing to do with this (it explores natural Hg isotopic composition and fractionation in the marine environment -- nothing to do with CFLs and fluorescence -- I actually learned from one experiment that is not included in my dissertation that Hg from wastewater treatment plants that sterilize with UV Hg lamps have craaaaaaaazy isotopic composition).

For your reading pleasure, here is five years of my life... ;-) : http://deepblue.lib.umich.edu/handle/2027.42/39366/browse?value=Gehrke%2C+Gretchen+E.&type=author

Cool, thanks! I was mainly looking for a citation for the most common forms of Hg, and/or a clear reference for which is found in CFL bulbs. But awesome!