Show your support for community science. Donate to Public Lab »

# PLab 3 Gain Correction

by stoft | 06 Mar 07:19

stoft was awarded the Basic Barnstar by warren for their work in this research note.

In a previous note ( http://publiclab.org/notes/stoft/02-25-2015/plab-spectrometer-gain-correction ) I used a very common light source (the Sun) as a broad-spectrum source for correcting the PLab spectrometer's gain un-flatness. While that source is free and available to nearly anyone, it does suffer from some variability of environmental conditions like winter, time of day and humidity so it's not always so easy. Still the technique does function well with clear mid-day sun.

A Better Reference: As an alternative, with finding an inexpensive reference standard in mind, I performed the same test procedure using a 4700K 12V halogen from Solux. The raw plot data has some averaging because 'finer' measurement detail, when using a broadband source, is just noise. The results are below:

While not deliberately wanting to specify a brand, Solux does manufacture smooth response, broadband, halogen sources (one specifically at 4700 deg-K) and, more importantly, they publish their detailed measurement data. While it is not a true calibrated standard, it is the closest option I've found.

Uncertainty, Not Error: While spectrometer measurements certainly have errors (values which are not accurate by some measure) those error values are not actually known. Instead, the potential magnitude of those errors, which is the uncertainty, can be specified.

Source Uncertainty: Solux states that, for their 4700K halogen, they specify a CCT accuracy of +/- 200K for a 12.0V lamp with 3-5min warm-up which will not deviate by more than +/- 100K over its 4000 hour lifetime and output illumination intensity will remain within 5%. The Solux 4700K lamp has a similar shape to an ideal black-body curve though it has some extended output toward the infrared and the peak and its peak output is about 550nm whereas a 4700K ideal black-body curve peaks at about 625nm. The curve shape has been published so the only requirement is to calculate the accuracy uncertainty related to the potential difference in output color temperature. Since the worst case for a new lamp is +/- 200K (i.e. no aging effects at that point) then it is necessary to calculate the impact of shifting the color temp by +/- 200K.

To do this, I generated a black-body curve in Matlab and calculated this effect. The first plot (below) shows an ideal black-body curve variation for the range of 4500K - 4900K color temperature:

Notice immediately that since the PLab spectrometer is limited to about 650nm, there is almost zero uncertainty there; the biggest difference is toward the UV. Below is a plot of both the difference and the percent uncertainty for this effect:

Observe that while the uncertainty rises consistently toward 400nm, worst case is still only about 10%. Compare this to "completely unknown" for the PLab device without any form of gain compensation.

Also note that these curves are all normalized and have no absolute intensity values because the gain-correction is all relative. The Solex spec of 5% for intensity aging has no impact on gain-calibration. For this first-order gain-correction, there are only two uncertainties to include; the peak color temp uncertainty of a Solux 4700K lamp (+/- 200K) and aging shift of that color temperature over 4000 hours burn time. If a Solux 4700K 12V lamp is only used for gain-correction, it will never suffer long-term aging effects leaving the single +/- 200K uncertainty as the only significant factor.

Cautions: Be aware of several factors with halogen lamps: - They get HOT !!! - The lamp must be warmed-up for 3-5 min to stabilize - The 12V lamp requires a 12V supply but that is good because it keeps the voltage regulated so the lamp intensity will not wander after warm-up - A Solux diffuser, which fits the 4700K 12V lamp, may reduce intensity variations at the spectrometer - The spectrometer should be positioned at least 6 feet from the lamp -- the lamp is VERY bright so the distance is necessary - A bulb housing which is designed to minimize the IR emitted from the back of the bulb could be beneficial -- though I've not tested or measured this effect

Practical Gain-Correction: First consider PLab spectrometer's un-corrected response to the halogen's light (Magenta) below. The Solux data is plotted in Black.

Next, observe the correction process. In the graph below, the Black shows the smooth response of the halogen while the Magenta shows the irregular sensitivity of the Sanm webcam. The Green show a 20X magnified view of the correction function and the Red shows the result. Note that the usable measurement range of the PLab device is really 400nm - 650nm. Beyond that band measurement data uncertainty increases dramatically because there is so little sensitivity to those wavelengths.

Now look closer, in the plot below, to see the residual uncertainty after the gain correction is applied. The 400-650nm band becomes more obvious as the Black line show the after-correction uncertainty.

In this case, the in-band error is less than 12% of the peak signal level at 530nm.

Finally, as a test, CFL data was used to verify the process.

Conclusions: Because the 4700K halogen has such a smooth response, even beyond the 400-650nm band, and because there is detailed published output data, the 4700K Solux source is a viable option for PLab spectrometer gain-correction. In addition, Solux specs for this 12.0V halogen lamp include a +/-200K uncertainty (on the exact peak color temperature of any single lamp of that type) which means the primary gain-correction uncertainty is about +/- 10% worst case at 400nm (which decreases to a minimum near 650nm); a substantial improvement over the previous PLab quality of "unknown".

UPDATE 20160108

Using my PLab 3.0 Prototype (with the newer "gumstick" camera) and MatLab interface to acquire 1600x1200 pix resolution images, I re-acquired and re-processed fresh 4700K Solux spectral data to construct an updated broadband Gain-Correction. The combined data is provided in a single plot below:

Here is the key to the plot colors: [ RED ] A CFL wavelength calibration plot using 435.833nm (blue) and 546.074nm (2nd Green peak)

[ BLUE ] The published 4700K Solux source spectrum (intensity is unitless) [Solux 4700K Spectrum Data]

[ BLACK ] The PLab-3 measured raw 4700K Solux spectrum

[ YELLOW ] The raw 400nm-650nm Solux gain-correction curve (intensity is unitless) Note: The data is limited to the 400-650nm range because the curve errors explode at the ends due to the wavelength sensitivity limits of the webcam.

[ GREEN ] The cosine-extended, smoothed 300-800nm Solux gain-correction (intensity is unitless)

[ MAGENTA ] The final gain-corrected PLab-3 measured 4700K Solux curve

The .TXT data files are attached below:

[ 4700K Solux Spec ] Solux4700k_spec.txt

[ PLab-3 Measured 4700K Solux ] PLab3Solux4700K_300-800nm.txt

[ Raw 400-650nm Correction ] GCalPLab3Solux4700K_400-650nm.txt

[ Cosine-Extended, Smoothed 300-800nm Correction ] GCalPLab3Solux4700K_300-800nm.txt

Help out by offering feedback! Browse other activities for "spectrometry"

### People who did this (0)

None yet. Be the first to post one!

This does look very promising. How was your spectrometer set up - blue dye removed from DVD-R grating ? (assuming you used a DVD-R), IR filter removed from lens? Any camera settings ? (auto WB turned off etc.)

I suspect that the IR response could be much better if both the blue dye is removed from the DVD-R and the IR filter is removed from the lens. I have seen some plots for the Bayer filters used in web-cams that show the red filter not as band pass, but as low pass. If the red filter really was a good band pass it would seem you would not need an extra IR filter, so I suspect the red filter often is a low pass. I hope to get around to doing some tests on IR response (with and without blue dye and IR filter) using an IR LED as a light source.

The blue filter does seem to be band pass and responsible for cutting off the short wavelengths.

The sample of the Sanmtech camera I got with my PL Spec 2 kit seems to be particularly bad, I don't know if they are all like that. It's noisy and it seems impossible to get the R,G and B to track at any colour temperature, which means there is always a gain imbalance between the channels. The Videw XJD camera supplied with Spec 3 seems to be a better camera, although I haven't had a chance to run any camera tests yet. If there is a lot of variation in camera performance it could perhaps make applying any corrections more difficult.

Is this a question? Click here to post it to the Questions page.

Reply to this comment...

I've just looked at the Solux lamps. As far as I can see they are halogen lamps with built in blue filters. What a great find.

I've tried daylight incandescent lamps before, but this was a while ago. The lamps I tried were ordinary tungsten bulbs with a blue filter (I have been meaning to try producing some plots for them with the PL spec.) I hadn't realised you could get daylight corrected lamps in halogen. I'll have to get some.

BTW you can also get lighting filters for daylight correction of tungsten filament lamps farily cheaply from Lee filters. Although I guess the Solux with the filtering built in are more convenient. I wonder how much different variation in the supply voltage makes to the lamp spectrum. Lab reference lamps do have regulated power supplies.

Reply to this comment...

This was the Sanm camera of the PLab 3.0 + my own upgrade. No blue-film on the DVD and no IR filter. While removing the IR filter from webcams does add some IR detection, the sensitivity is still poor -- so attempting to gain-correct beyond 650nm is not very effective. The noise is too high so the correction error is too large. I think it is not yet possible to analyze the variation between Bayer filters between Sanm cameras. I suspect that any variation is not with the camera but with the measurement configuration. More test with Sanm camera and more Solex 4700K lamps could build more statistics; then we'd have more data to assign an error value.

Yes, Solux apparently wanted to capture the fine art display market so needed much better halogen lights with broad / smooth spectra. I've not yet found better and none others with published spectral data. Only their 4700K lamp has such a flat spectra which is why it is ideal for this specific use - broadband gain correction. These are 12V lamps and come with a power supply. I've received some stability data, which looks good, but I"m waiting on clarification on the spectral precision -- I''l add it to the note when it's available. Filters don't help. While they can change the overall shape, they do nothing for the variations within a band -- just adds another layer of calibrated values to acquire.

Reply to this comment...

Sigh! well I wondered why I hadn't come across 'daylight' halogen before. It's cos they don't seem to be available in the UK, at least the Solux lamps aren't. The majority of daylight lamps available here are old style tungsten bulbs with a blue coating. The makers don't supply any kind of typical curve and they don't look as though they would be very consistent. There is an Osram 'daylight' halogen, but its a lower colour temperature and the Osram curve for it is an 'artists rendition' and highly idealised. They don't give tabulated measurements, just the curve.

As for the specialist lamp versus filter thing. I was under the impression that the Solux lamps were just ordinary halogen / filament lamps with a built in filter. The output of a tungsten filament lamp, I would have thought, would be fairly smooth. I have seen it said that tungsten lamps are a reasonable approximation to a black body. Never-the-less I take your point that the Solux lamp, which is manufactured to a known performance spec. with published data, is the best choice.

With a tungsten lamp and a Lee filter you'd be trying to build your own daylight lamp and would really need to calibrate it with a commercial spectrometer of known calibration.

Reply to this comment...

Yes, CIR, CCT and other lamp descriptions make knowing what you really get sometimes rather difficult. I have just added a section about calculating there uncertainties for this form of gain-correction and, with the 12V Solux, they look pretty good. As of suppliers, yes, it looks like primarily US: Amazon, Pegasus Lighting, Lamps Direct, etc. Also note that their other color temperature lamps do not have the same smooth spectral curve; the 4700K lamp is unique in this way.

Reply to this comment...

There's been a good parallel discussion of the need for this here: https://groups.google.com/d/msg/plots-spectrometry/ZtLXALzz4Us/F_nXP5sc6TgJ

I proposed that we start thinking (maybe in steps, or even pseudocode) about how to design a macro to accomplish this as a core function of Spectral Workbench.

Dave has very helpfully created a Github issue for it too: https://github.com/publiclab/spectral-workbench/issues/107

What would it look like, step by step?

Is this a question? Click here to post it to the Questions page.

Reply to this comment...

Whatever the reference source, that spectral response curve needs to be saved in SWB. Also, as in another SWB issue I filed, SWB data should be translated to uniform X-axes increments -- nm increments or 0.5nm increments and aligned to integer nm values. This will be the only way to keep plot comparisons and gain-adjust curves easily aligned. We don't need precision as exists now in the CSV files (like 436.435nm) as the spectrometer is, at very best, only ~2nm FWHM resolution. So, .....

Next, any form of gain-adjust reference should be a "smoothed" curve with curve detail below the resolution of the spectrometer. Any curve 'transients' in the reference (even if they are real and known) will only appear in the adjusted output as noise and so are meaningless to the spectrometer's data.

1 - Normalize the smoothed reference data (peak value = 1.0 relative intensity)

2 - Smooth the PLab's measurement data of that same reference source (eg. 15-point)

3 - Normalize the smoothed PLab measurement data (meas peak = ref peak = 1.0)

4 - Calculated the ratio (multiplicative not subtractive) difference over 400-650nm

5 - Use a cosine function to 'roll-off' the ratio curve to zero below 400 and above 650nm

6 - Smooth (eg 5-point) the result and save as a user's relative intensity gain-adjust cal

7 - User can then use this curve just like their CFL cal data

When SWB displays a spectra which makes use of a gain-adjust curve, it should overlay some graphical indicator of the 400-650nm gain adjust range so that the user can recognize that their data outside of this range should be treated as increasingly unreliable in terms of intensity relative to the data within 400-650nm. Remember, the existence of spectral information outside 400-650nm can be real, it is just increasingly less reliable because the spectrometer's sensitivity falls off so rapidly.

Reply to this comment...

Hmm... I'm wondering what the right graphical indicator for lower dynamic range at the outer edges of the spectrum should look like.

Reply to this comment...

How about just a partial 'fade-out' of the graphics plot range below 400nm and above 650nm (darkened / subdued colors) and then activate a text note in the UL of the plot area like 'Gain Cal Range'. The data, above and below, is still visible but it's obvious that data is, at least, less important (...less accurate, totally uncalibrated).

Reply to this comment...

This project might be of interest for finding a solution for the calibration problem:

https://jethomson.wordpress.com/spectrometer-articles/how-to-build-a-spectroscope/

It describes an amplitude calibration called "Determining the System Function" that corrects the effect of the RGB filter of the camera sensor (similar approach as suggested by Dave Stoft) plus a subsequent radiometric calibration using a separate light sensor. This allows quantitative measurements of spectral irradiance. All code for the calibrations is also available. Perhaps it is possible to add an economic light sensor to the DIY kit to get independent light measurements e.g. this one: http://www.yoctopuce.com/EN/products/usb-environmental-sensors/yocto-light-v3.

Reply to this comment...

While it continues to be an evolving prototype, the new Subtraction tools in Spectral Workbench 2.0 (preview) allow for subtraction of one spectrum from another. I'm curious if you might use this to attempt the correction you outline here. I noted a proposed process for a similar subtraction on the mailing lists a couple days ago:

• upload an image with two side-by-side cuvettes (hopefully you'll be able to distinguish the two spectra)
• calibrate it (using a calibration from the same setup)
• use the v2.0 interface (see the button along the bottom of the graph to switch into 2.0 preview)
• clone the spectrum so you have two copies
• use "set cross section" on each to choose your two different cross-sections of the spectrum, one for each cuvette, pressing "Apply" to accept the new cross-section
• use the "Subtract" tool to subtract one from the other
I'm VERY willing to help out in this process, as it's very much prototype code, and may still be buggy. Eventually, I'd like to use Sreyanth's Procedures code to help guide ppl through this process and automate some parts. Please, if someone wants to copy this procedure into a research note and try it out, and post results, that would be GREAT.

Reply to this comment...

OK, while we're tackling some of the discussion of gain correction in https://publiclab.org/notes/stoft/02-25-2015/plab-spectrometer-gain-correction#c13258, I wanted to confirm that this is the right product here, Dave:

http://www.amazon.com/18003-Daylight-Halogen-Degree-Kelvin/dp/B0002GS4OG/

How important is the power source for consistency? Something like this? http://www.amazon.com/LET-60-Class-Electronic-Transformer-Lightech/dp/B002RSOULS/

One other question; is there a way to reproduce this method with an edison screw-type bulb, to simplify/cheapen the setup? I guess I'm not finding much... http://www.bulbs.com/Halogen_Bulbs/Medium_(E26)-Base/results.aspx Just thinking of how to make this process easier for folks. On the other hand, if we can test a bunch of the webcams against one another, a standard correction for a webcam type is a good start, anyways.

Is this a question? Click here to post it to the Questions page.

Reply to this comment...

I used the 35W Solux 4700K. eg: http://www.amazon.com/35003-Daylight-Halogen-Degree-Kelvin/dp/B0002GS4GE/ref=sr_1_2?s=hi&ie=UTF8&qid=1452026828&sr=1-2&keywords=solux+mr16
These halogen lamps get really hot and there is no need for high power.

I could not find data on the spectral variance with source voltage, but since filament bulb intensity is a function of supply voltage, I believe using 12V is preferable. Solux specifies that their specs are based on a 3-5 min "warm-up" period for the bulb to obtain full stability. Since, at best, the calibration accuracy will be no better than about 10%, I see using a stable supply as a necessary part of a calibration.

I bought an Anchorn #AET 60W Class 2 12V supply for Halogen lamps -- but probably most qualified halogen 12V supplies would work.

Yes, because of the ~10% cal-error limit for the process (based on using a Solux lamp) I have suspected that individual calibration of webcams would likely prove unnecessary. I still suspect the webcam sensitivity variation between devices will prove to be much less than 10%. The test could be simple; buy one 12V 35V Solux + power supply and just measure 10 webcams using the exact same idealized setup. Then compare the resulting curve variation (after removing any intensity offset variations due to setup for each). The resulting curve error range should likely be very small. If so, just build-in the correction to SWB and make it transparent. Then, user's results could be specified with a theoretical %error over the 400-650 cal range.

Is this a question? Click here to post it to the Questions page.

Reply to this comment...

Posting the correction file and Solux published spectrum that Dave sent me here:

Solux4700KGCal_300-800nm_SANM.csv - Dave's correction curve from MatLab data

Solux4700kSpectrum_280-800nm.txt - Dave's extraction of Solux data from website

I'm just checking on this, but I believe we've changed the camera shipping in the desktop kit, which is one reason I'm interested in the process you used to generate the correction file, so it can be reproduced for any given camera. Just to re-state your 7 step process above in pseudocode:

// data from .txt file, captured from Solux website, in [nanometer wavelength, percent intensity]: var published_solux_spectrum = [[280, 0.049379], [290, 0.075516], ... ]; // data from a Public Lab Desktop Spectrometer v3 with a given webcam, in [nanometer wavelength, percent intensity]: var measured_solux_spectrum = [[280, 0.049379], [290, 0.075516], ... ]; var correction = []; measured_solux_spectrum.forEach(function(pixel, index) { var ratio = published_solux_spectrum[index][1] / pixel[1], // <= here we really need to match wavelengths, but for purposes of terseness in this pseudocode, we'll assume they're matched pair = [ pixel[0], ratio ]; // reformat into [wavelength, ratio] correction.push(pair); }); return correction;

Here I've skipped the "roll-off", although I agree we should do that. I just want to separate out the steps. Once this part is figured out, I have some questions about the roll-off. How does the above look?

Is this a question? Click here to post it to the Questions page.

Reply to this comment...

Sorry, fixed the formatting.

Reply to this comment...

Oops, one more bugfix to the pseudocode -- now I think it's what I intended.

Reply to this comment...

Hi, Dave - Mathew confirmed that the Desktop Spectrometry Kit v3.0 has always shipped with the JDEPC-OV04 webcam, not the SANM webcam. The SANM was used only in the v2.5 kit. If you have a completely standard kit, your data (and this whole post) should be for the JDEPC-OV04.

It looks like this (click to see full resolution):

Reply to this comment...

Ok, if you look back up above in THIS research note, you will see an Update I have added which includes a single combined plot set using my PLab-3 proto with the new camera and shows the 4700K Solux Gain-Correction curves. The curve data is also included as a set of .TXT files. I've included the 400-650nm correction; although the errors "blow-up" at either end of the webcam's wavelength sensitivity response curve. To use the gain-correction curve data (which is listed for 300-800nm in 1nm increments) just multiply: corrected_val = int16( double(PLab-3_val) * GainCorrection ) for every wavelength of interest. The result can be re-scaled if necessary. I used an arbitrary 530nm as the point to set the gain-correction to 1.0.

Reply to this comment...

That's super, thanks. I think we're pretty close here, but I just wanted to get explicit confirmation that the process I outlined is accurate for how you derived the correction -- mainly in the interest of reproducibility. Did the code I posted make sense, and does it accurately describe your process?

Thanks, Dave -- i'm also VERY close to posting the new Snapshots and Operations systems, which I hope to very soon follow up on with this correction code.

Is this a question? Click here to post it to the Questions page.

Reply to this comment...

Jeff, pretty close. I'll try again, in case I'm not reading your code accurately....

1 - Solux4700K_spec([nm(300:800),val(300:800)]) = the Mfg's specified spectral data for 300-800nm in 1nm increments

2 - Solux4700K_meas([nm(300:800),val(300:800)]) = the PLab V3 measured spectral data for 300-800nm in 1nm increments where the nm was obtained via a 2-point CFL cal using 435.833nm (blue) and 546.074nm (2nd Green peak)

3 - 530nm is arbitrarilly (selected based on the GRN response curve "flat region" when observing the Solux measurement) set as the "gain-correction=1.0 reference point.

4 - Yes, I'm assuming all raw data has been averaged and processed so as to be working in 300-800nm range with integer 1nm increments

5 - Given the Spec and Meas data sets, we then scale the Spec'd data to make the 530nm data points match such that the gain correction at 530nm will be ~1.0. I believe this is a useful step because it keep the final GCal correction curve data in a sensible value range so that applying the correction to new data will not result in huge mutiplicative shits in spectral intensity values.

6 - Calculate the "raw" gain correction curve where: GCal = SoluxSpec / SoluxMeasured. I believe this is what your code calculates.

7 - However, even at 400nm and 650nm (the two ends of the corrected range which are roughly the limits of using the correction w/o suffering too much error because the webcam's sensitivity is near zip. So, I apply a cosine-based rounding algorithm to both ends of the GCal(300:800) data from step 6. For example, the 'raw' GCal data can easily produce a magnification of ~2.5 or above at just 650nm due to low webcam sensitivity -- which would produce rather high error values so not be very honest.

Make sense?

Is this a question? Click here to post it to the Questions page.

Reply to this comment...

Hi, Dave and @ygstcu - I have a version of the gain correction code loaded into a local copy of my code, and am testing it out. Dave, can you link me to an un-corrected spectrum on Spectral Workbench which I can run the code on, and then we can compare the output of my correction code to the corrected data you generated? If they match, then we're very close.

Is this a question? Click here to post it to the Questions page.

Reply to this comment...

Jeff, I was going to provide you with a fresh plot but unfortunately SWB still fails on Firefox -- same "left half only" spectrum with no visible data to match what shows in the image -- so, SWB remains unusable. I'll try an update and see if that helps. Otherwise, I could probably do a download of an uncorrected spectrum you wanted to use and convert that to run locally.

Reply to this comment...

Hi, Dave - I left some requests on your issue to try to reproduce the error you're seeing; did you see that? https://github.com/publiclab/spectral-workbench/issues/285

Is this a question? Click here to post it to the Questions page.

Reply to this comment...

Ok, just took a look, replied no, still broken. I cleared cache with no effect. I filed another bug report as I cannot even use the switch to a 'new calibration' -- which just brings up the error page (The page you were looking for doesn't exist). Individual bugs like this are a pain if they are not easy to reproduce.

Reply to this comment...

Hi, Dave, thanks. I responded and split out a new issue with the calibration bug and added a couple questions.

Reply to this comment...

Ok, I got a fresh copy of Chrome to work so here's a Solux 4700K spectrum which was callibrated with a 'CFL2700K' new cal data. https://spectralworkbench.org/spectrums/71409 However, the data download is also broken as there is no wavelength data (all zeros) in the CSV format and 'null' in the XML format -- unless some other new feature must be run besides the 'calibrate' function to get that wavelength data to be included.

Reply to this comment...

Thanks, Dave - I believe exports were fixed in https://github.com/publiclab/spectral-workbench/issues/322, which was only published just now with the changes related to your issue. They work now and I confirmed manually.

Reply to this comment...

Hi, Dave - I believe I was able to apply the correction by running the following line in the new Scripting tab:

SpectralWorkbench.API.Core.useReference(false, spectrum); graph.reload_and_refresh(); 

This used your non-rolled-off correction file. I'm not sure yet how I want to integrate this into the Operations system, but I was able to download a CSV output using this and the JavaScript console:

SpectralWorkbench.API.Core.useReference(false, spectrum, function() { spectrum.average.forEach(function(line) { console.log(line.x + ',' + line.y)})}); 

The output was:

spectralwb-71409.csv

And it looked like this:

Reply to this comment...

The CSV data files have poorly formatted data. The text of the files often looks like this: 206.809,2.55,2.5500000000000003,7.6499999999999995,2.5500000000000003 . Also, since the wavelength precision of the measurements is only 1nm and the accuracy is less, the wavelengths after CFL cal should all be nice integer values with uniform 1nm increments. I'd thought this had been done already -- if not, it should be updated.

Reply to this comment...

Just extracted the Slux4700K original data w/rgb and your gain-corr curve data from above, scaled the curves to eliminate that factor and plotted. Doesn't appear to have had any effect....... the black is the original Solux spectra and the magenta is your processed data.

Reply to this comment...

Sorry, back online after a long weekend. Looks like there was just an error in my code -- i'm going to try and tackle some bugfixes today but after that i'll give it another go. Thanks for checking -- and sorry it's a bit of an onerous round-trip to actually test this out. I can both visually inspect the output vs. your graph now, and also compare it to a fork of itself without the correction.

Reply to this comment...

Understood; though independent checks and tests are still the best method to find hidden anomalies and overlooked issues. When you fix the glitch, I'd expect the ends of the corrected plot to show dramatic error expansion as the camera sensitivity goes to near zero -- i.e. this should be really obvious in the plot.

Reply to this comment...

Also, regarding the unit precision, I've opened an issue at https://github.com/publiclab/spectral-workbench/issues/335 and would love your input on managing precision there.

Reply to this comment...

Hi, Dave - i noticed that https://spectralworkbench.org/spectrums/71409 has a linearCalibration operation - I think by mistake, as it's not a CFL. I removed it (i'm an admin, so i'm able to) but can add it back in if it was intentional?

Also, I was just wondering, your black line in your most recent image does not seem to be an average of the channels, and does not match the data I see before attempting to apply the correction:

I'm working on getting the correction working now, but just wanted to check on this so that I see a clear before/after state that matches yours. Thanks!

Is this a question? Click here to post it to the Questions page.

Reply to this comment...

Jeff, thanks for pointing this out. I admit I do find the SWB terminology a bit overloaded and the user interface 'unexpected' as I'm not used to it. I'd thought that the Solux plot had used the CFL-2700k cal as a reference but I must have noticed the x-axis pixel values and hit 'calibrate', intending to tell SWB to 'calibrate' the spectra using the last CFL reference I'd specified in the interface when I took the data -- as opposed to 'use this plot to calibrate the spectrometer'. It does seem that SWB can't tell the Solux plot is not a CFL. As for the level, yes, it seems off but I was rushed (a bad thing) and guessed SWB was just scaling the data for display.

Reply to this comment...

OK, a bunch of solved problems!

• figured out the height difference -- needs to be resolved, but I believe due to a bug, we were multiplying each pixel by the final entry in your correction file -- 2.667. In any case it was just scaling the whole graph by 2.667, so my just-published fix resolves that.
• It's hard to get SWB to "know" that something is a CFL, but I'm going to try to add some more notices if, for example, operations are in an order that doesn't make sense -- like a linearCalibration after a calibrate operation. The tools are very new to everyone, so i'm sure it'll be helpful.

Now, on to the success! I believe I got this to work; see screenshot and check out the output data:

Reply to this comment...

Great! Yes, little glitches do creep in but yes, this Solux corrected plot has a much better shape. I'm guessing the slight curve variation between 500 and 650, plus the peaking at about 650nm are similar to what I saw, even though the ideal Solux curve is more uniform. So, I take this as good confirmation of the approach. It also looks like you've handled any error accumulation at the ends to the algorithm is well behaved. Without individual, calibrated-Solux calibrations there probably a little unresolved error, but I'd think this is a good first-order improvement on system gain correction.

Reply to this comment...