This is awesome, Dave! The noise above 700 and below 400nm is substantial, but I think that just defines the usable range of the spectrometer.

@warren, what do you think about adding a solar spectral correction into SWB? is that possible as a secondary calibration?

Thanks. Yes, the camera has virtually no relative sensitivity outside 400-700nm. This does need to be differentiated from observing a bright source outside that range where some signal may be observable -- i.e. measuring attenuation of a fluid using a bright 350nm source might be possible. The point to take from this gain-correction is that without a truely 'flat' light source, 400-700nm is really the limit for corrected spectra.

my experience has been similar-- just because a very bright source in the 800-1000nm range can be detected doesn't mean that we can trust the sensitivity of a fluorescence measurement in that range.

I'm wondering if you took any account of the web camera settings - in particular colour temperature / white balance. This has quite a large effect on the gains of R,G and B. And the sunlight reference curve looks like it is of relatively low resolution compared to the PL spec output. The PL Spec plot may not have noise so much as actual signal.

Thanks; interesting thoughts. Yes, digital cameras can have white balance settings which create a general shift in the RGB output ratios, but the SWB interface does not have that sophistication. SWB therefore has the same interface settings for all Sanm cameras -- whatever that actual color temp might be -- the specs on the Sanm device are sketchy about that. The sunlight reference curve is "smooth" compared with professional-grade plots which show more detail -- but that "detail" results from spectral "absorption holes" which are too narrow for simple digital camera spectrometers to detect. The webcam is, in fact, quite noisy where the sensitivity is low (like the ends of the detectable bandwidth) so that data really is just noise, not real signal data. There is no way to create a gain-correction based on noise so it is better to use reference data with similar detection bandwidth to the hardware being corrected. An incandescent light bulb also has a relatively "smooth" spectrum but it's hard to find light bulbs where you have the actual spectral curve of it's specific temperature. There's lots more very accurate data for the sun.

adding a solar spectral correction into SWB

YES! I'm finishing up some MapKnitter work next week, and debugging today and tomorrow, but would love to use this approach in a re-imagining of the new calibration procedure.

I'm looking for help using a solar spectrometer to gain current incident light values as a base to adjust my reflected image taken in a high quality digital image. I am doing aerial imagery and understand that often the incident light values change. I want to ratio the different intensity values to get a consistent value.

Might be helpful to post the topic in the PLab Spectrometer Google groups where you can get a wide range of feedback. Briefly, I'm not clearly following your intent but it seems you want to measure / reference the solar spectrum so as to remove that variable from the image measurements. I'm guessing you are wanting to color-balance your aerial photos taken in sunlight which can be difficult depending on the degree of accuracy. In theory, if you could measure the general solar spectrum curve (as it relates to the deg-K change during the day as measured on the ground you are photographing, then you might apply an idealized deg-K shift to the image. An alternative might be to white-balance the camera on the ground and take a photo of a color reference card prior to launching the camera. Neither method take into account atmospherics introduced by the altitude of the camera.

Hi, Dave - I'm circling back because I believe the new subtract: feature in Spectral Workbench 2.0 could enable you to do a standard gain correction (I'm hearing 'standard deduction' in my head, which is funny, but kind of similar -- i.e. it wouldn't be super accurate, but would be better than nothing, right?). But for now, the easiest way to try it out would be to upload a curve (as an image) which could be used as the standard. Is there any way you could upload a standard correction (for the sanm camera, for example, or even better for the camera that ships with the 3.0 Desktop Spectrometer) which we could use to test that out? We could try it vs. some lab specs to see how well the correction works.

Also, can you link to your source for the 5780K spectrum? Thanks a lot, Dave!

Jeff - Yes, it may be possible (with some modifications) to use your new feature code for a 'default' gain correction. Also recall my later research note dealing more specifically with using the Solux 4700K lamp for calibration: https://publiclab.org/notes/stoft/03-06-2015/plab-3-gain-correction

However, remember that correcting the camera sensitivity (gain) vs spectral wavelength is a multiplicative process, not a subtractive process as is used when comparing two measurements with a device where the gain remains the same. To correct the camera sensitivity, the raw camera data is multiplied by the Solux correction data and then scaled for plotting. Also remember that although the correction data spans 300-800nm, the actual usable range of corrected data from the camera is only 400-650nm because the camera sensitivity errors dramatically increase at each end where the camera becomes totally insensitive. But, perhaps you could re-use most of your code for processing curve data and just add implementation for correction by adding buried (non-visible) correction curve data and re-scaling (in the case of corrected spectral data exceeding 100%).

The Solux spectral data can be found here: http://solux.net/ies_files/SoLux%20Spectral%20Data.xls.pdf You will note that only the 4700K lamp has an emission curve which is sufficiently smooth and broadband enough for the spectrometer gain connection.

I can email you the Solux and PLab correction data; it is not data directly suitable for display in SWB.

Ah yes, re: multiplicative. Luckily there's an as-yet hidden feature called blend: which is like subtract, but which allows any arithmetic combination of spectra; it runs a math expression on every pixel (kind of like on Infragram.org). So multiplication shouldn't be hard. The new range:min-max operation can limit our work to the necessary range, too.

Is it possible to upload the correction data here? What format is it in? If it's not possible to convert it to an image, I might be able to help do that, either manually in Javascript, or potentially we could reformat it as JSON or CSV data which we could build an importer for in Spectral Workbench. This is exciting!

The correction data is in a CVS file and the Solux data is in a TXT list file -- both with 2 columns (nm,value). The data should remain in this (nm,value) pair format and not be converted to an image. There is no need to plot the correction data as the 'values' are not % units like the SWB spectral plots -- the data can just be kept in a background data file which is internally read when needed. The Solux data values are all relative linear intensity with no units and so would be similarly used as background data for individual lamp calibrations at some point. I'll e-mail the files.

I guess I'm thinking of how we might use different correction files for different lamps, and how people might upload and share new correction files for new lamps. Using the Spectrum database storage and infrastructure for that seems reasonable, and it lets us use the new Operations system to apply the correction. Does that make sense? Maybe we can flag them as corrections data in metadata so they aren't confused with a real spectrum.

Sure; they are just different file types (content). Initially, for testing, there could be just one "default" selection with buried data as a proof-of-concept and which allows the user to validate that any other cal file is producing at least reasonable results. It's a stepped process: a) unspecified error limits (now), b) ~10% error (Solux), c) lower error (other)? .... with the final error budget yet to be determined.

1 - Solux4700KGCal_300-800nm_SANM.csv (My cal curve from my MatLab data)

2 - Solux4700KSpectrum_280-800.txt (My extraction of Solux data from web site)

I got the two files:

Solux4700KGCal_300-800nm_SANM.csv

Solux4700kSpectrum_280-800nm.txt

So the .txt file is their reported spectrum from their website -- shall we use another absolute intensity calibrated spectrometer (when we can) to confirm this?

And just to be sure, do I have this right? Each data point of the correction .csv is the Solux reported lamp spectrum divided by the Public Lab v3 Desktop Spectrometer scan of a Solux lamp?

Yes, the Solux4700KSpectrum file contains Solux's 4700K lamp output spectrum, after a 3-5min warm-up period +/-200K on the center color temperature. In my related research note, I show that this +/-200K uncertainty equates to <10% intensity error in the curve data. So, any re-measurement of a Solux lamp would have to be made with a spectrometer that could significantly exceed this accuracy spec.

Yes, but not quite. The correction data is roughly 1x at 500nm so you would just multiply measured webcam spectrum data by the correction data to get the result. (With some spectra it might be necessary to re-scale due to the ~2x correction out around 600nm) However, because the webcam's sensitivity drops off rapidly below 400 and above 650, while the Solux lamp's spectra is broader, the webcam correction error becomes excessive (and also unrealistic because the noise exceeds the signal). To prevent the application of the .CSV correction data from creating horrible (and noisy) plots at each end, the calculated correction data has been arbitrarily "rolled off" using a cosine function. Doing so, results in a nice response plot for CFL spectra which then doesn't "explode" with noise at the ends. It also means it is best to indicate the 400 - 650nm valid region for the resulting plot. (The webcam data's valid range for intensity is really this limited anyway even though you might see some noise and signals beyond those limits.) These effects are all explained in the other note: https://publiclab.org/notes/stoft/03-06-2015/plab-3-gain-correction

Sorry, indeed I should've left my comment on the other post. Moving over there, thanks.

Question for @stoft. According to my measurements, the CMOS sensor has some sort of blooming effect before saturation. I tested my webcam with white LEDs, Look images with almost saturated condition and with 5 times lower light. White graph is sum of RGB signal. These efect is not because of sensor nonlinearity. On top of first graph you have picture stripe from which I take 100 lines, so y aksis is in 100 x bits. Because of that you get pik or hole in spectrum, depend on light intensity.

How do you eliminate that efect

@Tehnicni, I'd agree that any detector-specific non-linearity is likely very small compared with the effect you've presented. I'd guess the more likely cause is camera-internal firmware-applied compression -- so non-linearity of the camera output data due to internal processing. The green peak shows clear compression in the upper graph. Seems like the camera is attempting to soft-compress to reduce visual artifacts of hard compression. Compression can be helpful for typical web-video use but not helpful for spectrometers. If the camera exposure control (AGC) can be turned off that helps. Otherwise, 1) take care to avoid obvious (visual) signs of compression from high-level signals and 2) average the more-linear output at lower levels -- and/or choose a more controllable camera.

I'm using wery old cheap webcam with all controll on manual and get same result with Logitech C525 also in manual mode. I checked lineariti on all 3 colour, green ones with white LED and pik response is almost linear. There is also another sign for blooming. Combination of 660n LED with 100 bit pick & white LED with 20 bit at 660nm give both together 140 bit at 660nm response of red sensor. It look like that sensor became more sensitive if they receive much light. Could you please check lineariti of your spectrometer at different level of light. I want to found some reasonable price monochrome usb camera, but suplier_elp on ebay told me, that all ELP b&w camera use RGB sensors.

The camera's I've used seem different in this effect but you might try to hunt down the technical data sheets from the Mfg -- sometimes they have additional information. I did test a monochrome 'webcam' (see Research Note) but it did not offer a cure-all solution. That is not to say that a good quality monochrome camera would not do much better. I was under the impression, from what the Mfg told me, that the monochrome camera I tried was not just an RGB, but a true monochrome. I tend to believe it as the sensitivity and adjustments were different.The internal camera firmware can have a number of control features (as opposed to just one setting) so care is needed to really know what the control(s) is(are) doing -- the combined effect can require experimentation to find the right trade-offs. The following PLab notes might be of interest: https://publiclab.org/notes/stoft/05-17-2014/plab-s-webcam-non-linearity, https://publiclab.org/notes/stoft/10-23-2016/high-rez-webcam-spectrometers, and https://publiclab.org/notes/stoft/03-09-2014/hdr2-using-over-exposure-to-your-advantage. If I had to choose a simple webcam with good price/performance ratio, out of the limited number I've measured, I'd suggest the OVA-2710 from Amazon for about $45 with an 8mm M12 lens $8.

I also tried the third webcam, cheap Chinese. White Balance: R and G at max, B in the middle. Gama 2, Exposuer manually

Measured at different current through white 4500K LED. 100%, 50% & 20%. Sum graph again differ according to light level.

Yes, clearly all but the last image shows compression in the Green (and Red though less). Because the process of spectral data from a webcam does not inherently contain a reference level, the best approach is simply visual. Watch the RGB curves as the light intensity is increased and look for early signs of compression in the highest peak. The early signs generally include a change in shape to the peak -- the shape should not change while the sensor is operating within its linear region. Since the resulting spectra is a combination of RGB, allowing any signal compression on any of the RGB channels only adds distortion (error) and so is useless / false information. It is much better to average out a little noise than to distort the spectrum because of compression.

How a stupid mistake. I checked linearity of all 3 cameras at three points, thinking that it have an exponential dependency, which is not true, the dependence is in the S shape. I took the addiction in 10 points, interpolate it, and now things works very well. Thank you for your patience and advices

Ah, good. Glad you could view the additional data and figured it out.