Using a webcam or a pi camera, it quickly becomes clear that a single capture cannot encompass the large range of brightness in the scene, also known as the dynamic range. A solution is high dynamic range (HDR) imaging, a process that combines data from multiple exposures into one image that is neither overexposed, nor underexposed.
Previous notes that have discussed HDR include: HDR: In search of high-er dynamic range by stoft and High Dynamic Range (HDR) Imaging (revisited) by MaggPi.
This post extends the method described by MaggPi using the same OpenCV library for python, and focuses only on the Debevec algorithm. Debevec and Malik (1997) wrote an excellent paper on their algorithm that is very helpful to understand the method: Recovering High Dynamic Range Radiance Maps from Photographs
The Debevec Method
In brief, the algorithm is based on the principle that camera sensors have a non-linear response to light. That means, if you double the light energy in the environment, the pixel value is not doubled. For research where we are trying to extract information about the world using pixel data, it is super important to know how pixel values map to irradiance in the captured scene.
Debevec and Malik's method produces a function, and when plotted on a graph, the x axis represents pixel values (from 0-255) and the y axis represents sensor exposure, which is a product of scene illuminance (E) and shutter speed (Δt).
The steps include:
- Taking a series of photographs with known and different exposures of a scene with the same camera position
- Collect a sampling of pixels from the scene, and determine each pixel's value for every exposure time. This works even though we don't know the actual irradiance value for any pixel, since we assume that the scene irradiance is constant across the photograph set.
- Use an algorithm to combine the pixel data into a beautiful curve.
While steps 1 and 2 are straightforward, I found the algorithm quite complex which turns the graph below from the left to the one on the right. It's so great that it's already implemented in OpenCV! I mostly followed the code from this tutorial: https://www.learnopencv.com/high-dynamic-range-hdr-imaging-using-opencv-cpp-python/
Figure from Debevec and Malik 1997
This curve can then be used for any image set to map pixel values back into irradiance (E) since we know the sensor response function (f), the pixel value, and the shutter time (Δt) where
pixel value = f(E * Δt)
So what's the catch? In order to produce this sensor response curve, we need a suitable set of images, ideally over a wide range of exposures that overall contain data for each color (r,g,b) ranging from 0 - 255 values. We probably don't want to do this kind of elaborate process all the time, and that's why we can separate it into two steps. First, generate the sensor response curve. Then, save the response curve, and use it in the future to process all of your HDR photos.
The importance of a good photo sample is demonstrated in the 3 images below, ranging from poor to very good:
The cropped CFL image is better than its uncropped counterpart because the random sampling is more likely to fall on pixels with a greater variety of information. The LED image is much better than the CFL data because it has a broader spectrum, and using a wider exposure range, there is much more data for each color ranging from 0 to 255.
Just for comparison, here's also a Canon sensor response curve generated from a set of HDR photos found online:
Source images copyright by Wojciech Toman: http://hdr-photographer.com/hdr-photos-to-play-with/
I don't know if the PublicLab webcam is capable of manual control, but if it is it would be interesting to see its response curve too.
Generating an HDR image using the sensor response curve
Now that we have the sensor response curve, we can take any image and map it back into irradiance values. While they won't be representative of absolute irradiance (e.g. energy units) without further calibration, they will be proportionate to each other.
You only need two lines of code from here:
mergeDebevec = cv2.createMergeDebevec()
hdrDebevec = mergeDebevec.process(images, times, responseDebevec)
where images
is your image set, times
is their exposure times, and responseDebevec
is the .npy array from the response curve we generated earlier. The output, hdrDebevec
is now a numpy array containing the merged HDR data.
From here we have two choices:
A) Compress the data into an LDR (i.e. remap the values to 0-255) to present as a .png image
B) Directly extract the data from the HDR numpy array
Option A is nice because it's easy and you can see the pretty picture, but Option B will have better data resolution since we are not compressing the irradiance values. Let's compare them. For each, I've extracted data from the same pixel row.
Creating the LDR and saving the image is only 3 more lines of code using the OpenCV library as cv2:
tonemap = cv2.createTonemap()
ldr = tonemap.process(hdrDebevec)
cv2.imwrite('hdrHolo1.png',ldr * 255)
The uploaded spectrum is available on Spectral Workbench, where I extracted one line of data. https://spectralworkbench.org/spectrums/160042
The peaks are sharp and the shape is good, but the final test is the data comparison.
The HDR Plot
exported from python
The LDR Plot
extracted from the png uploaded to spectral workbench and plotted in Excel
Wow, for two things. Firstly, the HDR plot looks great. Secondly, a lot of data was lost in the compression stage. After all that work, it's really not worth it to discard all that great data.
So in conclusion, this process is promising, but the best data relies on a) manual control of your camera b) a good dataset for sensor calibration and c) using the HDR data directly without compression.
In further study, I am interested to know if getting the raw RGB data from the pi before jpg compression will be even better, since I am suspecting that certain dips next to the tallest peaks are due to compression.
Appendix:
I attached a screenshot of my camera settings on the pi camera web interface: IP_Camera_Settings_Pi_annotated.pdf
14 Comments
@warren awards a barnstar to jenjimah for their awesome contribution!
Reply to this comment...
Log in to comment
Good work and observations. Yes, I think HDR techniques can help extract additional dynamic range from these simple cameras. In reading through your notes I'm also wondering about some fundamentals which I'd noticed in my own experiments. Web-Cams are designed and optimized for taking images of 'world scenes' not spectra from diffraction gratings and so they tend not to react 'nicely' to such input. I suspect the Pi camera is much like other 'web cams' since they are all based on similar sensor chip technology.
The cameras have internal AGC loops (Automatic Gain Control) which seek to 'level' the analog gain in ALL pixel elements based on an overall image intensity average. This makes the camera output both a function of the specific 'scene' and a function of an active loop with an unknown, and even variable, loop bandwidth (rate of correction).
With one camera I used + a matlab interface, I was able to shut off the AGC and set a fixed image gain. At that point, the camera's response was quite linear (for such a simple detector) ..... but with AGC running, the camera did appear to be non-linear -- and difficult to control. (This can appear as instability in the spectral intensity over time -- which most tend to just average out, but which really shows as a significant intensity error value to the final data.)
The other factor, once the gain is fixed, is handling 1) the gain step size (another calibration) and 2) noise which tends to increase as the gain increases (a limit to the range of an effective HDR).
So, I'm just suggesting that it would be good to know if the observed non-linearity is a result of an active AGC loop. If it is, then the 'correction' could be further complicated by not having control over the specific range of the AGC's operation -- which is, in turn, a function of the image detail being observed (a variety of spectra with variable average intensities). True, extracting accurate data is not terribly straight-forward ;-)
Great question, it's pretty amazing you were able to shut off the AGC circuit. I had previously imagined that most ofthe pre-data export operations were hardwired or hardcoded in.
Yes it looks like the Debevec algorithm just jumps right over top of the camera functions and wraps it all into one to get from scene illuminance to sensor data. They acknowledge this in their paper.
"Digital cameras, which use charge coupled device (CCD) arrays to image the scene, are prone to the same difficulties [as film]. Although the charge collected by a CCD element is proportional to its irradiance, most digital cameras apply a nonlinear mapping to the CCD outputs before they are written to the storage medium. This nonlinear map- ping is used in various ways to mimic the response characteristics of film, anticipate nonlinear responses in the display device, and often to convert 12-bit output from the CCD’s analog-to-digital convert- ers to 8-bit values commonly used to store images. As with film, the most significant nonlinearity in the response curve is at its sat- uration point, where any pixel with a radiance above a certain level is mapped to the same maximum image value."
But the question still remains about how stable is the sensor response curve for our cameras? I guess we can really only know for sure through testing. Test the camera in a variety of situations and compare. Even better would be to test multiple cameras to see the quality control, to test if it could make sense to use one sensor curve and apply it to other people's hardware. That would really reduce the workload required to start doing hdr, although my hunch is it's probably still better to have a customized curve for each individual sensor.
Is this a question? Click here to post it to the Questions page.
Right, the camera's do automate as much as possible, which works against the needs of spectrometers. If the AGC cannot be shut off, then it becomes very difficult to make intensity measurements (even relative intensity) because the chip is intrinsically modifying the gain without knowing when or by how much.
In a simplistic way, it is easy to see the spectral curve change level with a change in source light -- even with an AGC. That is usually because the adjustment range of the AGC is limited to less than the sensor's dynamic range. However, when looking for small intensity changes, the data will be a mix of effects of both the AGC (which is typically based on average 'scene' intensity) and real changes in input light intensity. They are co-mingled and not easily separated.
I believe that an intensity correction curve is only possible if the gain can be controlled. I was not sure by the above description if the python code had some interface for the PI camera for switching the AGC off and for controlling the gain. Might look for that and/or do a bit of web search on controlling those parameters in the PI camera. Sometimes camera chips have the option but the chip interface card does not allow them to be programmed. Some cameras do.