Practical Multispectral Lighting Reproduction

Chloe LeGendre    Xueming Yu    Dai Liu    Jay Busch    Andrew Jones
Sumanta Pattanaik1   Paul Debevec
USC Institute for Creative Technologies     University of Central Florida1


Figure 1: The left images show each of two subjects photographed in real outdoor lighting environments. The lighting was captured using panoramic HDR photography and color chart observations, allowing the light to be reproduced in an LED sphere with six different LED sources of distinct spectra. The right images show each subject photographed inside the multispectral LED sphere under reproductions of the captured lighting environments, composited into background photos, producing close matches to the originals.

 

Abstract:

We present a practical framework for reproducing omnidirectional incident illumination conditions with complex spectra using an LED sphere with multispectral LEDs. For lighting acquisition, we augment standard RGB panoramic photography with one or more observations of a color chart. We solve for how to drive the LEDs in each light source to match the observed RGB color of the environment and to best approximate the spectral lighting properties of the scene illuminant. Even when solving for non-negative intensities, we show that accurate illumination matches can be achieved with as few as four or six LED spectra for the entire ColorChecker chart for a wide gamut of incident illumination spectra. A significant benefit of our approach is that it does not require the use of specialized equipment (other than the LED sphere) such as monochromators, spectroradiometers, or explicit knowledge of the LED power spectra, camera spectral response curves, or color chart reflectance spectra. We describe two useful and easy to construct devices for multispectral illumination capture, one for slow measurements of detailed angular spectral detail, and one for fast measurements with coarse spectral detail. We validate the approach by realistically compositing real subjects into acquired lighting environments, showing accurate matches to how the subject would actually look within the environments, even for environments with mixed illumination sources, and demonstrate real-time lighting capture and playback using the technique.



Introduction:

Lighting reproduction systems as in [Debevec et al. 2002; Hamon et al. 2014] surround the subject with RGB color LEDs and drive them to match the lighting of the scene into which the subject will be composited. The light is recorded as panoramic, high dynamic range images, or rendered omnidirectionally from a global illumination lighting system. While the results can be believable – especially under the stewardship of color correction artists – it is not clear how accurate they are since only RGB colors are used for recording and reproducing the illumination: there is significantly more detail across the visible spectrum than what is being simulated. [Wenger et al. 2003] noted in particular that light reproduced with RGB LEDs can produce unexpected color casts even when each light source mimics the directly observable color of the original illumination. Ideally, a lighting reproduction system could faithfully reproduce the appearance of the subject under any combination of illuminants including incandescent, fluorescent, LED, and daylight, and any filtered or reflected version of such lighting.

Recently, several efforts [Gu and Liu 2012; Ajdin et al. 2012; Kitahara et al. 2015] have produced controllable LED spheres with more than just red, green, and blue LEDs in each light source for purposes such as multispectral material reflectance measurement. These systems add additional colors such as amber and cyan, as well as white LEDs which use phosphors to broaden their emission across the visible spectrum. In this work, we present a practical technique for driving the intensities of such arrangements of LEDs to accurately reproduce the effects of real-world illumination environments with any number of spectrally distinct illuminants in the scene. The practical nature of our approach rests in that we do not require explicit spectroradiometer measurements of the illumination; we require only traditional high dynamic range (HDR) panoramic photography and one or more observations of a color chart reflecting different directions of the illumination in the environment. Furthermore, we drive the LED intensities directly from the color chart and HDR panoramas, with no need to explicitly estimate illuminant spectra, or even to know the reflectance spectra of the color chart samples or the spectral sensitivity functions of the cameras involved. Our straightforward process is:

  1. Photograph the color chart under each of the different LEDs
  2. Record the illumination using standard panoramic photography plus one or more color chart directions
  3. For each LED light source, estimate the appearance of a virtual color chart reflecting its direction of light from the environment
  4. Drive the light source LEDs so that they best illuminate the virtual color chart with the estimated appearance

Step one is simple, and step four simply uses a nonnegative least squares solver. For step two, we present two assemblies for capturing multispectral lighting environments which trade spectral angular resolution for speed of capture; one assembly acquires unique spectral signatures for each lighting direction; the other permits video rate capture. For step three, we present a straightforward approach to fusing RGB panoramic imagery and directional color chart observations. The result is a relatively simple and visually accurate process for driving multispectral LED sphere lights to reproduce the spectrally complex illumination effects of real-world lighting environments. We demonstrate our approach by recording several lighting environments with natural and synthetic illumination, and reproduce this illumination within an LED sphere with six distinct LED spectra. We show this enhanced lighting reproduction process produces accurate appearance matches for color charts and human subjects and can be extended to real-time multispectral lighting capture and playback.



Results:

We used a Canon 1DX DLSR camera to photograph a Matte ColorChecker Nano chart from Edmund Optics, which includes 12 neutral grayscale squares and 18 color squares, under incandescent, fluorescent, and daylight illuminants. The background squares of the first row of Fig. 2 show the appearance of the color chart under the three illuminants, while the circles inside the squares show the appearance under the reproduced illumination using six LEDs (red, green, blue, cyan, amber, and white, or RGBCAW) for direct comparison. This yields charts which are very similar in appearance, to the point that many circles are difficult to see at all. The second two rows show the results of reproducing the illumination with just four (RGBW) or three (RGB) LED spectra. The RGBW matches are also quite good, but the RGB matches are generally poor, producing oversaturated colors which are easily distinguishable from the original appearance.

  



Figure 2: Comparison color charts for three illuminants where the background of each square is the original chart appearance, and the circles (sometimes invisible) show the color chart under the reproduced illumination in the LED sphere. Six (RGBCAW), four (RGBW), and three (RGB) spectral channels are used.

Figure 3: Two subjects in different lighting environments (left columns) with three different lighting reproduction techniques. Generally, RGBCAW and RGBW reproduction produces accurate results, whereas using just RGB LEDs tends to oversaturate colors.

Fig. 3 shows two subjects in different lighting environments. The indoor environment (top row) featured an incandescent soft box light to the subject's left, and spectrally distinct blue-gelled white LED light panels to her right, with fluorescent office lighting from the ceiling. The lighting was recorded using a five-color-chart and reflective sphere capture technique. Then, as quickly as possible, the subject was photographed in the same environment. Later, in the LED sphere, the lighting was reproduced using 6-channel RGBCAW lighting, 4-channel RGBW lighting, and 3-channel RGB lighting solves. Generally, the matches are visually very close for RGBCAW and RGBW lighting reproduction, whereas colors appear too saturated using RGB lighting. The fact that the RGBW lighting reproduction to performs nearly as well as RGBCAW suggests that these four spectra may be sufficient for many lighting reproduction applications. The bottom row features a sunset lighting condition reproduced with RGBCAW lighting where the light of the setting sun was changing rapidly. We recorded the illumination both before and after taking the pictures of the subject, and averaged the two sets of color charts and RGB panoramas to solve for the lighting reproduction condition.



Conclusion:

In this paper, we have presented a practical way to reproduce complex, multispectral lighting environments inside an LED sphere with multispecrtal light sources. The process is easy to practice, since it simply adds a small number of color chart observations to traditional HDR lighting capture techniques, and the only calibration required is to observe a color chart under each of the available LED colors in the sphere. The technique produces visually close matches to how the subject actually would look in the real lighting environments, even with as few as four LED spectra available (RGB and white), and can be applied to dynamic scenes. The technique may have useful applications in visual effects production, virtual reality, studio photography, cosmetics testing, and clothing design.



Material:

SIGGRAPH 2016 Paper :

Related Projects: