We present a novel compression method for fixed viewpoint reflectance fields, captured for example by a Light Stage. Our compressed representation consists of a global approximation that exploits the similarities between the reflectance functions of different pixels, and a local approximation that encodes the per-pixel residual with the global approximation. Key to our method is a clustered sparse residual factorization. This sparse residual factorization ensures that the per-pixel residual matrix is as sparse as possible, enabling a compact local approximation. Finally, we demonstrate that the presented compact representation is well suited for high-quality real-time rendering.