With the advent of extremely powerful computers and graphics processors the worlds of 2D contextual images and 3D dynamic renderings are merging. Simulating a virtual environment, under multiple illuminants, and providing the capability to render realistic objects with selectable colors, special effects, and other color appearance phenomena, presents new challenges. The color and appearance attributes applied to an object are based on physics and the material properties of the phenomena. We combined multi-spectral imaging, BRDF technology and GPU-based rendering to meet these challenges and advance the state of the art.A physically-based visualization system is presented which realizes photo-realistic renditions of 3-dimensional objects with complex real-world material characteristics in captured real-world panoramic environments. The system runs in real-time and permits interactive manipulation of material parameters, change of environments or environment characteristics such as current illuminant, with immediate visual feedback.For a real-time system unprecedented visual accuracy is achieved through the use of multi-spectral, high-definition, and high-dynamic range descriptions of material reflectance and environments, and through a hybrid shading subsystem which draws on a number of recent rendering techniques including frequency-space reflection mapping based on Spherical Harmonics for accurate BRDF rendering.The system's real-time abilities are enabled by recent programmable graphics card hardware and extensive use of programmed shaders. This paper describes the system along with some of the methods used to acquire material characteristics and environments.