The concepts and technology for an open loop color management system such as that specified by the International Color Consortium (ICC) have been around for a number of years. The adoption of this workflow by the graphic arts industry has been slow. A major contribution to the lack of popularity is that the quality part of the ICC workflow is unregulated and the average user is unable to independently assess the quality of profiles and profile making software. This paper describes a number of test methods that can be used to evaluate the colorimetric accuracy of ICC scanner, monitor, and printer profiles. ICC profiles are being used in a number of color proofing scenarios. In order to understand the color reproduction abilities of such proofing systems it is necessary to quantify the accuracy of the underlying ICC profiles used in the workflow. A quality metric can be useful to provide feedback on how well a device has been characterized and therefore provide limits on the ability of a color managed system. Further, a universally defined merit figure will allow the comparison of results across manufacturers, allowing the user to make informed choices appropriate for their workflow. If we are able to establish a benchmarking procedure akin to the miles per gallon fuel consumption quoted for motor vehicles this provides a universal quality metric that can help raise the quality of profiling software, assist user choice, and ultimately lead to the greater acceptance of ICC color management in graphic arts and the printing industry. The quality of input profiles is described in terms of a ΔE calculation. Input profiles from ten different profiling packages were evaluated. Profiles were made for a flatbed scanner using Agfa, Fuji, and Kodak IT8.7/2 targets. The average ΔE for input profiles is shown to be in the range of ΔE 0.60–2.46. A procedure is described for evaluating the quality of monitor profiles in terms of measured gamma, white point and the color reproduction of 24 specially chosen colors. Eleven monitor systems were evaluated. Monitor profiles were made for an Apple Cinema HD LCD display and measurements were made to verify the accuracy of these profiles using a telespectroradiometer. It is shown that commercial products were able to reproduce 24 patches of a color checker on a monitor with an average ΔE of 2.92–5.81. A printer profile metric is also described. This research describes three possible metrics for a printer profile–accuracy of the B2A1 tag (PCS to Device), A2B1 tag (Device to PCS), and a round trip test. Data are presented to show that the average accuracy of the output profile colorimetric intent (average of B2A1 and A2B1) for an Epson ink jet printer can be between ΔE 1.72–3.49. The accuracy of the printer profile is useful when considering the use of an ink jet printer in color management proofing workflows. This research proposes a ΔE metric system that can be used to evaluate the quality of commercially created ICC input, display, and output profiles. The data presented here are fundamentally a methodology that can be used to estimate the colorimetric accuracy of profiles. The use of commercial profiling products is only to illustrate how the metrics may be determined in practice. The data, however, do show useful information about the state of color management products today and this analysis can be used to track the improvements and evolution in ICC profiling software.