Various color difference metrics were developed for characterizing the perceived color difference between individual color patches. Color difference between palettes containing multiple color patches, however, is critically important in product design and computer graphics. This study aimed to investigate how the perceived color difference between a pair of color palettes containing more than a single color patch is affected by the order and number of color patches in the palette. Two reference color sets were generated and each set had four color palettes containing 1, 4, 9, and 16 color patches that were arranged as 1 × 1, 2 × 2, 3 × 3, and 4 × 4 patterns. Human observers scaled the color differences between a color palette of the reference set and a color palette that had revised colors, or revised orders, or a combination of revised colors and orders compared to the reference palette. The calculated color differences between the two palettes were derived using the Minimum Color Difference Model (MICDM) algorithm proposed in a recent work with different color difference metrics, including CIELAB, CMC, CIE94, and DE2000. It was found that the perceived color differences of pairs of individual color patches were significantly larger than those containing multiple patches, when the calculated color differences were the same. The color differences metrics, except for CIE94, had similar performance when characterizing perceived color differences between color palettes.