In contrast to three primary displays, multi-primary displays provide an essential enlargement of the color gamut and a higher degree of freedom to reproduce colors. The degree of freedom can be used to optimize color differences between color stimuli of the reproduction and the original for different observers.The simplest and least time consuming approach to control the primaries from input stimuli is the application of a matrix. Yet, one matrix is performing well for a certain number of stimuli only. Another matrix might achieve improved results for other stimuli but is performing worse for the first group. The basic idea of the algorithm presented here is to combine different matrices to fill the gap between oppositional requirements and to obtain adequate results in face of expense and quality. Altogether, four different matrices have been chosen in this study. None of the four matrices is appropriate on its own to achieve small color differences as desired for all colors.The matrices are constructed from the optimization of color differences for different observers (24 in total) and a large number of typical color stimuli. To display a specific color stimulus, one of the four matrices is selected with respect to minimum color errors for all observers. By combining different matrices, a fast control of a display at high color quality is realized.