This paper proposes a color interchange model between different objects in different scenes based on image segmentation. Proposed system is applied to preferred color reproduction, scene simulation, industrial design, etc. First, unsupervised image segmentation method in our previous paper is improved by coupling the Bayesian decision rule with K-means classifier as a starter for setting the initial seeds points. Here, we tested new methods for choosing the initial seeds points, such as, finding the higher population density center in color distribution or searching the mountain peaks in color histogram. Secondly we applied this model for automatic segmentation of key color areas in an image. After the segmentation, each segmented color area is projected onto PC (Principal Component) space and the paired segments are selected to transfer or mutually interchange the colors one to another between the corresponding pairs. Finally a color atmosphere in reference scene is transferred into a source scene by matching the PCs through the hue rotation and the variance scaling in PC axes between the corresponding paired segments. The paper introduces the experiments on typical applications to scene color change, scene-referred color correction, or flesh-tint reproduction for facial images.