White lighting and neutral-appearing objects are essential in numerous color applications. In particular, setting or tuning a reference white point is a key procedure in both camera and display applications. Various studies on observer metamerism pointed out that noticeable color disagreements between observers mainly appear in neutral colors. Thus, it is vital to understand how observer metamers of white (or neutral) appear in different colors by different observers. Most observers who participated in a visual demonstration reported that white observer metamers appear pinkish or greenish but rarely yellowish or bluish. In this paper, this intriguing question, "Why observer metamers of white are usually pinkish or greenish?," is addressed based on simulations. Besides, it is also analyzed that which physiological factors play an essential role in this phenomenon and why it is less likely for humans to perceive yellowish or bluish observer metamers of white.