We present a new approach to making in-pixel color filters for a wearable display such as LCOS (liquid crystal on silicon) or a DLP (digital light processing) system. Unlike current color filters or methods used in these devices, our approach enables Red-Green-Blue (RGB) color images using color filters fabricated using semiconductor ultra-thin film. These filters are less than 100 nm thick, making them better suited to integration with light modulators than traditional pigment-based filters. Additionally, these films are fabricated via e-beam evaporation or RF sputtering, which is compatible with most modern chip manufacturing. In this paper, we present the design concept for these filters, including simulation results that demonstrate improved liquid crystal control when using these filters, demonstrate an example RGB filter array and also present an enhanced structure which could meet requirements of DLP based multispectral imaging. This is the first time the use of these filters is presented for use in liquid crystal based displays and DLP.