This paper presents an automatic color segmentation algorithm for paper-based topographic maps. The algorithm uses an eigenvector line-fitting technique to overcome false colors introduced by the scanning process due to RGB misalignment. which is caused by the optical characteristics of the scanner lens and is mostly evident in regions of color change. The resulting false colors render traditional color clustering schemes ineffective. This approach based on an eigenvector line-fitting technique uses the color information of pixels in a 3 by 3 window to determine the true color of the pixel. This algorithm has been experimentally verified to be robust and accurate.