Field of view of the traditional camera is limited such that usually more than three cameras is needed to cover the entire surveillance area. The use of multiple cameras usually requires more efforts regarding camera control and set up as well as they need additional algorithms to find the relationships among the images of different cameras. In this paper, we present a multi-feature algorithm that employs only one omnidirectional camera instead of using multiple cameras to cover the entire surveillance region. Here we use the image gradients, the local phase information based on phase congruency, the phase congruency magnitude, and the color features, and they are fused together to build one descriptor named as "Fused Phase, Gradients and Color features (FPGC). The image gradients, and local phase information based on phase congruency concept are used to extract the human body shape features. Either LUV or grayscale channel features are used according to the kind of camera used. The phase congruency magnitude and orientation of each pixel in the input image is computed with respect to its neighborhood. The resultant images are divided into local regions and the histogram of oriented phase, and the histogram of oriented gradient are determined for each local region and combined. A maximum pooling of the candidate features is generated for one channel of the phase congruency magnitude and the three LUV color channels. All these features are fed to a decision tree Adaboost classifier for training and classification between the classes. The proposed approach is evaluated on a challenging omnidirectional dataset and observed promising performance.