A fast calculation method for full-color holographic system with real objects captured by a depth camera is proposed. In this research, the depth and color information of the scene is acquired using a depth camera and the point cloud model is reconstructed virtually. Because each point of the point cloud is distributed precisely to the exact coordinates of each layer, each point of the point cloud can be classified into grids according to its depth. A diffraction calculation is performed on the grids using a fast Fourier transform (FFT) to obtain a computer-generated hologram (CGH). The computational complexity is reduced dramatically in comparison with conventional methods. The numerical simulation results confirm that our proposed method is able to improve full-color CGH computational speed.