Trajectory prediction is crucial for autonomous systems, but traditional deep learning models, typically trained on specific pre-collected trajectories, often fail to generalize to unseen scenarios due to distribution shifts. Recent approaches address this by integrating online learning for adaptive deployment. However, existing online learning methods face two major challenges: (1) long training times, which prevent real-time execution, and (2) failure to account for variations in input data speed, leading to performance degradation when processing high-speed dynamic scenarios. To overcome these limitations, we introduce a latent-space predictor that forecasts future trajectories by aligning learned latent representations with encoded ground truth. This approach enhances robustness to distribution shifts while reducing reliance on direct coordinate regression. Additionally, we incorporate a lightweight online learning module, enabling efficient real-time adaptation without full model retraining. We evaluate our method on nuScenes, Waymo, and Lyft L5 datasets, focusing on data distribution shift scenarios. Experimental results demonstrate that our model outperforms state-of-the-art online learning methods, achieving approximate 9.9% improvement in trajectory prediction accuracy while significantly reducing optimization time up to 54%.