Rate control is an essential module in video coding. Rate control strategies strive to deliver a stable playback experience as well as achieving high compression efficiency for modern video applications, constrained by restricted bandwidth and buffer limits. The difficulty of rate control often lies in the adaptation ability of the underlying algorithm to capture the variability of content and temporal correlation across frames. In this paper, we present a rate allocation algorithm to model the distortion propagation in the hierarchical coding structure premised on the temporal dependency model at frame level. Our experiments show that with the information collected from the temporal dependency model, the proposed rate allocation algorithm significantly improves the coding efficiency over the AV1 baseline on a set of variable user generated video clips.