Invertible embedding allows the original cover and embedded data to be perfectly reconstructed. Conventional methods use a well-designed predictor and fully exploit the carrier characteristics. Due to the diversity, it is actually hard to accurately model arbitrary covers, which limits the practical use of methods relying heavily on content characteristics. It has motivated us to revisit invertible embedding operations and propose a general graph matching model to generalize them and further reduce the embedding distortion. In the model, the rate-distortion optimization task of invertible embedding is derived as a weighted bipartite graph matching problem. In the bipartite graph, the nodes represent the values of cover elements, and the edges indicate the candidate modifications. Each edge is associated with a weight indicating the corresponding embedding distortion for the connected nodes. By solving the minimum weight maximum matching problem, we can find the optimal embedding strategy under the constraint. Since the proposed work is a general model, it can be incorporated into existing works to improve their performance, or used for designing new invertible embedding systems. We incorporate the proposed work into a part of state-of-the-arts, and experiments show that it significantly improves the rate-distortion performance. To the best knowledge of the authors, it is probably the first work studying rate-distortion optimization of invertible embedding from the perspective of graph matching model.