Similarity between faces in portraiture is incredibly informative for art historical questions involving the sitter's identity, as well for setting a painting in its historical context to understand why someone was depicted a certain way. A set of royal portraits from Song dynasty, China, has been the subject of rich art historical scholarship. Here, I demonstrate the usefulness of computer vision-based quantitative metrics in complementing existing rich subjective evaluations. Working with the portrait set, I show that L2 distances generated by OpenFace support the accepted hypothesis that emperor Lizong is depicted in Listening to the Wind in the Pines. I then use the technique to gain insight into whether the zither player in Listening to the Zither resembles emperor Huizong and why that might be, as well as what degrees of similarity between emperor portraits in the set may mean in terms of metaphorical inclusion or exclusion from the lineage. I then extend discussions on metaphorical inclusion to women in this set by exploring spousal similarity. Fascinating mysteries surrounding posthumous portraiture float amidst confounding factors in the clouds of memory, and this study shows the promise of using computer vision-based techniques as complements to subjective analyses in exploring these mysteries.