Breast ultrasound serves as a cornerstone non-invasive technique for the screening and diagnosis of breast diseases. To address the challenge of limited feature generalization in few-shot learning for breast ultrasound image classification, this study proposes a novel cascaded Cross-Covariance Vision Transformer (XCViT) framework. A dual-stage architecture is introduced. Stage A employs hierarchical feature distillation via a three-level cascaded-XCViT module with Teacher–Student frameworks, generating dual-path tokens and utilizing token spectral clustering for semantic downsampling. Stage B uses frozen encoder outputs to derive prototype vectors from support-set features, enabling training-free inference via distance metric learning. Evaluation showed accuracies of 79.8% (1-shot) and 89.7% (5-shot) on CIFAR-FS, 47.2% (1-shot) and 67.3% (5-shot) on FC100, and 61.5% (5-shot) on a clinical breast ultrasound dataset, outperforming existing methods by 2.0–5.6%. The method demonstrates strong performance in low-data regimes and offers a promising solution for few-shot clinical auxiliary diagnosis.