Most vision-language AI models can identify a golden retriever but stumble when asked to place it correctly in a hierarchy — dog, then canine, then mammal, then animal.
Researchers at Peking University's ICST-MIPL lab published a method called Hierarchical Representation Regularization, or HiR², that adds taxonomy awareness to large multimodal models as a plug-and-play module rather than a ground-up redesign. The system builds a visual tree by pulling coarse-to-fine features from intermediate layers of an existing language model, guided by text cues. Two loss functions do the enforcement work: one uses hyperbolic geometry to make the model respect parent-child concept relationships, and another pushes similar-but-distinct concepts apart angularly without disrupting the hierarchy. The researchers say it works across different model architectures and fine-tuning setups, and the code is already public.
The gap this targets is real and underappreciated. When a model conflates a category with its parent class — or ranks a sibling concept as more similar than a parent — downstream tasks like image search, medical imaging, and product classification break in subtle, hard-to-debug ways. A regularizer that slots into existing fine-tuning pipelines is a more practical fix than asking teams to redesign their training objectives.
The catch is that benchmarks on hierarchical visual recognition are still thin, and a technique that scores well on research datasets does not always survive contact with messy production data — a gap the field has seen before with structured prediction methods that looked strong in the lab.