What Constitutes Food Similarity? Visual Representations and Retrieval in Indian Cuisine

The study treats food similarity as a relational idea. It asks which foods appear near one another when a visual representation compares them, rather than asking only whether each image is correctly identified. This matters because retrieval, recommendation, substitution, and other food computing tasks depend on meaningful relationships among foods.

The researchers examine image-to-image and image-to-dish neighborhoods. They also study whether those neighborhoods reflect culinary families, reveal surprising contrasts, or cross culinary boundaries. At the dish level, they aggregate image representations and compare the resulting visual structure with recipe-derived similarity for the same dishes.

The central finding is that similarity depends on the representation, modality, and abstraction level. Image-level neighborhoods vary substantially across models, while dish-level aggregation exposes some shared visual structure. That structure corresponds less strongly with recipe-based relationships, so recognition accuracy alone cannot define food similarity.

The study compared seven pretrained visual representations of food. These representations came from four broad model types named in the abstract: convolutional, transformer-based, self-supervised, and image-text aligned models. The comparison was designed to test whether different ways of encoding images produce different notions of food similarity.

Each representation supported image-to-image retrieval and image-to-dish neighborhood analysis. The researchers examined culinary-family structure, contrast cases, and agreement across representations. They also aggregated image representations to estimate visual similarity between dishes, then compared that structure across models and against recipe-derived similarity.

The important point is that the models were not interchangeable. Image-level neighborhoods varied substantially across representations, including similarities between foods from different culinary families. This means the number and type of representations matter when studying food relationships, not just recognition performance from one model.

The study found that image-based similarity does not reliably follow culinary identity. Foods that look alike can be placed near one another even when they belong to different culinary families. Meanwhile, foods sharing a culinary family may be separated when their appearances differ.

The researchers tested this through image-to-image retrieval and image-to-dish neighborhoods. They examined culinary-family structure, contrast cases, and agreement among seven visual representations. The same food collection could therefore produce different relational neighborhoods depending on which pretrained representation encoded the images.

At the dish level, aggregating image representations revealed some common visual structure across models. Yet image-level neighborhoods varied substantially, and cross-family similarities remained visible. The result is a warning for retrieval and recommendation systems: visual proximity can reveal useful connections, but it does not automatically represent culinary relationships.

Indian cuisine gives the researchers a demanding test of food similarity. The abstract identifies a key tension: visually similar foods may have different culinary identities. At the same time, foods that belong to the same culinary family can vary substantially in appearance.

This tension lets the study test more than recognition. Image-to-image and image-to-dish retrieval can reveal whether a representation groups foods by shared visual cues, culinary family, or some mixture of both. The researchers also examine contrast cases and agreement across representations to see which relationships remain stable.

The setting therefore exposes weaknesses that an easier collection might hide. A model can recognize dishes while organizing them in ways that cross culinary-family boundaries. Studying Indian cuisine helps show that food similarity is representation-dependent, and that visual structure should be examined before combining it with recipe-based information.

When visually similar foods have different culinary identities, an image-based system may retrieve them as neighbors despite their different culinary roles. When foods from the same family look very different, the system may place them far apart. In both cases, appearance and culinary relationship do not align.

The study investigates these effects through image-to-image and image-to-dish neighborhoods. It examines culinary-family structure and contrast cases across seven pretrained visual representations. The key mechanism is the visual representation itself: its encoded image features determine which foods become close or distant in the learned space.

The observed consequence is unstable relational structure. Image-level neighborhoods vary substantially across representations and can cross culinary-family boundaries. This matters for retrieval, recommendation, and substitution. A system should not assume that visual closeness captures culinary similarity, and future multimodal systems must understand these mismatches before combining image and recipe information.

Image-based similarity is built from visible appearance, while recipe-derived similarity comes from information about the recipes. These modalities describe dishes through different signals. A dish can therefore look similar to another dish without having a closely related recipe-based description.

The researchers tested this by aggregating image representations for the same dishes. They compared dish-level visual similarity across models with similarity derived from recipes. Aggregation reduced some local image variation and revealed common visual structure across representations, but that structure did not align as strongly with recipe-derived similarity.

This difference matters because multimodal food systems often need one coherent notion of similarity. The findings show that visual and recipe information should not be merged as though they already describe the same relationships. Their structures need to be studied first, so retrieval, recommendation, and substitution systems can combine them more carefully.

A learned visual representation converts an image into an internal pattern of features. Images with similar encoded patterns become neighbors in that representation's space, making them candidates for retrieval. The study uses pretrained representations to examine relationships among food images, rather than relying only on dish labels.

The mechanism is relational. A model compares encoded images and ranks nearby items for image-to-image or image-to-dish retrieval. Different model families may emphasize different visual signals, so the same foods can occupy different neighborhoods. The researchers compare seven representations and measure their agreement, culinary-family structure, and contrast cases.

This explains why a learned neighborhood is not a universal map of food. The study finds substantial variation at the image level across representations. Aggregating images at the dish level reveals more common visual structure, but it corresponds less strongly with recipe-derived similarity. Representation choice therefore shapes what “similar” means.