A new survey takes stock of the wild variety of ways researchers coax the Segment Anything Model into doing its job.
SAM, the foundation model that segments objects in images without needing retraining for new categories, works by taking a "prompt" - a click, a box, a bit of text - and turning that into a mask. This paper reviews how that prompting has evolved across SAM's fast-growing ecosystem, sorting methods into three buckets: geometric prompts (points and boxes), textual semantic prompts, and multimodal fusion prompts that combine both. It also tracks a shift away from prompts a person types or clicks by hand, toward automated ones generated by object detectors, prototype learning, reinforcement learning, and vision-language models. The authors survey where this has actually paid off: medical imaging, remote sensing, industrial inspection, and anomaly detection.
The interesting part is what the survey admits doesn't work well yet. Prompt sensitivity, cross-modal misalignment, and computational inefficiency are named as open problems, not solved ones. That matters because SAM's entire pitch was zero-shot generalization - point at something, get a mask, no fine-tuning required. If the prompt interface is still this fragile, that pitch is doing more work than the marketing suggests.
The model that promised to segment anything still needs something, or someone, to carefully spell out what "anything" means.