AI/ sign-language · accessibility · nlp · retrieval

Tool Turns Movement Descriptions Into Sign Language Words

SignTrace lets users describe a sign's movement in plain language and retrieves the matching word from a 6,699-entry Chinese sign-language dictionary.

A new tool called SignTrace turns a description of hand movements into the sign language word you have been groping for.

SignTrace is built for a specific problem: users remember how a sign moves but not its name or the formal codes linguists use to catalog it, so the system combines LLM-based dictionary enrichment, automated action extraction, dictionary-style query rewriting, seven-channel retrieval, and reranking across 6,699 dictionary entries. On a 500-query benchmark built from dictionary text, it returned the correct sign as the top result 94.0 percent of the time, and within the top nine results 97.4 percent of the time, with a mean reciprocal rank of 0.954. Reranking is doing most of the work, lifting top-result accuracy from 71.8 percent to 94.0 percent. It has already been tried by real users, who gave positive informal feedback, though each query takes a median of about 13 seconds to process when six people are searching at once.

Reverse lookup, describing what something looks or feels like to find its name, is a hard search problem even for text, and sign language has resisted it more than spoken language since there is no string of letters to type into a search box. If SignTrace's approach holds up outside the lab, it is a real accessibility win for beginners, not just people already fluent enough to know what to search for.

The catch: the benchmark queries were built from the dictionary's own wording, not from how actual learners phrase things, so the reported accuracy may not survive contact with real-world descriptions.

TR

The Revision

Written by an AI system from the public sources credited above. How we write →