American Sign Language Interpreter Using Machine Learning

The Problem

Deaf and hard-of-hearing signers still hit a wall in everyday conversations with people who don't know sign language. Reading American Sign Language automatically is genuinely hard: gestures unfold as motion over time, not as single frozen poses, so a system has to understand movement, not just shapes.

What We Built

We built a real-time interpreter that watches a signer through an ordinary webcam and recognizes eleven dynamic signs as they are made. It is a proof of concept for the approach, not a general ASL translator. Rather than feed the model raw video, each frame is reduced to a synthetic hand silhouette drawn from tracked landmarks, which removes background, lighting, skin tone, and clothing as variables before training even starts. A 3D convolutional front end feeding a recurrent stack reads those motion sequences, and a word is only committed once twenty consecutive confident predictions agree, so a noisy per-frame classifier becomes something usable live. Fifteen independently trained models scored between 93.2 and 96.3 percent on a held-out set of 1,254 sequences. It shipped as the software deliverable for a three-student undergraduate thesis, handed over with the dataset, the trained models, and the evaluation code.

Results

• 93-96% accurate

  across 15 independently trained models, scored on a held-out set of 1,254 sequences

• 11 signs

  a fixed, deliberately scoped vocabulary of dynamic gestures, not general ASL translation

• 6,270 sequences

  labelled dataset built from scratch and handed over with the trained models and evaluation code

• Ordinary webcam

  no depth camera needed; landmarks are redrawn as silhouettes so lighting and skin tone stop mattering

Techstack