PHONEME · proof-of-concept demo
Offline pronunciation checking for young readers, at the sound level
A child reads a word, the phone listens, and a phoneme model running on the device checks every sound. It can tell "rabbit" from "wabbit" and gives gentle feedback. No server, no upload, no internet needed after install.
- Audio never leaves the device
- ONNX Runtime (Web here, Android in the APK)
- React Native Android release APK
- 5 open phoneme models benchmarked
Try it
Pick a word, then play a sample voice or read it into your microphone. The model runs inside this page (WebAssembly). Nothing is uploaded.
1 · Sample voices synthetic TTS, not real children
2 · Your voice
Records 2.5 s at 16 kHz in the browser. Audio stays in memory on this device.
Sound-by-sound
sound OK flagged vowel, shown but not scored (accent-tolerant mode)
How to test offline: load the model once, switch on airplane mode and reload. The page, the model and the samples all come from this device's cache.
Android app: React Native + ONNX Runtime Mobile
Same engine running natively on the phone. The release APK does not request the INTERNET permission, so it physically cannot send audio anywhere. Microphone capture uses AudioRecord at 16 kHz and inference uses ONNX Runtime Android (CPU). Scoring and feedback are the same JavaScript as this page.
Install: download on an Android phone, allow "install unknown apps" for your browser, open the file. Demo-signed build.
Verified on device: I ran all 120 test clips through the native app pipeline (ONNX Runtime Android + the shared scoring). It gave the same results as the Python benchmark: 68% of correct words accepted, 93% of errors detected. Model time was about 120–230 ms per 2 s clip on the x86_64 emulator. Screenshots and video are from that emulator, not a phone. Measuring real mid-range phones is part of the PoC.
Model tests I ran for this demo
Five open phoneme models, exported to ONNX and quantised to int8, tested on 120 clips: 10 words × 6 voices × (correct + a typical child error such as r→w, k→t, th→f, s→t). Same scoring code as the demo. Consonants only. The voices are synthetic TTS, not real children, so these numbers are a smoke test and not a final accuracy figure.
| Model | Params | Precision | Size | 2 s audio (laptop CPU) | Correct words accepted | Errors detected | Error named right | License |
|---|
Shipped model, by voice
Bars: correct words accepted / errors detected. The child-style voices (Ana, Maisie) are the hardest for a model trained on adult speech. That is the core risk this PoC needs to measure with real recordings of children.
What I learned
- int8 is the right default: 3× smaller and faster than fp32, with almost the same accuracy, as long as the conv feature encoder stays fp32. When I quantised the convs too, error naming fell from 58% to 32%.
- Bigger isn't free: the 315M multilingual models catch more errors but are 356 MB in int8, too heavy to ship as they are. The likely path is to distil or fine-tune a small model on child speech.
- Child voices are the weak spot for every adult-trained model. Fine-tuning on child speech (with consented recordings) is the main lever.
- Accent ≠ error: the Singapore and Indian voices said "tree" for "three". That's normal local English. The scoring policy has to be configurable per market.
- Speed is fine: about 60 ms of model time for 2 s of audio on a laptop CPU, plus about 10 ms of scoring. A mid-range phone should be well inside 1–2 s, but that still has to be measured.
How it works
- Capture16 kHz mono, 2–3 s per word. No storage, no upload.
- Phoneme modelHuBERT/wav2vec2 CTC, int8 ONNX. It outputs per-frame probabilities over 39 English phonemes.
- CTC goodness-of-pronunciationScores the expected sounds against every one-sound substitution or deletion. No forced aligner needed.
- Child feedbackTurns "R→W at position 1" into "listen for the r sound… growl like a tiger". Adults see the detail.
Why this approach
A plain speech-to-text model "corrects" what it hears. Children say "wabbit" and the model writes "rabbit". This one compares the audio against the exact sounds the child was asked to read. So it can say which sound was off and what it sounded like, which is what reading feedback needs.
Same core on iOS + Android
ONNX Runtime Mobile runs on both: CPU/XNNPACK everywhere, with optional CoreML (iOS) and NNAPI/QNN (Android) acceleration. The scoring is plain TypeScript shared by the React Native app and this web page, so iOS gets the same results.
Proposed paid PoC: what it would deliver
Model shoot-out on child speech
Score candidate models on children's recordings (public child-speech corpora where licences allow, plus your own consented samples). Report per-sound detection and false-alarm rates, by accent.
Make it small and fast
Fine-tune or distil the best model, quantise it, and measure latency, memory and battery on real mid-range Android phones and iPhones. Target: feedback in under 1–2 s.
RN integration, both platforms
A reusable React Native module for iOS and Android: mic capture, VAD, model, scoring, plus a simple child-facing screen like this demo. Hand-over with test results.
This demo already covers the plumbing: model export, quantisation, the on-device runtime, scoring and the Android build. What it does not have yet is accuracy on real children's speech. That is what the paid PoC would measure.