Jump to content

Offline captions for deaf and hard of hearing people

From IdeaWazaWiki

How Speech Recognition That Runs on the Phone Turns Captioning From a Booked Service Into a Private, Always-On Tool

Offline captions for deaf and hard of hearing people are written transcripts of speech produced by software that runs on the listener's own phone or computer, with no internet connection and no outside server. This page is about futurism and solving problems in living: it argues that local speech recognition can move captioning from scarcity toward abundance, turning a skilled service that people must request and wait for into a private tool they carry everywhere, while showing why the people who read the captions still need control over errors and the right to ask for human interpreters.

The core thesis is simple. For most of history, access to spoken conversation for a deaf person depended on someone else: an interpreter, a captioner, a relative, or a company server. Captions that run on the device shift that dependence back toward the listener. The interesting question is not whether the software is perfect, because it is not, but who controls it, who sees the words, and who gets to correct it.

Why this matters

The World Health Organization estimates that over 5 percent of the world's population, about 430 million people, need rehabilitation for disabling hearing loss, and it projects that over 700 million people, about one in ten, will have disabling hearing loss by 2050. The same fact sheet says nearly 80 percent of people with disabling hearing loss live in low and middle income countries, and that more than 25 percent of people older than 60 are affected.

Those numbers describe a large group of people whose access to ordinary talk at a pharmacy counter, a family dinner, a union meeting, or a bus station is limited by the supply of human help. Human real-time captioning and sign language interpreting are skilled work, and skilled work is scarce, especially in poorer regions and in small, unplanned moments. A tool that runs on a phone already in someone's pocket changes the supply curve for those small moments.

From cloud service to pocket tool

Two changes made offline captions practical.

The first was smaller, faster models. In October 2019 Google introduced Live Caption on the Pixel 4. Its product manager wrote that captions were produced in real time and completely on the device, that the feature worked without cell data or Wi-Fi, and that the captions stayed private and never left the phone. At launch it captioned media and spoken audio on the device but did not caption phone and video calls. Apple now offers Live Captions on iPhone 11 and later models, covering audio in apps, calls, and conversations picked up by the microphone, with an option that announces to callers when a call is being transcribed.

The second was open models. In 2022 OpenAI released Whisper, a family of speech recognition models trained on 680,000 hours of audio paired with transcripts from the internet, including 117,000 hours covering 96 languages other than English. The authors released the model weights and code. Their smallest model has 39 million parameters, and they reported that their best English results came close to professional human transcribers on a set of 25 recordings. Because the weights are public, independent projects such as whisper.cpp have rebuilt the model in plain C and C++ so it can run on laptops, phones, and even a Raspberry Pi, and its developers demonstrated it running fully offline on an iPhone 13.

Together these changes mean captioning no longer has to be a service someone grants. It can be a capability someone owns.

Privacy is part of access

Captioning a conversation means recording what other people say. When that audio goes to a remote server, a deaf person's ordinary life becomes a data stream that a company can store, analyze, or be forced to hand over. Hearing people do not have to send their conversations to anyone in order to understand them. On-device processing removes that unequal burden.

This is why offline captions fit naturally with local-first software and the broader idea of local AI as a shared public resource. The value is not only convenience. It is that a disabled person should not have to trade privacy for participation.

Where the machine gets it wrong

A fair account has to describe the failures, because the people reading captions are the ones who carry the cost of errors.

A 2024 study presented at the ACM Conference on Fairness, Accountability, and Transparency, titled "Careless Whisper," ran 13,140 audio segments through the Whisper API in 2023. The researchers found that about 1.4 percent of transcriptions contained entire phrases or sentences that were never spoken, and that 38 percent of those invented passages contained harmful content such as violence, made up names or health statuses, or false claims of authority like website links. The invented text appeared more often for speakers with aphasia, a language condition that produces longer pauses: 1.7 percent of their segments compared with 1.2 percent for the control group. The authors linked the problem to silence and noise inside the audio. When they retested in December 2023, most of the earlier cases were gone, but 12 of 187 problem segments still produced invented text. They found no comparable invented passages from several other commercial services on the same segments.

The Whisper paper itself notes that accuracy is still poor for many languages, and that error rates fall roughly by half for every sixteen-fold increase in training data for a language. That means languages with little recorded text online get worse captions, which falls hardest on the low and middle income regions where most people with disabling hearing loss live. Apple's own support page warns that Live Captions accuracy may vary and should not be relied upon in high-risk or emergency situations.

The lesson is that a transcript that looks fluent can still be wrong in ways a reader cannot detect, and that the risk is not shared equally.

Design principles that keep the listener in charge

If offline captions are going to expand freedom rather than create a new kind of dependence, a few principles help.

  • The listener should decide when captions run and where any saved transcript goes, with saving turned off by default.
  • Captions should show uncertainty, for example by marking low-confidence words, so a fluent invented sentence does not pass as fact.
  • Transcripts should never be treated as an official record of what someone said without human review, especially in hiring, courts, police contact, or medical notes.
  • Open models and open code let communities test accuracy on their own languages, accents, and speech patterns and publish the results.
  • Software captions should add to, not replace, the right to a qualified human interpreter or captioner when the stakes are high.

Limits of this approach

Offline captions are not a substitute for sign language. For many Deaf people a signed language is a first language, and reading fast English text is a second-language task. Captions also struggle with crosstalk, distant speakers, and noisy rooms. Phones need battery, memory, and recent hardware, and the newest features often arrive first on expensive devices. Finally, a capability being available does not mean institutions will stop owing human access; there is a real risk that employers, schools, or agencies point to a free phone feature as a reason to cut interpreter budgets. This page describes technology and its social effects. It is not medical, legal, or audiology advice.

  • Explain how moving captioning from a booked human service to a phone feature changes who holds power in a conversation between a deaf person and a hearing institution.
  • Compare the privacy costs of cloud captioning and on-device captioning, and argue whether disabled people should ever have to accept recording as the price of access.
  • Describe how invented sentences in a transcript could harm someone in a job interview, a court hearing, or a doctor's note, and propose safeguards that keep the speaker in control.
  • Analyze why speech recognition works worse for languages with little recorded text online, and design a community project that could close that gap without exporting people's voices to distant servers.
  • Argue whether free caption features on phones should strengthen or weaken the legal duty of schools and employers to provide human interpreters.
  • Imagine a city in 2040 where every public counter, bus, and meeting has open, offline captions, and describe what new problems in living would appear and how they could be handled.

See also

References

Retrieved from "https://ideawaza.com/index.php?title=Offline_captions_for_deaf_and_hard_of_hearing_people&oldid=131605"