A voice memo feels like a file. For a writing system, it is better understood as an evidence package: audio, timing, words, uncertainty, revisions, and the decisions that connect them. Speech recognition can produce a transcript, but a trustworthy writing workflow must answer harder questions. Which recording did a sentence come from? Where was the model unsure? What did an editor change? Which phrases carry the speaker’s personality and should survive cleanup?

Those questions change the design. The goal is not merely to turn sound into text. It is to build a small, inspectable data pipeline that carries a thought from a phone to a durable draft without silently rewriting the person who spoke it.

Capture first, interpret second

The safest boundary begins on the phone. Apple’s Voice Memos app can display and copy transcriptions on supported devices and languages, which makes a manual paste a useful first bridge: it is simple, visible, and does not pretend that a website has access it does not have. Apple documents that workflow directly.

Automatic transcription can come later. OpenAI’s Whisper work demonstrated that large-scale, weakly supervised training can produce robust multilingual speech recognition, while Mozilla’s Common Voice provides an open speech dataset designed to broaden the voices represented in language technology. Together, they point to two infrastructure requirements: the recognizer should be replaceable, and the data used to evaluate it should reflect the people who will actually use the system. See the Whisper overview and Mozilla Common Voice datasets.

The raw audio should remain immutable. A derived transcript can be regenerated; the original recording cannot. The capture record needs a stable identifier, local filename, duration, recording time when known, language, and a cryptographic checksum. The checksum answers a quiet but important question: are we still talking about the same recording?

A transcript is a set of observations

A useful transcript is not one long string. It is a sequence of segments linked to ranges in the audio. A minimal model might contain:

recording(id, checksum, duration, language, created_at)
transcript(id, recording_id, engine, engine_version, created_at)
segment(id, transcript_id, start_ms, end_ms, text, confidence)
revision(id, segment_id, before, after, reason, created_at)
anchor(id, recording_id, phrase, note)

This structure separates facts from interpretations. The recording exists independently of any model. A transcript identifies the engine and version that created it. Segments retain timing and uncertainty. Revisions preserve the difference between what the recognizer heard and what an editor decided. Voice anchors identify phrases, rhythms, or recurring expressions that should resist generic cleanup.

Confidence is useful, but it is not truth. Some engines calibrate it poorly, and a fluent-looking error can receive a high score. The interface should therefore expose uncertainty without turning the writer into a quality-control operator. Flag improbable names, very low-confidence spans, and abrupt language changes; leave ordinary text alone.

Editing needs two lanes

Mechanical cleanup and authorial rewriting are different acts and should be stored differently.

The first lane is conservative. It can normalize repeated filler, obvious spacing, and duplicated fragments while leaving word choice and sentence order intact. Every change should be reversible. The second lane is editorial: reorganizing an argument, tightening prose, or suggesting alternatives. That lane should never replace the source silently, and its suggestions should appear as a diff or parallel draft.

This distinction is how the system preserves “voice and soul” in operational terms. Voice is not a mystical model setting. It is observable evidence: characteristic phrases, sentence length, degree of formality, favorite transitions, intentional repetition, and the writer’s tolerance for rough edges. Store those preferences as anchors and constraints, not as permission for a model to imitate a person without review.

Local-first is a data architecture

Privacy is more than a banner that says “no upload.” A local-first implementation must make the network boundary testable. Parsing, cleanup, preview, and export can run in the browser. If automatic recognition later requires a server, the interface should state exactly when audio leaves the device, where it is processed, how long it is retained, and how the user can delete it.

The default retention policy should be short and explicit. Raw recordings do not belong in browser storage by accident. A prototype can keep only pasted text and local file metadata, never the audio bytes. A production system can use encrypted object storage for audio, a relational database for provenance and revisions, and a search index generated from approved transcript text. Each layer should have its own deletion path so removing a recording also removes its derived artifacts.

Encryption protects stored bytes, but access rules protect people. The system should separate the person who can listen to audio, the service that can transcribe it, and the editor who can approve text. Audit events should record access and transformations without copying sensitive transcript content into logs.

Evaluate the whole journey

Word error rate is necessary, not sufficient. A recognizer can score well while consistently damaging names, technical terms, or a speaker’s dialect. Evaluation should combine several measures:

  • word and character error rates by language, device, and acoustic setting;
  • named-entity accuracy for people, places, and domain vocabulary;
  • time needed for a person to reach an approved draft;
  • number and type of editorial reversals;
  • retention of voice anchors; and
  • privacy checks confirming that audio and text travel only where promised.

The best system is not necessarily the one with the smallest raw transcription error. It is the one that helps a person move from thought to trustworthy draft with the fewest hidden decisions.

A practical first release

The honest first version is intentionally modest: select a Voice Memo for local metadata, paste the phone’s transcript, mark the phrases that sound most like the speaker, apply a gentle cleanup, compare the result, and export Markdown. No simulated cloud transcription. No invisible grammar service. No claim that a local webpage can read an iPhone library.

That narrow workflow creates the right seams for a more capable system. An automatic recognizer can later populate the same transcript records. A style engine can propose revisions through the same diff. Sync can move encrypted evidence between devices without changing the data model. Starting with provenance makes those additions easier; starting with a text box and an opaque model makes them harder.

Speech recognition is the visible feature. The lasting product is the chain of custody from a person’s voice to words they still recognize as their own.