#Sound: voices, music, effects and the mix

Everything you hear in a cinecode film is made by code except the voices, which come from Gemini TTS and are cached on disk. The score is composed and played by a synthesizer, the effects and ambience beds are synthesized, and the mix is done in JavaScript before ffmpeg's loudness normalization.

node bin/cinecode.mjs voice films/my-film   # generate missing lines (needs GEMINI_API_KEY)
node bin/cinecode.mjs music films/my-film   # the score alone → out/music.wav
node bin/cinecode.mjs mix films/my-film     # everything → out/mix.wav, mastered to −16 LUFS

#Voices

Every say line is spoken by one of the 30 prebuilt Gemini voices (model gemini-3.8-flash-lite-tts). A film maps its speakers to voices in voices (storyboard.md):

"voices": {
  "narrator": { "voice": "Sulafat", "tone": "warm, gentle, unhurried" },
  "pip":      { "voice": "Leda",    "tone": "bright, eager" }
}

The voices, with the casting note the engine keeps for each (VOICES in engine/audio/voice.mjs):

Names are matched without regard to case. None of the voices is a child's: for young girls the starter uses Leda with "bright, eager" and the demo film Leda with "small, high, earnest".

#Tone

A line is sent as [tone] text: the bracket is a direction the model should follow, not read. The tone is the voice's tone joined with the line's own tone (duplicates dropped), and the first try keeps only the last three descriptors, so the line's own delivery, which comes last, always survives.

Use delivery adjectives only — nouns get read aloud. Good: "dry", "soft, tender", "squeaky, disbelieving". Bad: "grumpy old badger", "voice catching", a list of six adjectives, a stage direction. Those come back spoken before or instead of the line. Pauses are beats ("after": 0.6), not "…" in the text.

When a take reads the direction aloud, the next try sends only the last descriptor, and the one after that no direction at all (the voice alone carries the line). Such a take can pass with the line's words right and its delivery gone; takes.mjs (below) flags it TONE-CUT.

No word is safe every time. Read aloud in the demo film or a test film: "wondering", "hopeful", "certain", "less sure" (each even as the whole direction), "steady", "simple", "deadpan", "whispering", "gently offended", and [warm, gentle, soft] as a whole (it took with "tender" for "soft"). Never read aloud there: "teasing", "tender", "scared", "grateful", "delighted", "excited", "quiet", "solemn", "earnest", "breathless". When a tone is read aloud, change the word rather than retrying it.

Made-up names are often misheard by the transcriber ("Glim" came back as "Clem", then "Glem"), which fails a good take and uses up tries: read what was heard before you redo a take, and prefer names that are real words or names.

Non-verbal lines: a text starting with [ ("[laughs]") is a non-verbal caption. Give the voice something to say with tts ({ "text": "[laughs]", "tts": "Hahaha!" }): the caption stays, the voice says the tts text.

#QA

Every new take is transcribed by gemini-3.8-flash and compared with the line:

• words are compared lowercased, without punctuation or accents (don't = dont, 7 = seven up to twenty);
• a take passes when its word error rate is at most 0.2 and no words were inserted before the line (a take that starts "Huh, …" or "Um, …" fails);
• a take that speaks the direction (inserted before the line, or instead of it) is marked and the direction is simplified for the next try;
• a non-verbal line has no words to check: it passes unless a word of the direction is heard (a laugh that comes out as "Giggling. Hahaha!" fails).

A line gets up to three takes; the best one is kept (lowest error, with a penalty for inserted words). If the transcription service fails, the take is kept unchecked. The voice command then prints every line (the start of its id, the speaker, the take's length, its error rate and the text), marks those still above 0.2 with ! and counts them.

node .claude/skills/cinecode/scripts/takes.mjs films/<slug> lists flagged and non-verbal takes with what was asked and what was heard, every try of a flagged line, and TONE-CUT for a kept take that was sent with less direction than asked (or none) because earlier tries read it aloud. With --all it lists every take, each with its length against the estimate and any pause of 0.35 s or more inside it: read them all, since a take can pass and still be wrong. Two traps seen: a one-word line or a homophone ("Ours." was heard as "hours" and "Walrus" on every try; "Ours, then." passed), and a slow voice (Algenib, "gruff, low") taking twice the estimate with a pause between two sentences. To fix a take, simplify its tone (that makes a new line), or delete its two cache files and run voice again.

The kept take is trimmed of silence below −45 dBFS (10 ms windows, 60 ms kept around the speech), resampled to 48 kHz and given short edge fades. Its loudness envelope at the film's fps drives the speaker's mouth (lip sync). A line's trimEnd cuts the end of the take after the cache, with a two-frame fade; timing and lip sync follow the cut.

#The voice cache

Each line is stored in films/<slug>/voices/ as <id>.wav (48 kHz mono, 16-bit) and <id>.json (text, voice, tone, the prompt sent, the models, the QA transcript and error of every take, the envelope). The id is a hash of who speaks, the voice, the spoken text and the tone, so changing any of them makes a new line; an entry is reused while its text, voice and tone still match. Two identical lines share one take. A line whose text, tone or voice changes leaves its old take behind; it is never used again. takes.mjs lists those unused takes with the command to delete them, which is safe (they only cost a request if you change the line back).

Once every line is cached, the film plans, mixes and renders with no network and no key. plan uses the real take lengths (unvoiced lines show an estimate of words / 2.6 + 0.3 s, marked ~). voice sends only missing lines, two at a time, retrying on rate limits (HTTP 429), server errors and network errors with exponential backoff. render voices missing lines first when GEMINI_API_KEY is set and stops when it isn't; mix and animatic skip unvoiced lines with a warning.

voice needs GEMINI_API_KEY in the environment, even when every line is cached. Keep the key in your shell profile; don't paste it into files or command lines.

#Music

The score is composed from the film's music cues each time it is rendered (seeded by the film's top-level seed, so it is the same every time). A scene sets a cue with music (a mood name or { "mood", "intensity", … }), and a music beat changes the cue, plays a sting or silences the score at that moment (storyboard.md). "music": "none" fades the score out over 0.3 s.

#Moods

Intensity (0..1, default 0.5) brings layers in at phrase boundaries: pad, accompaniment, melody, bass, counter-line, percussion, doublings. 0.3 is sparse, 0.8 is full.

How cues fit together: music is written in four-bar phrases, and the tempo is nudged (up to ±13%) so a cue's last tonic chord lands about 3 s before it ends. Consecutive cues of the same mood merge into one continuous cue whose arrangement follows the intensity changes; different moods crossfade over 1.5 s. Two consequences to listen for (or measure with node .claude/skills/cinecode/scripts/levels.mjs films/<slug>, which flags a second or more of near-silence under a cue):

• A new mood that starts on a cut without "downbeat": true can open with a near-silent lead-in of a second or more (seen: 1.5 s under a line, at −55 to −70 dB). Give a mood change on a cut "downbeat": true.
• The last cue runs under the end card, but its final chord lands about 3 s before the film's end and dies away, so under a 2.5 s end card the score is already gone. "hold": true on the last cue sustains it to the end. Without a score key, each cue picks a key from its mood's preferred keys, close on the circle of fifths to the previous cue.

#The motif

A film can have a tune of its own, given once in score:

"score": { "key": "G", "motif": "D5:1 B4:0.5 D5:0.5 E5:2 G5:3" }

motif is note:beats tokens (G4:1, Bb3:0.5, F#5:2) written in key ("G", "Eb", "F#m", "A minor"). With a score, every cue is in the score key and the composed melodies grow out of the motif's opening intervals. The motif is stored as scale degrees, so in a minor mood it is stated in the minor mode (the A of a C-major motif becomes A-flat in C minor).

A cue states the motif when asked:

The chords under a statement are chosen so every long or strong motif note is a chord tone. A motif cue starts exactly on its time; the cue before it holds its final chord into the statement only when that chord fits.

A good plan: a few notes of the motif early ("motif": 3), quietly, on a solo voice; the full statement saved for the turn or the ending ("motif": true, "motifVoice": "tutti").

#Stings

Short accents placed with their hit exactly on the beat's time, in the key of the music around them, with the score dipped underneath:

#Sound effects

An sfx beat plays a synthesized effect (storyboard.md). Effects are calibrated to sit under the dialogue; gain scales one, and params.level trims it in dB. Each event is seeded by its name, time and shot, so editing one leaves the others alone.

Aliases: footstep (grass), squeak, tweet, flap, hoot, croak, creak and gate-creak (door-creak), leaves (rustle), bird (birdsong). Every effect also takes level (a trim in dB). A hum is treated like a voice: the score ducks under it ("params": { "duck": false } turns that off, true ducks the score under any effect).

A film can add its own sounds in custom.mjs (effects.md).

#Automatic sounds

You don't write footsteps. Every step of a walk or run plays a footstep for the ground under it (the set's surface and paths: grass, dirt, stone, wood), louder for heavier rigs; hops and landings play hop and land; owls and songbirds flap when they fly; steps in water splash and wading sloshes. Running on dirt kicks up dust (a visual fx). Sounds are panned by where they happen on screen.

#Ambience

Each scene plays an ambience bed under everything: "ambience": "auto" (the default) takes the kit's day or night bed by the scene's time (sets.md), a name forces one, "none" is silence.

Beds never loop (every layer is a seeded process), start already alive, and crossfade over 1 s when the bed changes.

ambience takes only these eight. For a bed of your own (rain, a storm), write it as a sound in custom.mjs and lay it as one long effect at the start of the scene, with its length in params rather than on the beat: { "sfx": "rain-bed", "params": { "dur": 30 } } takes no time in the shot and plays on across the cuts (a beat-level "dur": 30 would hold the shot for 30 s). The same works for the continuous built-ins (wind, stream, fire, cricket).

#The mix

mix (and animatic and render, which mix first) builds the soundtrack:

out/music.wav is the score alone (48 kHz 24-bit) for judging the moods. Lines without a voice yet are skipped with one warning, so a film can be mixed (and an animatic made) before it is voiced. The mastering step runs ffmpeg from PATH.

This page on GitHub