What rs-rich-doom needed from rs-rich: gaps for core¶
Copied on 2026-10-25 from rs-rich-doom's docs/core-gaps.md (its branch
claude/next-priority-items-3c5p8e, at 2ca5783). The maintainer moved
these gaps to 0.0.23, except gap 4, which lands in
0.0.22 with workstream 2. This copy is what the
plans cite; Doom's own copy may move on.
Doom is a demanding, repeatable workload for rs-rich: 35 new frames a second
of 320x200 pixels, keys held and released, a clock to keep, and sound. This is
what it needed and had to build itself, written against the published 0.0.21
crates (rs-rich 0.0.11, rs-rich-ext 0.0.18, rs-rich-art 0.0.14,
rs-rich-interact 0.0.9, rs-rich-intuituive 0.0.6). The integration branch
claude/next-priority-items-3c5p8e-wip-integ3 (2135f4b) was checked too:
rs-rich-doom builds and passes on it, and none of these gaps is closed there.
Each gap below now opens with its status in 0.0.23.
Each entry follows rs-rich's rule that core holds generic pieces only. Each proposal names a second, non-Doom use, and nothing Doom-specific (WAD, DMX, MUS, OPL, Doom's keys or pan law) is proposed for core. Those stay here.
What already landed and is used here: pixel cells (Canvas::put_rgb and the
pixels_fit node), style-table compaction, held keys (held_keys,
all_keys, synthesize_releases), fixed_step with catch-up, rate,
StatsHandle, DrawCx::rect() and the audio output (audio::output, volume).
The review's three findings, verified on 0.0.21¶
| Finding | Verdict on 0.0.21 |
|---|---|
A live in-memory frame source for intuiTUIve's art image nodes |
Still open (gap 1). image_bytes_of takes encoded bytes, decodes each change on a worker thread and shows "loading…" until it is done. That's fine for a cover picture, but at 35 frames a second it means a PNG encode and decode per frame plus a flicker. Each new picture gets a new Kitty id, and they are freed only when the app ends. Sixel always goes through the adaptive quantizer at CellPixels::DEFAULT rather than the measured cell size. rs-rich-doom keeps its own PlacementSource. |
| A public indexed-plus-palette Sixel encoder in rs-rich-art | Still open (gap 2). sixel::encode_indexed is pub(crate) and maps indices through the fixed ANSI palette. rs-rich-doom keeps graphics::sixel_indexed. |
| Style-table compaction over a long run | Fixed and measured. tests/app.rs runs the real engine under the headless driver at 80x24 in Braille (averaged colours, the most styles) for 120 s of game time, walking and turning. Styles kept: 1,337 after the first second, peak 2,766, peak in the last minute 2,605, 2,056 at the end. That is bounded by twice the screen's cells (3,840), as the compaction rule says. Resident memory went from 24.9 MB to 25.3 MB. The test asserts the bound. |
Gaps¶
1. A live frame image node (intuiTUIve, art plugin)¶
Status: closed in 0.0.23 (wave 1, ad84120f). intuiTUIve art's
picture_of(picture: impl Fn() -> Option<Picture>, view: ImageView) draws
each new frame at once, with no worker and no "loading". rs-rich-art's
Picture::from_rgba and Picture::next_rgba make a frame from raw RGBA,
and a frame of the same picture keeps its id: Kitty gets it again under
that id (raw pixels, f=32), and image_graphics frees it once. The
pictures are fitted to the cells as the cells were drawn. Not taken
from the proposal: ImageView::aspect (a picture shown at a ratio other
than its pixels'; gap 12's PixelBuf::draw_fitted does it for pixel
cells, and an app can resample a frame before handing it over),
ImageView::cell_px and keep_last_row. The maintainer dropped those
three on 2026-10-26: they are not coming. Tests: image.rs's
live_frames_keep_one_kitty_id_and_are_freed_once,
sixel_frames_are_drawn_again_and_counted and
an_overlay_picture_partly_covered_is_not_drawn_over_what_covers_it, and
picture_of's doc-test (half blocks, no worker).
- What Doom needed: show a new in-memory RGBA frame 35 times a second as real pixels (Kitty, Sixel, iTerm2), fitted to the node and kept to an aspect ratio, with a cell fallback and cleanup.
- Workaround:
src/graphics.rs(Pictures, aPlacementSource, andPicture, aGraphic) andapp::ImageView, a widget that draws Kitty placeholders and hands the frame and its screen rectangle (DrawCx::rect()) toPictures. It also keeps one Kitty image id, reserves Sixel's last row, centres a capped Sixel footprint, and sizes images from the measured cell size. - Proposed API (rs-rich-intuituive,
art):image_frames(frame: impl Fn() -> Option<Arc<RgbaFrame>> + 'static, view: ImageView) -> NodewhereRgbaFrame { width, height, pixels: Vec<[u8; 4]>, palette_exact: bool }(or aPixelBuf). It draws synchronously, with no worker and no "loading". One protocol image id per node, reused for every frame and deleted on unmount.ImageView::cell_px(default: the measuredGraphicsEnvironmentsize).ImageView::aspect(Option<f64>)for a picture shown at a ratio other than its pixels' (Doom's 320x200 is shown at 4:3).ImageView::keep_last_rowfor Sixel. Withpalette_exactand 256 colours or fewer, Sixel uses gap 2's encoder. - Why generic: an emulator's screen (NES, Game Boy, CHIP-8), a video or webcam preview, a live plot rendered to pixels, a remote desktop.
- Acceptance tests: under the driver, a node fed 100 frames shows each
frame's placement with one Kitty id (the escape stream holds one
a=d, on unmount). Resizing the node moves and refits the placement. Removing the node repaints its cells. The cell fallback (Pixels::Cells) draws half blocks with no worker. A Sixel image capped by width is centred. A frame at 4:3 in 80x24 cells with 1:2 cells covers 64x24.
2. A public Sixel encoder for indices and a palette (rs-rich-art)¶
Status: closed in 0.0.23. rich_art::sixel::encode_palette(indices,
palette, width, height) -> Option<String> and
rich_art::sixel::index_exact(rgba) -> Option<(indices, palette)>, behind
sixel. The crate's own fixed-ANSI encoder is now encode_palette over
the 256-colour palette, byte for byte as before. Its bytes match
rs-rich-doom's encoder but for P1, which the raster attributes override
(9 here, as in every encoder in the crate; 0 there). Tests:
a_palette_of_ones_own_encodes_exactly_and_validates (rs-rich-doom's
bytes for a 3x7 image, icy_sixel decodes the size and #123456 within
Sixel's precision, None left unset, a long run split, past the palette,
short or past MAX_PIXELS refused) and
exact_indexing_keeps_up_to_256_colours. Since 023-avmedia,
SixelArt::encode and encode_cells (so Picture's Sixel path too) go
through sixel::encode_rgba, which uses encode_palette for a raster of
no more colours than max_colors (256 by default) and the adaptive
encoder past that. Tests: a_picture_of_few_colours_keeps_its_exact_palette,
transparency_is_kept_by_the_exact_palette and
more_colours_than_the_palette_fall_back_to_the_adaptive_encoder.
- What Doom needed: encode a frame that already has at most 256 colours with exactly those colours, quickly: no quantizing, no resampling.
- Workaround:
graphics::sixel_indexed(about 90 lines), with a fallback toSixelArt::encode_cells(adaptive) past 256 colours. - Proposed API (rs-rich-art,
sixel):pub fn encode_palette(indices: &[Option<u8>], palette: &[[u8; 3]], width: usize, height: usize) -> Option<String>(Noneis transparent). It is the existingencode_indexedwith the palette passed in, keeping its repeat-count splitting andMAX_PIXELScheck. Pluspub fn index_exact(rgba: &[[u8; 4]]) -> Option<(Vec<Option<u8>>, Vec<[u8; 3]>)>for images of 256 colours or fewer. - Why generic: pixel art and sprite editors, emulators with fixed palettes, indexed PNG and GIF frames, terminal plots in a few colours.
- Acceptance tests: a 3x7 two-colour image encodes to the same bytes
rs-rich-doom's encoder makes. Decoding with
icy_sixelgives the size and every palette colour back within Sixel's 1% precision, including non-ANSI colours (#123456).Nonepixels are left unset (P2=1). More thanMAX_PIXELSreturnsNone. A run longer than 65,535 is split. A palette index past the palette's end returnsNone.
3. Kitty raw-pixel transmission and placement ids (rs-rich-art)¶
Status: closed in 0.0.23. Wave 1 added kitty::transmit_rgba
(f=32, chunked), which Picture uses for raw frames. This gap's work
added the rest: kitty::transmit_rgb (f=24) and
kitty::transmit_raw(&RawImage, &Transmit), where Transmit carries the
image id, a placement id (p), the cells and zlib compression (o=z,
through flate2, which png already brings). Tests: rs-rich-art's
raw_pixels_are_chunked_and_round_trip_compressed_or_not (keys, chunks
of at most 4096 bytes, payloads read back, compressed or not), and
rs-rich-record's tests/kitty_raw.rs, where the emulator decodes RGB and
RGBA, compressed or not, to the pixels sent.
- What Doom needed: send a new frame each tic without a PNG encode.
- Workaround: PNG through
graphics::encode_pngeach frame, about 180 kB a frame and the main cost of Kitty mode. - Proposed API (rs-rich-art,
kitty):transmit_rgba(id, placement: Option<u32>, rgba: &[u8], width, height, cols, rows, compress: bool) -> String(f=32,o=zwhen compressing, chunked), andtransmit_rgb(f=24). - Why generic: video, emulators and live plots in Kitty, and any image that changes often.
- Acceptance tests: escapes parse as the Kitty protocol (
f=32,s,v,o=z,m=1chunks of at most 4096 bytes). rs-rich-record's Kitty emulator decodes them to the same pixels. Compressing is optional and round-trips.
4. Made-up releases with a first-press hold and a repeat hold (intuiTUIve)¶
Status: closed in 0.0.22 (e7651f81): App::synthesize_releases_with(Holds::new(first, repeat)).
Tests: tests/input_timing.rs, with Holds::new(400 ms, 120 ms).
- What Doom needed: on terminals without release reports, hold a key across the keyboard's repeat delay after the first press, but let it go soon after the last repeat. Before, rs-rich-doom used 400 ms after the first press and 120 ms after each repeat.
- Workaround: none that keeps using held keys.
synthesize_releasestakes one hold, so rs-rich-doom passes--hold-ms(400). After a key is let go on a legacy terminal, the player keeps moving for up to 400 ms (before: 120 ms). - Proposed API (rs-rich-intuituive):
App::synthesize_releases_with(Holds { first: Duration, repeat: Duration }), wheresynthesize_releases(Some(d))staysHolds { first: d, repeat: d }. Optionally learnrepeatfrom the measured gap between repeats. - Why generic: any game or held-key control on a legacy terminal: a block puzzle's soft drop, a raycaster's movement, a scrubber held on a timeline.
- Acceptance tests: under the driver with
Releases::NoneandHolds { first: 400 ms, repeat: 120 ms }: a press at 0 is held at 399 ms and released at 400 ms. Repeats at 450, 480 and 510 ms keep it held, and it is released at 630 ms. A key whose release is reported never times out.
5. A worker on the app's clock, in lockstep (intuiTUIve)¶
Status: closed in 0.0.23. stepped_worker(period, max_catch_up, |steps:
Steps<T>| ...) -> Signal<Option<T>>, and stepped_worker_with(Lockstep,
work) -> (Signal<Option<T>>, Timer) for the timeout (250 ms by default)
and the timer. Steps::wait() -> Option<u64> blocks for ticks and returns
None once the app has gone; Steps::publish(T) hands output over. It is
a fixed_step underneath. Tests: tests/lockstep.rs (a second at 35 a
second is 35 ticks and tick 35's output; a 300 ms stall hands over
max_catch_up; a worker that ends makes each step return at once and the
app quits; a worker asleep for a second times out and catches up), and
stepped_worker's doc-test.
- What Doom needed: run a simulation that must live on its own thread (the C engine is global and blocking) so that it advances exactly as the app's clock does: real time in a terminal, exact time under the driver. The app takes its output once the simulation has caught up.
- Workaround:
engine::Paceandengine::Pacer. Afixed_stepsends the tics given so far. The engine's sleep reports "idle at tic n" and blocks for more. The step waits (at most 250 ms) for the report, then takes the newest frame from a shared slot. Exit drops the channel, so the step learns at once. - Proposed API (rs-rich-intuituive):
stepped_worker(period, max_catch_up, |steps: Steps| { /* worker thread */ })returns aSignal<Option<T>>of the worker's output.Steps::wait()blocks until the app gives ticks and returns how many.Steps::publish(T)hands output over. The app's step waits for the worker to finish the ticks, with a timeout. Generic overT; no Doom. - Why generic: an emulator core, a physics or cellular-automaton simulation, a WASM or script VM, a deterministic replay. Anything whose stepping blocks and must be tested exactly under the driver.
- Acceptance tests: under the driver,
advance(1 s)at a 1/35 s period gives the worker exactly 35 ticks, and the signal holds tick 35's output whenadvancereturns. A stall of 300 ms hands over at mostmax_catch_upticks. A worker that ends makes the next step return at once (no timeout) and the app can quit. A slow worker (sleeps 1 s) lets the step time out and catch up later.
6. Frame timings in FrameStats (intuiTUIve)¶
Status: closed in 0.0.23 (wave 1, 7d285128). FrameStats has
draw, paint, write, painted, and the totals total_bytes,
total_draw, total_paint, total_write, frames and skipped;
App::timedemo runs a script at full speed and reports each frame.
Tests: tests/timedemo.rs (a widget that sleeps 5 ms shows draw of
5 ms or more; total_bytes is the sum of each render's bytes).
- What Doom needed:
--fpsand--statsused to show the time a draw took and the bytes over a run. - Workaround: dropped.
--statsreports frames, tics, the newest engine frame, frames held back and the last frame's bytes. Per-node timings exist only through the inspector (Driver::inspector_timings). - Proposed API (rs-rich-intuituive):
FrameStatsgainsdraw: Duration(widgets' draw),paint: Duration(diff and escape writing), and cumulativetotal_bytes: u64andtotal_draw: Duration. - Why generic: any app's performance overlay, benchmarks under the driver, regression gates like the ratatui comparison.
- Acceptance tests: a widget that sleeps 5 ms in
drawshowsdraw≥ 5 ms.total_bytesis the sum of each render's output length. Durations are zero under a fake clock if the driver says so.
7. A voice mixer for sound effects (intuiTUIve, pcm/audio)¶
Status: closed in 0.0.23. audio::Mixer (with pcm):
play(channel, clip: Arc<Clip>, gain, pan) -> Voice (a sound on a
channel stops the one there), Voice::set(gain, pan), stop,
is_playing, Mixer::source(Box<dyn Source>) for a stream, and
stop_all. Clips and streams at any rate are resampled linearly. The pan
is a balance: 0 plays the sound at its gain on both sides, -1 the left
only; Doom's own pan law stays in Doom. Mixer::new(&Output) (with
audio) mixes 256-frame blocks into the output on its own thread;
Mixer::detached(rate, channels) and Mixer::mix(&mut [f32]) mix by
hand, from a renderer (gap 8) or a test. Tests: tests/audio_mixer.rs
(two clips sum sample for sample; 11,025 Hz into 44,100 Hz lasts four
times as many frames; pan -1 is left only; stop silences the next
block; is_playing ends within a block of the clip's end; a stream under
the voices; the thread into a null output, read through a tap), and
Mixer's doc-test.
- What Doom needed: play many short sounds at once on numbered channels, each at its own sample rate, gain and pan, stopped or changed while playing, mixed into the output.
- Workaround:
sound::Mixerandsound::Voice, plus a mixer thread that writes 256-frame blocks toaudio::Output. The DMX lump format and Doom's pan law (gains) are Doom's own and stay. - Proposed API (rs-rich-intuituive,
audio):audio::Mixer::new(output: &Output) -> Mixerwithplay(channel: usize, clip: Arc<Clip>, gain: f32, pan: f32) -> Voice,Voice::set(gain, pan),stop,is_playing, andMixer::source(Box<dyn Source>)to add a stream (music). Clips at any rate are resampled linearly; it mixes on its own thread into the output. - Why generic: sound effects for any game (the block-puzzle and raycaster clones), UI sounds, a sampler.
- Acceptance tests: with a test sink (gap 9), two clips started together
sum sample for sample. A clip at 11,025 Hz into 44,100 Hz lasts four times
as many frames. Pan −1 is left only.
stopsilences from the next block.is_playingturns false within one block of the clip's end.
8. A render callback output for synthesizers (intuiTUIve, audio)¶
Status: closed in 0.0.23. audio::output_with(options, render: impl
FnMut(&mut [f32], u16) + Send + 'static) -> Result<Output, AudioError>:
the device's callback (or the null output's clock) calls render for each
block. Volume, pause, tap and failed work as for output; write is
refused. A panicking renderer stops the output and failed() says
"the renderer panicked"; the app goes on (Rust's panic hook still prints
the message to stderr). A Mixer::detached can be mixed from it. Tests:
tests/audio_render.rs (a 440 Hz sine makes 440 rising crossings in a
second; volume 0 is silence while the renderer goes on; a panicking
renderer is reported), and output_with's doc-test.
- What Doom needed: the OPL synthesizer produces samples on demand, and its song timing runs off the samples it makes.
- Workaround: the mixer thread pulls music into each block and writes it to the output's queue. The queue adds 60 ms of latency to effects, the price of a push model.
- Proposed API (rs-rich-intuituive,
audio):audio::output_with(options, render: impl FnMut(&mut [f32], channels) + Send + 'static) -> Output: the device's callback callsrenderdirectly. Volume, mute andfailedwork as foroutput. - Why generic: synthesizers, trackers and chiptune players, tone generators, anything that makes sound rather than playing a file.
- Acceptance tests: with a test sink, a 440 Hz sine renderer produces
440 zero-crossing pairs a second. Volume 0 gives silence without stopping
the renderer. A panicking renderer reports
failed()rather than taking down the app.
9. A paced test sink and capture for audio output (intuiTUIve, audio)¶
Status: closed in 0.0.23. OutputOptions::null() opens an Output
with no device (44,100 Hz in two channels unless set) that takes samples
at its rate on the real clock, never more than its queue ahead.
Output::tap(keep: Duration) -> Tap records what any output's device
takes, after the volume: the newest keep of it (Tap::samples), the
frames in all (Tap::frames), and Tap::to_wav(path) (32-bit float)
with wav. The null output runs on the real clock only, not the app's:
under the driver a test waits real time for it. The maintainer kept it
that way on 2026-10-26. Tests:
tests/audio_null.rs (44,100 frames a second within 2%, a writer held to
the queue; volume 0 taps only zeros; a 1 s tap keeps 1 s; hound reads the
WAV back the same), and OutputOptions::null's doc-test.
- What Doom needed: prove that sound reaches the device (sample counts, silence on mute, volume) on a machine with no sound card, and record the game's sound for a video.
- Workaround: a temporary
~/.asoundrcwith ALSA'sfileplugin overnull, writing into a FIFO that a reader drains at the sample rate (tests/audio_device.rs;docs/media/play/paced_tap.pyfor the video). Unpaced, the null device takes samples as fast as the CPU allows. An earlier version of the test wrote gigabytes to/tmpthat way. - Proposed API (rs-rich-intuituive,
audio):OutputOptions::null()opens a device-less output that consumes at its rate on a clock (real or the app's).Output::tap() -> Taprecords what the device takes, bounded (Tap::samples(),Tap::to_wav(path)withwav). - Why generic: every audio feature's tests; recording an app's sound for a demo or a bug report; CI machines without sound cards.
- Acceptance tests: a null output at 44,100 Hz takes about 44,100 frames
a second (±2%) and never more than its queue ahead. With volume 0 the tap
holds only zeros. A tap capped at 1 s keeps 1 s.
to_wavwrites a file hound reads back with the same samples.
10. ALSA's messages kept off the screen for the stream's life (intuiTUIve, audio)¶
Status: closed in 0.0.23, another way. Each output's and capture's
stream callbacks put intuiTUIve's keeping handler on the thread that runs
the stream (cpal's), the first time they run, for that thread's life,
which is the stream's. It stays thread-local (snd_lib_error_set_local),
so nothing else in the process that uses ALSA is changed, and no C is
needed: the process-wide handler (snd_lib_error_set_handler) is a
variadic C function, which stable Rust cannot define, and alsa-lib gives
no way to read the handler before it to put back. audio::messages()
returns the lines as before. Test: tests/audio_alsa_quiet.rs, a child
process (HOME with an .asoundrc whose default is ALSA's file plugin
onto a path it cannot write) writes nothing to stderr, and its
messages() holds pcm_file.c's line; without the change the test fails
with the lines on stderr. messages' doc-test.
- What Doom needed: ALSA's complaints (an underrun) must never land on the game's screen.
- Workaround:
csrc/sound.c'srs_doom_quiet_alsainstalls a process-wide no-op handler (snd_lib_error_set_handler). intuiTUIve's handler is thread-local and in force only while devices are listed and opened. - Proposed API (rs-rich-intuituive,
audio): while any output or capture is open, install the keeping handler process-wide and restore the previous one when the last closes.audio::messages()returns them as today. - Why generic: any TUI that plays or records sound on Linux (the music player, the scope).
- Acceptance tests: with an output on a device that reports an error
after opening (ALSA's
fileplugin onto an unwritable path), nothing is written to stderr andmessages()holds the line.
11. A palette-keeping two-colour rule for Braille and quadrants (intuiTUIve, pixels)¶
Status: closed in 0.0.23. Node::pixel_colors(PixelColors::Average |
PixelColors::Nearest) for pixels and pixels_fit nodes, Average the
default. With Nearest, each side's colour is the one of its pixels
nearest the side's average. Tests: pixels.rs's
nearest_colours_keep_to_the_cells_own (a 2x4 cell of red, dark red and
black, in Braille and quadrants), averaged_colours_are_the_default and
cells_written_directly_match_the_segments_they_replaced (unchanged), and
pixel_colors's doc-test.
- What Doom needed: Braille cells in the frame's own colours. Before, rs-rich-doom chose each side's colour as the member pixel nearest that side's mean, never a blend.
- Workaround: none. On pixel cells, Braille now uses the core's averaged colours: more distinct styles (see the compaction measurement) and colours the palette doesn't have.
- Proposed API (rs-rich-intuituive):
Node::pixel_colors(PixelColors::Average | PixelColors::Nearest)forpixels/pixels_fitinQuadrantsandBraille, withAveragethe default. - Why generic: pixel art and palette-limited graphics (emulators, 16- or 256-colour art) keep their palette. It is also cheaper on the style table.
- Acceptance tests: a 2x4 cell of red, dark red and black in Braille with
Nearestdraws only colours present in the cell. WithAverage, output is as today, byte for byte.
12. An aspect-fitted pixel buffer (intuiTUIve, pixels)¶
Status: closed in 0.0.23. PixelBuf::draw_fitted(src, width, height,
&Fitted) -> Rect, where Fitted::new(mode) takes .aspect(f64),
.cell_aspect(f64) (default 0.5), .fit(PixelFit::Contain | Cover |
Stretch) and .filter(Filter::Nearest | Average); it returns the cells
used. One change from the proposal: the fit is intuiTUIve's own
PixelFit, not rs-rich-art's ImageFit, because pixels is core and
rs-rich-art is behind the art plugin; and Fitted names the cell mode,
which says how many pixels a cell holds. Tests:
a_picture_is_fitted_at_its_ratio_in_whole_cells (320x200 at 4:3 into
80x24 cells of 1:2 is 64x24 at column 8; aspect unset fills the buffer;
Nearest keeps the source's colours; margins stay CLEAR; Braille; cover)
and the_average_filter_blends_what_each_pixel_covers, and
draw_fitted's doc-test.
- What Doom needed: a 320x200 frame shown at 4:3 in whatever cells the node has, from the cell's real aspect ratio, centred, the margins transparent.
- Workaround:
render::Fitandrender::draw_intofit in whole cells and resample nearest-neighbour into thepixels_fitbuffer. - Proposed API (rs-rich-intuituive):
PixelBuf::draw_fitted(src: &[Rgba], width, height, fit: ImageFit, aspect: Option<f64>, cell_aspect: f64, filter: Filter)(Filter::Nearest | Average), returning the cell rectangle used.ImageFitandImageAnchorare rs-rich-art's. - Why generic: emulators (8:7 or 4:3 pixels), video, any source picture in a pixel node.
- Acceptance tests: 320x200 at 4:3 into 80x24 cells of 1:2 is 64x24 at
column 8. With
aspect: Noneit fills the buffer.Nearestkeeps the source's colours exactly; margins stayCLEAR.
13. Sound in recordings (rs-rich-record)¶
Status: built (branch 023-avmedia), with option (a) below, the
maintainer's choice. Set Audio on in a tape, or rich record
--audio, off by default. The recorder makes a directory and names it in
the program's environment as RICH_RECORD_AUDIO; a program writes what
it plays there, one timed stream a file (a header with the rate, the
channels and the wall-clock time of the first sample, then f32
samples; rich_record::audio documents it). intuiTUIve's output() and
output_with() do it by themselves (audio::RECORD_AUDIO_ENV): under
the recorder they play on the device as usual and tee every block it
plays into the stream, timed by the device's callbacks (latency counted,
an underrun kept as silence); with no device they fall back to a null
output paced on the real clock, named "rich record", whose sound goes to
the stream alone (the maintainer's call: on the device as well, not
instead of it). The recorder puts the streams on the
recording's clock, cuts what played while hidden, and mixes them: an MP4
gets an AAC track placed with -itsoffset, and a GIF, APNG, page or cast
a WAV beside it (rich_record::audio::Audio::wav). The acceptance test
(crates/rich-record/tests/audio.rs) records a program playing a 1 s
440 Hz tone and checks that the WAV's tone, and the MP4's decoded, start
within one frame of the frame that announces it and last 1 s.
The options as they were laid out:
- How the program's sound reaches the recorder. rs-rich-record does
not depend on intuiTUIve, so either (a) a contract through the
environment: the recorder sets, say,
RICH_RECORD_AUDIO=<path>and intuiTUIve'soutput()opens a null output tapped to that file, with its start time on the recording's clock; (b) an ALSA device the recorder provides (HOMEwith an.asoundrconto a paced FIFO, as rs-rich-doom's video script does), which covers any program but is Linux-only and needs the recorder to pace the reads; or (c) a PulseAudio null sink and its monitor, which needs a running server. - The tape syntax and the format.
Set Audio onin a tape, orrich record --audio, and what happens to outputs other than MP4 (GIF, APNG and the HTML player have no sound; a WAV beside them?). -
Sync. The recorder's frame timestamps are its own clock's; the sound's start must be put on that clock (
-itsoffsetfor ffmpeg), and ffmpeg must have an AAC encoder (the built-inaacis enough). -
What Doom needed: a video of the game with its sound.
- Workaround:
docs/media/play/record.sh: xterm on Xvfb captured by ffmpeg, the sound through gap 9's ALSA-FIFO tap, muxed with ffmpeg.rich recordmakes GIFs, APNGs and videos without sound. - Proposed API (rs-rich-record):
Set Audio onin a tape, orrich record --audio. The recorded program's sound goes through an environment the recorder sets (an ALSA or Pulse device it provides, or intuiTUIve'sOutput::tapwhen the program asks for it). It is muxed as AAC into MP4 output, in sync with the frames' timestamps. - Why generic: demos of any app with sound (the music player, games), bug reports.
- Acceptance tests: a tape running a program that plays a 1 s 440 Hz tone from its second frame produces an MP4 with an AAC track whose tone starts within one frame of that frame and lasts 1 s ±1 frame.
Not gaps¶
- Overlays in
layers: a text layer fills its rectangle (and covered the picture) unless placed..place(Align::Start, Align::Start)is the documented way, and works. fixed_stepitself, held keys,rate,StatsHandle,put_rgb, compaction: used as they are, no changes needed.