Per-repo documentation — each repo's docs/ and README, ingested and associated with the repo. Rendered through the <<<RepoDocs>>> tag.
← omni-git docs · docs/recursive-ingest-spec.md
Status: Accepted · open questions resolved (§9) · index nested containers all the way down.
Today omni indexes one level deep: add outer.tar.gz stores the tar whole and indexes each member as one opaque content-addressed blob — a member that is itself a .qcow2 (or another tar) is a single blob whose inner files are never enumerated. So a file buried inside tar → qcow2 → tar is invisible to per-file dedup and to omni find.
Recursive ingest makes the per-file content-addressing reach all the way down: on ingest, a member or guest-file that is itself a recognized container is opened and indexed too, deduping its contents against the rest of the store. Opt-in and depth-bounded; the default one-level behavior is unchanged.
The whole container blob is still stored at every level, so pull stays byte-identical (see §6). Recursion adds structure: when a member/guest-file sniffs as a container, omni ingests it as its own artifact (records its manifest) and the parent's member record references that sub-manifest.
A nested-container member record carries both: - blob — the container's whole bytes (as today; keeps the byte-identical pull), and - nested — the sha of the sub-container's manifest (its own recorded artifact).
nested is added to BLOB_POINTER_KEYS, so blobs_referenced_by, fsck, coverage, and export-git's closure all follow the chain for free — a nested artifact is just more manifests+blobs in the same content-addressed store. Dedup falls out: identical inner files (anywhere, any depth) collapse to one blob; an identical nested container collapses to one blob and one manifest.
Recurse only into recognized container kinds (tar family, img/qcow2). A member is sniffed from its bytes (imgfmt.sniff_image_kind, tarfile.is_tarfile / detect_kind), never its filename — matching omni's existing "the suffix opens the question; the bytes answer it" discipline (store.py detect_kind). A member whose bytes are not a known container is a leaf (one opaque blob, as today).
- Depth: --max-depth N (default per §9/Q1). At the cap, a deeper container is stored opaque (one blob) with a recorded depth_capped: true note, surfaced by ls/coverage/fsck — never silently (a truncated index that reports full coverage is the rounds-8/80 disease). - Expansion: a total unpacked-bytes budget --max-expand BYTES guards against decompression bombs and runaway fan-out; past it, the ingest is refused in omni's voice with nothing orphaned in objects/ (the round-80 mid-stream-death discipline: buffer/commit so a refusal leaves the store byte-identical). - Cycles: content-addressing is self-protecting (a container cannot contain itself by sha); the expansion budget covers pathological non-cyclic blow-up. - Corrupt nested container degrades, never fails the outer add: if an inner container will not parse, its whole blob is still stored and the member is recorded nested_unindexed: <reason> (still-pullable, not pull-doomed) — exactly the additive/degrade pattern the diskforge wiring uses for MemberIndexError/ImportError. The outer artifact ingests green.
_add_tar_gz previously buffered every member payload in memory. It now streams each member to a temp file (chunked, ~OMNI_CHUNK_BYTES) during the parse and folds them in via put_file only after the archive reads to its end — so the "no orphans on a mid-stream death" property is preserved AND peak memory is ~one chunk per member, not the unpacked archive (measured: a 200 MiB tar member ingests at ~4 MiB peak). Recursion reads those same temp files (no re-buffering); nested images reuse diskforge's streaming guest walk + get_blob_to_path. Net: recursive ingest of a large nested image runs within the OMNI_CHUNK_BYTES cap.
Recursion is index-only. Because the whole container blob is stored at each level, omni pull <outer> reproduces the outer artifact byte-for-byte from that blob — it does NOT reassemble from the nested index (no risk of a padding/ordering byte diverging). The nested index powers omni find <deep/path> and dedup; and because each nested container is its own recorded artifact, omni pull <nested-addr> can materialize just that inner container.
omni add <path> [--kind K] [--recursive] [--max-depth N] [--max-expand BYTES]
Default is one-level (unchanged). --recursive walks into nested containers (tar family, img/qcow2),
sniffed by bytes, to --max-depth (bounded by --max-expand total unpacked bytes). find resolves deep
paths through the chain.
1. Recursion fidelity. tar → qcow2 → tar: with --recursive, every leaf file is content-addressed and omni find <deep/path> resolves; without it, the nested container is one opaque blob and the deep path is an absence. Neg control: a recursion that stores nested members but drops the parent's whole blob — outer pull is no longer byte-identical. 2. Cross-nest dedup. The same file inside two differently-nested containers stores one blob. Neg control: one-level ingest stores it twice (once inside each container blob). 3. Byte-identical outer pull. pull <outer> == original bytes with and without --recursive. Neg control: a design that reassembles the outer from the nested index diverges by a byte. 4. Depth cap is loud. --max-depth N stops at N and records depth_capped, surfaced by coverage. Neg control: silent truncation reports full coverage. 5. Expansion budget. A decompression bomb past --max-expand is refused in omni's voice, nothing orphaned. Neg control: unbounded recursion exhausts disk / leaves orphans. 6. Corrupt nest degrades. A rotted inner qcow2 → outer artifact still ingests (whole blob), the nest recorded still-pullable, fsck flags it non-doomed. Neg control: a corrupt inner container fails the whole outer add. 7. Bounded memory. Recursive ingest of a large nested image runs at ~OMNI_CHUNK_BYTES. Neg control: the current buffer-all-payloads tar path OOMs under a tight cap.
- Q1 — --max-depth default 8; --max-expand default 0 (off). Depth 8 covers any realistic nest; the expansion budget is opt-in (0 = unbounded) because recursion itself is opt-in — set it for untrusted input. Both take env defaults OMNI_MAX_DEPTH / OMNI_MAX_EXPAND (omni is env/flag configured — no config file), so a shim can pin them like OMNI_CHUNK_BYTES. - Q2 — nested linkage = nested → sub-manifest reference (chosen). The member record gains nested: <sub-manifest-sha>; the sub-container is a fully recorded artifact. This reuses manifests, dedup, find (which sweeps every recorded artifact, so deep files are found for free), fsck, and export-git's closure. Inline nested index is rejected (duplicates structure, no dedup of the nest). - Q3 — opt-in in v1. --recursive off by default; one-level behavior is unchanged. Revisit default-on only after the streaming member fix and real-world use. - Q4 — no separate fan-out cap; --max-expand (bytes) is sufficient. A huge member count of tiny files is bounded by total bytes too; a second knob adds surface for no new protection. - Q5 — export-git carries nested manifests via the closure once nested is in BLOB_POINTER_KEYS (whole-store export already carries all manifests; selective export follows nested transitively). A round-trip pin over a recursively-ingested store is part of acceptance (§8).
detect_kind / imgfmt.sniff_image_kind (sniff by bytes), diskforge.wire.index_image_members (guest walk), put_file / get_blob_to_path (streamed, #497), put_manifest / _record_manifest, blobs_referenced_by + BLOB_POINTER_KEYS (add nested), fsck as the integrity gate, and the additive/degrade pattern from the diskforge wiring (MemberIndexError → store opaque, record a note).
Not reassembling the outer artifact from the nested index (pull uses the whole blob). Not recursing into arbitrary formats — only recognized container kinds. Not auto-recursion without the flag in v1.