omni-git

A content-addressed store with a CLI: ingest files, folders, tarballs, git repos, and disk images as sha256-addressed blobs+manifests, then resolve/pull them back byte-identically — with no git commands leaking through. Pure Python standard library (no third-party pip dependencies). The store self-verifies: every object is re-hashed on read, so tampering is caught.

Disk-image ingest additionally walks the guest filesystem and content-addresses/dedups each guest file — powered by diskforge, which is bundled in this package (the diskforge/ package), so installing omni-git installs diskforge with it. See Image ingest.

Install

Prereqs: python3 >= 3.9 and SSH access to the repo host.

Preferred — pip install (gives the omni-git and omni console scripts, with diskforge bundled):

pip install git+ssh://[email protected]/dudezilla/omni-git.git

or from a local clone: git clone … && pip install ./omni-git.

Or run from a clone without installing (a launcher on PATH as ~/bin/omni-git):

#!/bin/bash
export PYTHONPATH="$HOME/code/installed/omni-git${PYTHONPATH:+:$PYTHONPATH}"
export OMNI_ROOT="${OMNI_ROOT:-$HOME/code/installed/omni-store}"
exec python3 -m omni.cli "$@"

chmod +x ~/bin/omni-git. (diskforge is bundled in the clone, so no extra path is needed for it.)

Verify diskforge is bundled and active:

python3 -c "import diskforge.wire; print('active:', diskforge.wire.__file__)"

Full oracle: omni-git selftest → 254/254 (its boot pins need the diskforge Docker Sandbox; the tool itself runs without it).

The tool is stateless code; the actual data lives in the store at $OMNI_ROOT — whose tool default is ./.omni in the current directory (so it follows your cwd unless you pin OMNI_ROOT, e.g. in the launcher shim). Store(root).init() creates the store if the path is empty and opens it if it already exists, so one OMNI_ROOT setting covers both "new store" and "existing store". A fresh location starts empty — copying an existing machine's stored artifacts is a separate data copy (or share one store live via OMNI_API), not a git pull.

Configuration (environment)

| Var | Meaning | |---|---| | OMNI_ROOT | store location. Tool default: ./.omni in the current working directory. Set it to pin a fixed store; a launcher shim usually does this (this machine's shim pins ~/code/installed/omni-store). | | OMNI_API | point every command at a remote omni-git serve URL (REST) instead of the local store | | OMNI_TOKEN | bearer token for a remote OMNI_API | | OMNI_INDEX_DIR | override the index directory | | OMNI_CONSTRUCTION_LOG | path for the construction log | | OMNI_CHUNK_BYTES | streaming chunk size = the working-set RAM cap for large-file ingest/verify (default 1 MiB, floored at 4 KiB). A multi-GB image streams through a buffer this size instead of being read whole into RAM. |

Large files

Both ingest and pull are streamed: omni-git add hugefile.qcow2 reads the file in OMNI_CHUNK_BYTES pieces (hashing + writing chunk-by-chunk), and omni-git pull materializes a blob to its destination the same way (via a temp file + atomic rename, so the destination never holds unverified bytes). Peak memory is ~one chunk regardless of file size — a 200 MiB file round-trips in ~2 MiB of RAM at the 1 MiB default, over REST too. Set the cap explicitly to bound it:

OMNI_CHUNK_BYTES=8388608 omni-git add hugefile.qcow2   # ~8 MiB working set

The store is a flat content-addressed store (one file per blob), not real git, so there is no packfile/delta/history bloat on large binaries — a big blob is just one big file, deduped by sha256. (Note: streaming bounds file-bytes memory; an artifact with millions of tiny entries still uses memory proportional to the entry count for its manifest — a separate, much smaller dimension.)

Usage

omni-git add  <path> [--kind K]   ingest file | folder | tar.gz | git | img | qcow2
omni-git pull <addr> <dest>       materialize it back, as the right kind
omni-git boot <addr>              pull an image and BOOT it in the hypervisor
omni-git ls                       every address in the store
omni-git find <query>             resolve a name / sha prefix / member path
omni-git cat  <query>             the bytes behind a query, to stdout
omni-git probe                    are pi-img / bakery usable here? (recomputed)
omni-git coverage                 findability coverage (recomputed, never a stored number)
omni-git fsck                     sweep every manifest+blob for rot/gaps (report-only)
omni-git git-guard [status|install]   make `git ... omni:...` fail loudly, not read as an ssh host
omni-git export-git <git-dir> [addr…]  write the store (or listed artifacts + closure) into a real git
omni-git import-git <git-dir> [--ref R]  rebuild/merge $OMNI_ROOT from such a git (every object re-verified)
omni-git serve [host:port]        run the REST API over the store
omni-git selftest                 run the oracle (exit 0 iff every check passes)

Git round-trip (export-git / import-git)

omni-git export-git <git-dir> writes the whole store (objects + manifests verbatim, addressed by omni-sha256 in the tree path, plus OMNI-STORE.json) into a real git repo as a commit on the omni-store branch — snapshots over time = history. omni-git import-git <git-dir> rebuilds or merges $OMNI_ROOT from it, re-verifying every object on the way in (a tampered or mislabelled object is refused, never stored). The pair is a byte-faithful inverse — import-git(export-git(store)) == store — is idempotent, and streams (a multi-GB blob round-trips within OMNI_CHUNK_BYTES). Both are local-store operations (unset OMNI_API). Design + guarantees: docs/export-git-spec.md. (Installed via pip, the same commands are the omni-git / omni console scripts.)

Image ingest (diskforge, bundled)

omni-git add <img|qcow2> is additive: the whole image is still stored byte-for-byte as one content-addressed blob (pull stays byte-identical), and in addition the guest filesystem is walked in userspace (no mount/loopback/qemu-nbd/kernel) via import diskforge.wire, so each guest file is content-addressed and identical files/hardlink sets collapse to one blob (RFC-0009: sha256 is authoritative identity). Ingest sets members_via=diskforge; the import is lazy under try/except ImportError, so a partial install degrades to stock whole-image behavior.

diskforge is bundled at the repo root (diskforge/, importable as diskforge) — a copy of [email protected]:testmonkeyalpha-group/jazz-forge/diskforge.git; see diskforge/_VENDOR.txt for the pinned source commit and drift-detection note.

Source of record

Two remotes kept consistent: GitHub [email protected]:dudezilla/omni-git.git (deployed / pip home) and GitLab [email protected]:testmonkeyalpha/omni-git-entry-claude.git (tournament lineage), branch main. Tracked by congruency ticket #493 (diskforge: #495).