Per-repo documentation — each repo's docs/ and README, ingested and associated with the repo. Rendered through the <<<RepoDocs>>> tag.
← omni-git docs · docs/go-port-spec.md
Status: Draft · a byte-equivalent Go implementation, proven against the Python oracle.
Produce a Go implementation of omni-git that is byte-for-byte interoperable with the Python one: the same content addresses, the same on-disk store, the same REST wire — so a store written by either can be read/verified/pulled by the other, and either CLI is a drop-in. Correctness is not asserted; it is proven against the existing Python 254-pin oracle and a differential equivalence harness (the #489/#490 capability-parity / monolith-retirement discipline; precedent: congruency-go and docs/research/go_write_route_equivalence.md). UNPROVEN ≠ pass — the Go port is adopted for any purpose only where the gate is green.
Size of the subject (measured): the tool is ~8.3k lines of Python (store.py 5.7k, cli.py, api.py/REST, mirror.py, imgfmt.py, forge.py); diskforge (bundled) is ~6.2k lines; the selftest.py oracle is ~18.6k lines and is REUSED, not ported.
The Go store MUST produce, for identical inputs, identical content addresses and identical stored object bytes as the Python store. Everything else follows from this. The address is omni:<kind>:<sha256(canon(manifest))>, so the port lives or dies on reproducing canon() and each kind's manifest schema exactly. If addresses match, dedup, find, pull, fsck, export-git, and cross-language interop all fall into place; if they drift by one byte, nothing composes.
Extract from the Python and pin as a normative sub-spec (the Go and Python both cite it): - Manifest canonicalization — the exact bytes of canon(obj) (store.py): key ordering, separators, unicode/escaping, number formatting. The Go canon must be byte-identical. - Per-kind manifest schema — the exact fields for file, folder, tar.gz, img/qcow2, git (incl. member records, nested, shadowed, xattr/mtime/mode fields, dev nodes, symlink targets, hardlink handling). - Address + object layout — omni:<kind>:<sha>; blobs at objects/<sha[:2]>/<sha[2:]>; the manifests/<sha> marker; the guard file (OMNI-ADDRESS.txt). - REST wire — every route, method, request/response bytes, and error shapes (api.py), so a Python serve and a Go client (and vice-versa) interoperate. - Streaming/verify semantics — sha-on-read verification, self-heal on dedup-write, OMNI_CHUNK_BYTES. This sub-spec is the single source both languages implement to; changing it changes both.
Two complementary harnesses, both preventive and mechanical (no model calls): 1. Differential harness (the go_write_route_equivalence pattern): run identical operation sequences through the Python and Go CLIs/servers in identical worlds; assert byte-identical addresses, object bytes, manifest bytes, REST replies, and pulled artifacts — equal modulo the clock (mtime/ts columns). Any divergence blocks. 2. Oracle reuse — the Python selftest.py already drives a store through RemoteStore over REST (the api: pins). Point that oracle at a Go omni serve: the REST-drivable pins become a cross-language conformance suite at zero test-authoring cost. The non-REST pins are covered by the differential harness or ported per-kind. A phase is "done" only when both are green for that seam. UNPROVEN (a kind with no oracle coverage, a harness with zero cases) blocks, never passes.
- Store core — the 5-method transport surface (put_blob/get_blob/has_blob/_record_manifest/ manifest_shas) is the natural seam; everything else is written in terms of it (as in Python). Port this first; it is where address-equivalence is won. stdlib: crypto/sha256, os, io (streamed put_file/get_blob_to_path — chunked at OMNI_CHUNK_BYTES). - Git kind — shell out, do NOT reimplement git. The Python makes 116 git subprocess calls; Go does the same via os/exec. The bundle/stash/reflog/rebase/bisect/gitdir "side-pocket" logic is orchestration, not a git engine. (Do not adopt go-git — it would diverge from the Python's exact git behavior the oracle pins.) - tar/folder/file — archive/tar, os; reproduce member indexing, shadowing (last-writer-wins), hardlink/symlink/dev handling, and recursion exactly. - REST — net/http mirroring api.py routes and byte shapes. - CLI — mirror cli.py command surface, flags, exit codes, and the "in omni's voice" error strings the oracle checks (confusion:* pins assert the CLI names every command).
Image guest-file indexing needs diskforge (~6.2k lines of usermode ext4/FAT/exFAT + qcow2). Options, pick one in Q2: - (a) Defer — Go-omni ships image ingest as whole-image blob only (no guest walk), degrading exactly like the Python MemberIndexError/ImportError path. Fully byte-equivalent on the whole-image manifest; guest-file dedup simply absent until (b). Recommended for v1. - (b) Port diskforge to Go — large, but it is kernel-judged (e2fsck/mount in a sandbox), so it reuses diskforge's own oracle; a separate sub-port with its own equivalence gate. - (c) Interop — Go-omni shells to a diskforge binary/service. Fastest to guest-file parity, but couples the Go port to the Python runtime. Whichever: the whole-image archive path stays byte-identical, so image pull is unaffected.
0. Freeze the §3 contracts + stand up the differential harness. 1. Store core + 5-method seam; prove address-equivalence on file/folder. 2. tar.gz (+ recursion) + streaming; oracle's tar pins pass differentially. 3. REST serve + client; run the Python oracle's api: pins against Go serve. 4. git kind (git-shelled) — the deepest tool logic. 5. img/qcow2 per the §6 decision. 6. export-git/import-git, git-guard, mirror, forge. Ship/adopt only the seams whose gate is green; the rest keep using Python (both interoperate on one store).
1. Address equivalence. For a corpus of file/folder/tar/git artifacts, Go and Python emit the same omni:<kind>:<sha> and identical object+manifest bytes. Neg control: a manifest field reordered in Go → different address → blocked. 2. Cross-language pull. A store written by Python pulls byte-identically via the Go CLI, and vice-versa. Neg control: a Go pull that reassembles from members instead of the whole blob diverges. 3. REST interop. Python serve ↔ Go client and Go serve ↔ Python client round-trip; the Python oracle's api: pins pass against Go serve. Neg control: a differing error-shape byte fails a pin. 4. Verify/heal parity. A rotted object is caught on read and healed on re-ingest identically in both. Neg control: a Go dedup that trusts the filename (skips re-hash) keeps the rot. 5. Streaming parity. A multi-GB blob ingests/pulls at ~OMNI_CHUNK_BYTES in Go too. Neg control: a whole-file read OOMs under a tight cap. 6. git-artifact fidelity. A repo with stash/reflog/rebase/bisect carries round-trips identically. Neg control: a missing side-pocket carry → the Python oracle's git pins go red against Go.
Not a rewrite of behavior — the Python is the reference; the Go matches it, including its edge cases. Not a git engine (shell to git). Not (v1) a diskforge port unless Q2 chooses (b). Not a new address scheme or store format.
- Q1: cross-language interop required (byte-identical stores, recommended) vs. Go as an independent store? Interop makes the equivalence gate the acceptance bar and is strongly preferred. - Q2: diskforge — defer (a, recommended v1), port (b), or interop (c)? - Q3: repo home — a new omni-git-go (GitLab/GitHub) tracked beside the Python, mirrored into forge. - Q4: how much of the 18.6k-line oracle is REST-drivable vs. needs a differential port? (Measure the api:-tagged pin fraction first — it sets the "free coverage" baseline.) - Q5: retirement policy — does Go ever replace Python, or run as a verified peer? (The #490 gate governs any replacement; peer-forever is the safe default.)
The Python selftest.py oracle (as the conformance suite over REST), the go_write_route_equivalence differential method and the #489/#490 capability-parity gate, congruency-go as the porting-under- equivalence precedent, the existing git CLI (shelled, not reimplemented), and diskforge's kernel oracle if Q2 picks (b).