Storage & ArcologyFS

One continuous line from an interrupt to a filesystem

Storage did not start as a filesystem design. It started with a timer: could the substrate route a hardware interrupt at all, reliably, and prove it under QEMU rather than assume it. Once that held, a dispatch loop could register hooks against it. Once the loop held, a block-storage substrate could expose a RAM disk and a read-only FAT32 provider behind one typed contract. Only then did ArcologyFS, the native, object-identity-first filesystem The Arcology was always going to need, have solid ground to stand on.

This has been the most difficult, tedious part of the entire project so far. Not any single hard problem, but the sheer number of small ones that had to be gotten right in order, each one a precondition for the next, with no shortcut past any of them.

Each layer is its own numbered RFC, each with its own QEMU-executed proof before the next layer was allowed to depend on it. Nothing here is a diagram of an intended design; every claim below is backed by a fixture that actually ran.

What exists now

A routed hardware timer interrupt and a substrate dispatch loop hooks register against
A typed BlockDevice contract with a RAM-disk implementation and a real UEFI Block I/O provider, dispatched by policy alongside the RAM disk rather than replacing it
ArcologyFS: stable 64-bit object identity independent of path, colon-delimited namespace, real handles
A copy-on-write commit protocol proven crash-safe by deliberately crashing it mid-commit
Reflinks with genuine copy-on-write: two objects share one physical page until either writes
Snapshots, rollback, a self-healing boot-activation policy, and a distinct recovery-environment path
A real Arcology System Namespace that attaches an ArcFS volume and delegates path resolution into it
A GOP-framebuffer volume-health indicator, rendered and verified pixel-by-pixel under QEMU
A real byte-per-block allocation bitmap and a real reuse-before-growth data allocator, so a volume can free space and mean it
A growable on-disk object tree, built and walked as a real B+tree rather than a fixed array
The first artifact in the entire project confirmed on real physical hardware: written to an actual USB stick, booted on an actual laptop, its own boot marker photographed on an actual screen

Proof stopped being optional the day it left QEMU

Every claim on this page was true under QEMU months before it was true anywhere else, and that gap was never hidden. But an emulator is, in the end, a piece of software written to be honest with another piece of software. Physical media is not obligated to cooperate with anyone. The first time this project's own boot and storage work touched a real USB stick and a real laptop, it worked, and a photograph of that machine's own screen exists as the actual receipt.

That single photograph matters more than its size suggests. It is the line between "the architecture is sound" and "the architecture survived contact with a world that was never designed to make this easy." Everything upstream of it, the commit protocol, the reflink gate, the self-healing recovery, was already proven rigorously. This is the proof that rigor and reality actually agree.

Identity is the object, not the path

ArcologyFS’s central design decision: a path is a lookup expression, not identity. Renaming or moving a file changes only a namespace record; the object’s own identifier, and therefore every open handle and every reference to it, never changes. The excerpt below is the real path-resolution walk. It never once touches the object’s own identity while descending through parent directories.

ArcoBASICArcologyFS source: resolving a colon-delimited path to a stable object ID

' Paths are root-relative and begin with ':', e.g. ":home:documents:notes.txt".
' The OID returned here is stable across rename, move, and reboot -- it is
' never derived from the path string itself, only looked up through it.
FUNCTION ArcFS.Resolve(pathBuffer AS MMIOPTR, pathLength AS U64) AS U64
    IF pathLength = 0 THEN RETURN 0
    IF MEMORY.Read8(pathBuffer) <> 58 THEN RETURN 0
    IF pathLength = 1 THEN RETURN 1 ' the root itself has no name of its own

    LET current AS U64 = 1
    LET cursor AS U64 = 1
    WHILE cursor < pathLength
        LET componentStart AS U64 = cursor
        WHILE cursor < pathLength AND MEMORY.Read8(ADDRESS.Offset(pathBuffer, cursor)) <> 58
            cursor = cursor + 1
        WEND
        LET componentPtr AS MMIOPTR = ADDRESS.Offset(pathBuffer, componentStart)
        LET componentLength AS U64 = cursor - componentStart
        ' A namespace lookup returns the CHILD'S stable OID -- renaming that
        ' child later only ever touches this namespace record, never the OID.
        LET nextOid AS U64 = ArcFS.Lookup(current, componentPtr, componentLength)
        IF nextOid = 0 THEN RETURN 0
        current = nextOid
        cursor = cursor + 1
    WEND
    RETURN current
END FUNCTION

Crash safety proven by actually crashing it

A copy-on-write commit writes every new record to fresh sectors the previous generation never touched, then becomes durable through exactly one single-sector write to the superblock. Splitting "prepare" from "publish" is what makes the crash-safety claim testable without simulating a real power failure: calling Prepare and simply never calling Publish is the crash. The fixture that proves this calls Prepare, stops, remounts, and confirms the old generation is the only one any reader can see. Then it completes the commit and confirms the new generation is visible instead.

ArcoBASICArcologyFS source: the atomic publish boundary

' Everything up to this function writes to sectors the active generation
' never used. Nothing becomes visible until this ONE sector write lands.
FUNCTION ArcFS.PublishCommit() AS BOOL
    LET state AS MMIOPTR = ArcFSMountStateAddress()
    LET pendingCheckpointSector AS U64 = MEMORY.Read64(ADDRESS.Offset(state, 72))
    IF pendingCheckpointSector = 0 THEN RETURN 0 ' nothing was ever prepared

    LET scratch AS MMIOPTR = ArcFSSectorScratchAddress()
    ArcFSWriteSuperblockRecord(scratch, pendingCheckpointSector, RAMDisk.SectorCount())
    IF RAMDisk.WriteSectors(0, 1, scratch) = 0 THEN RETURN 0

    ' The superblock now points at the new generation. A crash one line
    ' earlier leaves the OLD generation as the only one that ever existed.
    MEMORY.Write64(state, pendingCheckpointSector)
    MEMORY.Write64(ADDRESS.Offset(state, 72), 0)
    RETURN 1
END FUNCTION

' The convenience wrapper for the ordinary, non-crash-testing case.
FUNCTION ArcFS.CommitImage() AS BOOL
    IF ArcFS.PrepareCommit() = 0 THEN RETURN 0
    RETURN ArcFS.PublishCommit()
END FUNCTION

Reflinks that actually share, until they can’t

A reflinked file shares its source’s physical data. Nothing is copied at reflink time. The gate below is what makes that safe: every mutating access checks whether its data slot is still exclusively its own, and only copies when it discovers it is not, immediately before the write that would otherwise have corrupted a second object’s visible content. For every object created before this existed, the check is a single comparison that returns instantly. Reflinks did not slow down, or change the behavior of, a single line of code written before them.

ArcoBASICArcologyFS source: the copy-on-write gate every write passes through

FUNCTION ArcFSEnsurePrivateSlot(row AS U64) AS U64
    LET base AS MMIOPTR = ArcFSObjectRowAddress(row)
    LET dataSlot AS U64 = MEMORY.Read64(ADDRESS.Offset(base, 24))
    LET refcount AS U64 = MEMORY.Read64(ADDRESS.Offset(ArcFSSlotRefCountAddress(), dataSlot * 8))

    ' The overwhelmingly common case: nothing else references this slot.
    IF refcount <= 1 THEN RETURN dataSlot

    ' Shared -- copy the bytes to a fresh slot BEFORE the caller writes,
    ' so a reflinked write can never become visible on the other object.
    LET newSlot AS U64 = ArcFSAllocateDataSlot()
    IF newSlot = ArcFSMaxObjects() THEN RETURN ArcFSMaxObjects() ' out of room; refuse the write

    LET i AS U64 = 0
    WHILE i < ArcFSFileCapacityBytes()
        LET v AS U8 = MEMORY.Read8(ADDRESS.Offset(ArcFSDataPoolAddress(), dataSlot * ArcFSFileCapacityBytes() + i))
        MEMORY.Write8(ADDRESS.Offset(ArcFSDataPoolAddress(), newSlot * ArcFSFileCapacityBytes() + i), v)
        i = i + 1
    WEND
    MEMORY.Write64(ADDRESS.Offset(ArcFSSlotRefCountAddress(), dataSlot * 8), refcount - 1)
    MEMORY.Write64(ADDRESS.Offset(base, 24), newSlot)
    RETURN newSlot
END FUNCTION

What a snapshot is actually for

ArcFS's allocator originally never reused a sector at all, which meant nothing had ever literally been destroyed -- every past generation's bytes just sat on the device, unreachable but intact. At that point, a snapshot protected against nothing yet; it only pinned a generation so a real reclaiming allocator, whenever one existed, would know not to touch it. That honest, slightly uncomfortable admission turned out to be worth writing down, because it is exactly what got built next: `ArcFS.ReclaimGeneration` now exists as a genuinely ordinary commit, and a pinned snapshot is the one thing standing between an old generation and real, physical reuse of its space.

Rollback follows the same honesty: undoing a bad update republishes a previously pinned generation as active through the exact same single-sector publish boundary shown above. It is a real, atomic undo, not a restore-by-copy, and the fixture proves the volume is fully read-write usable immediately afterward, not merely readable.

Recovery that repairs itself, and says so

Two different boot policies consume the same underlying scrub and repair primitives. Ordinary boot activates a volume, and if it finds a structural defect it can actually fix, repairs it and re-verifies before ever reporting anything to whatever runs next: self-healing, silent when it succeeds. A distinct recovery-environment boot artifact stages the identical repair as three separately observable steps instead, because a recovery operator needs visibility an ordinary boot does not.

ArcoBASICArcologyFS source: self-healing activation, re-verified before trusting itself

' 0 = Unavailable, 1 = Healthy/read-write, 2 = ReadOnlySafety
FUNCTION ArcFS.ActivateSystemVolume() AS U64
    LET result AS U64 = ArcFS.MountImageSafe()
    IF result = 0 THEN RETURN 0
    IF result = 1 THEN RETURN 1

    ' A recoverable defect was found. Fix it, then CONFIRM the fix held --
    ' never report success on a repair that was only attempted.
    ArcFS.RepairReattachOrphans()
    IF ArcFS.RepairCommit() = 0 THEN RETURN 2

    LET reactivated AS U64 = ArcFS.MountImageSafe()
    IF reactivated = 1 THEN RETURN 1
    RETURN 2
END FUNCTION

A namespace ArcFS can actually attach to

ArcFS always had an internal namespace; what it never had was somewhere above it to attach into. The Arcology System Namespace is that: a general, colon-delimited tree where a path prefix can be marked as a filesystem attachment, so resolving a path that crosses the attachment boundary reconstructs the unconsumed remainder and hands it to the attached volume’s own, completely unmodified Resolve. ArcFS itself needed zero changes for this to work.

ArcoBASICArcology System Namespace source: delegating across the attachment boundary

' Reached a Filesystem Attachment before the path was exhausted --
' everything still unconsumed becomes ArcFS's own problem, verbatim.
IF kind = NamespaceKindFilesystemAttachment() THEN
    LET remainderPtr AS MMIOPTR = ADDRESS.Offset(pathBuffer, cursor)
    LET remainderLen AS U64 = pathLength - cursor
    IF remainderLen = 0 THEN
        remainderPtr = NamespaceRootLiteralAddress() ' the attachment IS the query -- delegate to root
        remainderLen = 1
    END IF
    LET arcfsOid AS U64 = ArcFS.Resolve(remainderPtr, remainderLen)
    IF arcfsOid = 0 THEN RETURN 0
    ' Two different, non-comparable ID spaces -- tag the result rather
    ' than let a caller silently mistake one for the other.
    MEMORY.Write64(result, 2)
    MEMORY.Write64(ADDRESS.Offset(result, 8), arcfsOid)
    RETURN 1
END IF

Honest current limits

The limits named the first time this page was written have mostly been closed since: real free-space reclamation now exists, files are no longer pinned to one fixed extent, the object tree now grows as a real on-disk B+tree instead of a fixed array, and typed attributes now survive a remount. Naming a gap in public was never meant to be permanent; it was meant to be a promise with a date attached, and most of that particular list of promises has now been kept.

What has not: the Namespace and Attribute trees are still flat arrays rather than the same growable structure the object tree now uses, sparse files and persistent full health scoring remain unbuilt, and deleting an object still does not yet free its own row for reuse. These are the honest limits as they stand today, not the ones from a year ago left uncorrected out of convenience.

What was proven from the very first version, and has not needed revisiting since, is still the part that is hardest to get right and easiest to get subtly wrong: the commit protocol survives a crash at any point with no impossible hybrid state, reflinks cannot leak a write across objects, and repair never trusts its own fix without re-checking it. Everything named above was built on top of that foundation, not around it.

The part worth stealing

None of the specific mechanisms above are the actual point. A different project will have a different object model, a different commit boundary, different failure modes worth testing. What generalizes is the discipline underneath them, and it is worth naming directly rather than leaving a reader to infer it from the code.

Never trust that hardware or a runtime behaves the way the documentation says it does. Prove it under a real environment before anything else is allowed to depend on it. That is why this whole line of work started with a timer interrupt instead of a filesystem.
Find the single point where a change becomes real, and make it exactly one atomic operation. Everything written before that point is disposable; nothing after it is optional. Once that point exists, crash safety is a property you can point at, not a hope.
Test a safety claim by actually breaking it. Stop a process at the exact boundary you claim is safe and check what a fresh observer sees, rather than reasoning your way to confidence. If the claim survives that, it is a claim. If it does not, you just found the bug before a user did.
When a design decision turns out to mean something different than assumed, let the reframing change the design instead of forcing the original assumption to stay true. Discovering that a snapshot was never protecting against deletion, only pinning work for a future step, was worth more than defending the original framing would have been.
Build the smallest layer that proves the hard contract, then only build the next layer on top of something already proven. Speed comes from the proof gate, not from skipping it.
None of this is filesystem-specific, and that is the point. Crash safety, identity kept separate from naming, and copy-on-write sharing show up anywhere state has to survive a failure, anywhere two things need to safely share something until one of them changes it, and anywhere a name needs to be free to change without breaking what depended on it. If you are building something else entirely and any of that sounds familiar, it should.

LEARNING LAYER

Key terms, in plain language

You do not need a systems background to follow the work. These are the specialized terms used on this page.

OIDObject ID: A stable identifier for a filesystem object that never changes, even when the object is renamed or moved. A path finds an object; the OID is what the object actually is.
Copy-on-write: A technique where shared data is only ever duplicated at the moment something tries to change it, not before. Until then, two references can safely point at the exact same bytes.
Checkpoint: A committed, self-consistent record of everything a filesystem needs to describe its current state. A new checkpoint only becomes official through one final, atomic write.
Reflink: A fast file duplication where the copy initially shares the same underlying data as the original, instead of copying every byte immediately.
Block device: A storage device, or a stand-in for one, that reads and writes data in fixed-size chunks called sectors rather than arbitrary byte ranges.
Checksum: A small computed value derived from a larger block of data, stored alongside it and rechecked later to detect whether that data was corrupted.
Namespace attachment: Connecting a filesystem’s own internal folder structure to a location inside a larger, shared naming system, so paths can cross from one into the other.
UEFIUnified Extensible Firmware Interface: The standardized firmware environment that starts a modern computer before an operating system takes control. Arcology OS currently enters through UEFI during hardware bring-up.