S3 - Mirage

The S3 VFS mounts an Amazon S3 bucket (or any S3-compatible service) at some prefix such as /s3/. All operations involve network I/O to the remote object store. Uses aioboto3 for async S3 access. Compatible with: AWS S3, MinIO, Cloudflare R2, Supabase Storage, DigitalOcean Spaces, and any S3-compatible service. Install the extra:

pip install "mirage-ai[s3]"

Config

from mirage import MountMode, Workspace
from mirage.vfs.s3 import S3VFS, S3Config

config = S3Config(
    bucket="my-bucket",
    region="us-east-1",
    # Optional:
    # aws_access_key_id="...",
    # aws_secret_access_key="...",
    # aws_profile="my-profile",
    # endpoint_url="http://localhost:9000",  # MinIO
    # path_style=True,  # For MinIO/R2
    # timeout=30,
    # proxy="http://proxy:8080",
)

vfs = S3VFS(config)
ws = Workspace({"/s3": vfs}, mode=MountMode.READ)

S3VFS(config) takes an S3Config object with the bucket name and optional credentials, endpoint, and connection settings. Both READ and WRITE modes are supported.

Field	Description
`timeout` default `30`	Seconds a request may wait to connect, or between two reads of its answer, before it fails; a download that keeps flowing is never cut off.
`region`	AWS region requests are signed for and sent to. Unset leaves it to the AWS SDK’s environment variables and profile.
`aws_access_key_id` secret	Access key ID; set it with `aws_secret_access_key`. Unset uses the AWS credential chain.
`aws_secret_access_key` secret	Secret access key paired with `aws_access_key_id`.
`aws_session_token` secret	Session token for temporary credentials, sent with the access key pair.
`aws_profile`	Named profile in the shared AWS config and credentials files that supplies credentials when no access key pair is given.
`bucket` required	Name of the bucket to mount.
`endpoint_url`	Endpoint URL of an S3-compatible server. Unset uses `AWS_ENDPOINT_URL_S3` or `AWS_ENDPOINT_URL` when set, else AWS’s endpoint for `region`; an endpoint set only in an AWS profile is not read.
`path_style` default `False`	Address the bucket in the URL path (`<endpoint>/<bucket>/<key>`) instead of as a subdomain of the endpoint.
`proxy` secret	Proxy URL for every HTTP and HTTPS request to the endpoint.
`key_prefix`	Key prefix inside the bucket that the mount treats as its root; keys outside it are not visible. Leading slashes are dropped and a trailing `/` is added.
`default_content_type`	`Content-Type` stored with every object the mount writes. Unset sends none, so the store applies its own default.

Fields marked secret are redacted in snapshots.

VFS.md

The VFS.md an agent reads with S3 mounted at /<path>.

Rendered
Raw

Virtual Filesystem

Path	Description
`/<path>`	`s3`, write

Supported Unix Commands for VFS

Unix commands work on each path as on a local filesystem, as far as the path supports them:

Path	Supported
`/<path>`	Almost all Unix commands.

Limitations of Unix Commands for VFS

Where a path differs from a local filesystem:

Path	Limitations
`/<path>`	`grep -r`, `find` and `du` stop after 1000 files; a glob matching more, `tar -c`, `cp -r` and `rm -r` are refused.

Run man <cmd> or <cmd> --help for one command’s usage.

---
name: workspace-filesystem
description: The virtual filesystem layout of this workspace.
---

# Virtual Filesystem

| Path | Description |
| --- | --- |
| `/<path>` | `s3`, write |

## Supported Unix Commands for VFS

Unix commands work on each path as on a local filesystem, as far as the path supports them:

| Path | Supported |
| --- | --- |
| `/<path>` | Almost all Unix commands. |

## Limitations of Unix Commands for VFS

Where a path differs from a local filesystem:

| Path | Limitations |
| --- | --- |
| `/<path>` | `grep -r`, `find` and `du` stop after 1000 files; a glob matching more, `tar -c`, `cp -r` and `rm -r` are refused. |

Run `man <cmd>` or `<cmd> --help` for one command's usage.

The S3 VFS maps S3 object keys to virtual paths under the mount prefix. S3 “directories” are prefix-based - there are no real directory objects. For example, if bucket my-bucket contains:

data/file.txt
data/config.json
reports/q1.csv
reports/q2.csv

Then mounting at /s3/ exposes:

/s3/
  data/
    file.txt
    config.json
  reports/
    q1.csv
    q2.csv

Path mapping: virtual /s3/data/file.txt maps to S3 key data/file.txt.

Cache

The S3 VFS uses IndexCacheStore with index_ttl = 600 (10 minutes). Directory listings are cached for up to 600 seconds before being refreshed from S3. This reduces API calls for repeated directory traversals. A configured index: replaces this lifetime with its own ttl, and the mount’s ttl: caps it.

Conditional writes

Under write: conditional a write carries the version of the file mirage read, and S3 refuses it when the file has changed since. The endpoint decides which writes can be conditioned: AWS’s conditions every write, copy and delete; any other endpoint_url (or AWS_ENDPOINT_URL_S3/AWS_ENDPOINT_URL) only a write, so mirage refuses what would copy or delete there. An endpoint set only in an AWS profile is not read: give a non-AWS server its endpoint_url. If the session last saw no file at the path, the write sends If-None-Match: * and fails if someone created the file in the meantime. A conditional single-file move attempts rollback when source deletion is refused. The move still fails and warns if the destination may remain. Cleanup uses the copy’s S3 version or, for a new destination, its ETag. Unversioned overwrites, same-ETag replacements, retried versioned copies, and uncertain delete outcomes limit recovery; inspect both paths after a failure. An absent destination must remain absent until copying. This guard may add one metadata request; cleanup adds one delete and may require s3:DeleteObjectVersion. Directory and cross-mount moves do not use rollback. A directory mv, rm -r or rmdir looks at the directory again when it is done: a file written into it that the command never listed stays, and the command fails with Directory not empty. Under write: conditional, a directory mv that finds a file changed before it could copy it removes nothing from the source; the copies it made stay at the destination. An append or another rewrite that observes a missing file uses create-only commit under either write policy. S3 checks If-None-Match: * at PUT or final multipart completion, so a delayed creation cannot replace a competing writer’s bytes. A lost condition returns an error without automatic retry. Existing-file appends under unconditional retain last-write-wins behavior.

Refused keys

When credentials cannot read an S3 object (S3 returns AccessDenied, or a bodiless HTTP 403 response on a HEAD request), Mirage reports EACCES (Permission denied) for that file operand. Shell commands follow standard POSIX conventions:

cat a b c prints the contents of a and c, writes a permission error for b to stderr, and exits with a non-zero status.
grep -r logs an error on inaccessible files and exits with status 2.
cp b x || fallback triggers the fallback branch.
test -e b evaluates to false.

IAM permissions and existence checks

Scoped grants: When access is granted below a specific prefix (for example, s3:GetObject on bucket/dir/* alongside s3:ListBucket on the bucket), Mirage treats dir as a directory during stat.
Missing ListBucket: Without s3:ListBucket permission, S3 returns HTTP 403 even for non-existent keys to avoid leaking existence. As a result, non-existent files report Permission denied (EACCES) instead of No such file or directory (ENOENT).
Listed but unreadable keys: If ListObjectsV2 includes an object that your credentials cannot read, stat and test -e report that the file exists, while read attempts (cat) fail with Permission denied.

Integration

What S3 answers through its own API, so fewer files are read. Integration explains each command.

Command	On S3
`md5sum`	The object’s ETag, when it is the MD5: a single-part upload, unencrypted or under SSE-S3
`find`, `du`, `ls -R`, `tree`	One recursive listing

Example

import asyncio

from mirage import MountMode, Workspace
from mirage.vfs.s3 import S3VFS, S3Config

config = S3Config(
    bucket="my-bucket",
    region="us-east-1",
)

vfs = S3VFS(config)


async def main() -> None:
    ws = Workspace({"/s3/": vfs}, mode=MountMode.READ)

    r = await ws.shell("ls /s3/")
    print(await r.stdout_str())

    r = await ws.shell("cat /s3/data/file.txt")
    print(await r.stdout_str())

    r = await ws.shell("tree /s3/")
    print(await r.stdout_str())

    r = await ws.shell("find /s3/ -name '*.json'")
    print(await r.stdout_str())

    r = await ws.shell("grep example /s3/data/config.json")
    print(await r.stdout_str())

    r = await ws.shell("stat /s3/data/file.txt")
    print(await r.stdout_str())


if __name__ == "__main__":
    asyncio.run(main())

Shell Commands

The S3 VFS supports the full set of shell commands since it operates on real file content (text, binary, JSON, CSV, etc.). Large files benefit from range reads to avoid downloading entire objects.

Customize a builtin command

The generic command definitions are available before a workspace is created. Copy the definition with the behavior you want to change, then register that copy on one mount:

from dataclasses import replace

from mirage import generic

cat = generic("cat")

async def logged_cat(accessor, paths, texts, opts):
    print("reading", [path.virtual for path in paths])
    return await cat.fn(accessor, paths, texts, opts)

ws.mount("/s3/").register(replace(cat, fn=logged_cat))

The builtin definition and other mounts stay unchanged. Registration is a host application action; an agent can discover and run the resulting cat command, but cannot alter the command table.

Read Commands

Command	Notes
`cat`	Read file content
`head` / `tail`	First/last N lines
`grep` / `rg`	Pattern search (file or directory level)
`jq`	Query JSON fields
`wc`	Line/word/byte counts
`stat`	File metadata (name, size, type, modified)
`find`	Recursive search with `-name`, `-maxdepth`
`tree`	Directory tree view
`nl`	Number lines
`du`	Disk usage summary
`file`	Detect file type
`strings`	Extract printable strings from binary
`xxd`	Hex dump
`md5`	MD5 checksum
`sha256sum`	SHA-256 checksum

Text Processing

Command	Notes
`awk`	Pattern scanning and processing
`sed`	Stream editor
`tr`	Translate or delete characters
`sort`	Sort lines
`uniq`	Remove duplicate lines
`cut`	Extract fields/columns
`join`	Join lines on a common field
`paste`	Merge lines side by side
`column`	Columnate output
`fold`	Wrap lines to a specified width
`expand`	Convert tabs to spaces
`unexpand`	Convert spaces to tabs
`fmt`	Simple text formatter
`rev`	Reverse lines
`tac`	Concatenate and print in reverse
`look`	Display lines beginning with a given string
`shuf`	Shuffle lines
`tsort`	Topological sort
`comm`	Compare two sorted files
`cmp`	Compare two files byte by byte
`diff`	Compare files line by line
`patch`	Apply a diff patch
`iconv`	Character encoding conversion

File Operations

Command	Notes
`cp`	Copy files
`mv`	Move/rename files
`rm`	Remove files
`mkdir`	Create directories
`touch`	Create empty file or update timestamp
`ln`	Create symbolic links
`tee`	Write stdin to file and stdout
`mktemp`	Create temporary file
`split`	Split file into pieces
`csplit`	Split file by context

Path Utilities

Command	Notes
`basename`	Strip directory from path
`dirname`	Strip filename from path
`realpath`	Resolve path
`readlink`	Print symbolic link target
`ls`	List directory contents

Compression

Command	Notes
`gzip`	Compress files
`gunzip`	Decompress gzip files
`zip`	Create zip archives
`unzip`	Extract zip archives
`tar`	Archive files
`zcat`	Cat compressed files
`zgrep`	Grep compressed files

Encoding

Command	Notes
`base64`	Base64 encode/decode

Use Cases

AI agents accessing cloud data: Mount S3 buckets for agents to read and process remote datasets
Data pipelines: Read and write S3 objects with shell-like commands
Sandboxed cloud storage access: Restrict agent operations to a specific bucket and prefix
FUSE mounting: Expose S3 buckets through a virtual FUSE mount for external tools

Scoping a VFS to a key prefix

Pass key_prefix: str | None = None to S3Config to transparently scope every operation to a subpath of the bucket:

S3VFS(S3Config(
    bucket="app-data",
    region="eu-west-1",
    key_prefix=f"users/{user_id}/",
))

When set, every read/write/list/stat/copy/rename/delete operation is transparently scoped to that bucket subpath. Agents see clean paths like /data/notes.md; the underlying bucket key is users/{user_id}/data/notes.md. Useful for multi-tenant systems. Pairs naturally with STS AssumeRole session policies for AWS-side enforcement. Unset behavior is unchanged. Normalization: leading slashes are stripped and a trailing slash is added automatically. Both None and an empty string are treated as “no prefix.”

Getting Started

Access

AI Agents

VFS

CLIs

Runtimes

S3

Config

VFS.md

Virtual Filesystem

Supported Unix Commands for VFS

Limitations of Unix Commands for VFS

Cache

Conditional writes

Refused keys

IAM permissions and existence checks

Integration

Example

Shell Commands

Customize a builtin command

Read Commands

Text Processing

File Operations

Path Utilities

Compression

Encoding

Use Cases

Scoping a VFS to a key prefix

​Config

​VFS.md

​Virtual Filesystem

​Supported Unix Commands for VFS

​Limitations of Unix Commands for VFS

​Cache

​Conditional writes

​Refused keys

​IAM permissions and existence checks

​Integration

​Example

​Shell Commands

​Customize a builtin command

​Read Commands

​Text Processing

​File Operations

​Path Utilities

​Compression

​Encoding

​Use Cases

​Scoping a VFS to a key prefix

Config

VFS.md

Virtual Filesystem

Supported Unix Commands for VFS

Limitations of Unix Commands for VFS

Cache

Conditional writes

Refused keys

IAM permissions and existence checks

Integration

Example

Shell Commands

Customize a builtin command

Read Commands

Text Processing

File Operations

Path Utilities

Compression

Encoding

Use Cases

Scoping a VFS to a key prefix