Documentation Index
Fetch the complete documentation index at: https://docs.mirage.strukto.ai/llms.txt
Use this file to discover all available pages before exploring further.
The S3 resource mounts an Amazon S3 bucket (or any S3-compatible service) at
some prefix such as /s3/. All operations involve network I/O to the remote
object store. Uses aioboto3 for async S3 access.
Compatible with: AWS S3, MinIO, Cloudflare R2, Supabase Storage,
DigitalOcean Spaces, and any S3-compatible service.
Config
from mirage import MountMode, Workspace
from mirage.resource.s3 import S3Resource, S3Config
config = S3Config(
bucket="my-bucket",
region="us-east-1",
# Optional:
# aws_access_key_id="...",
# aws_secret_access_key="...",
# aws_profile="my-profile",
# endpoint_url="http://localhost:9000", # MinIO
# path_style=True, # For MinIO/R2
# timeout=30,
# proxy="http://proxy:8080",
)
resource = S3Resource(config)
ws = Workspace({"/s3": resource}, mode=MountMode.READ)
S3Resource(config) takes an S3Config object with the bucket name and
optional credentials, endpoint, and connection settings. Both READ and
WRITE modes are supported.
Filesystem Layout
The S3 resource maps S3 object keys to virtual paths under the mount prefix.
S3 “directories” are prefix-based - there are no real directory objects.
For example, if bucket my-bucket contains:
data/file.txt
data/config.json
reports/q1.csv
reports/q2.csv
/s3/ exposes:
/s3/
data/
file.txt
config.json
reports/
q1.csv
q2.csv
/s3/data/file.txt maps to S3 key data/file.txt.
Cache
The S3 resource uses IndexCacheStore with _index_ttl = 600 (10 minutes).
Directory listings are cached for up to 600 seconds before being refreshed
from S3. This reduces API calls for repeated directory traversals.
Example
import asyncio
from mirage import MountMode, Workspace
from mirage.resource.s3 import S3Resource, S3Config
config = S3Config(
bucket="my-bucket",
region="us-east-1",
)
resource = S3Resource(config)
async def main() -> None:
ws = Workspace({"/s3/": resource}, mode=MountMode.READ)
r = await ws.execute("ls /s3/")
print(await r.stdout_str())
r = await ws.execute("cat /s3/data/file.txt")
print(await r.stdout_str())
r = await ws.execute("tree /s3/")
print(await r.stdout_str())
r = await ws.execute("find /s3/ -name '*.json'")
print(await r.stdout_str())
r = await ws.execute("grep example /s3/data/config.json")
print(await r.stdout_str())
r = await ws.execute("stat /s3/data/file.txt")
print(await r.stdout_str())
if __name__ == "__main__":
asyncio.run(main())
Shell Commands
The S3 resource supports the full set of shell commands since it operates
on real file content (text, binary, JSON, CSV, etc.). Large files benefit
from range reads to avoid downloading entire objects.
Read Commands
| Command | Notes |
|---|
cat | Read file content |
head / tail | First/last N lines |
grep / rg | Pattern search (file or directory level) |
jq | Query JSON fields |
wc | Line/word/byte counts |
stat | File metadata (name, size, type, modified) |
find | Recursive search with -name, -maxdepth |
tree | Directory tree view |
nl | Number lines |
du | Disk usage summary |
file | Detect file type |
strings | Extract printable strings from binary |
xxd | Hex dump |
md5 | MD5 checksum |
sha256sum | SHA-256 checksum |
Text Processing
| Command | Notes |
|---|
awk | Pattern scanning and processing |
sed | Stream editor |
tr | Translate or delete characters |
sort | Sort lines |
uniq | Remove duplicate lines |
cut | Extract fields/columns |
join | Join lines on a common field |
paste | Merge lines side by side |
column | Columnate output |
fold | Wrap lines to a specified width |
expand | Convert tabs to spaces |
unexpand | Convert spaces to tabs |
fmt | Simple text formatter |
rev | Reverse lines |
tac | Concatenate and print in reverse |
look | Display lines beginning with a given string |
shuf | Shuffle lines |
tsort | Topological sort |
comm | Compare two sorted files |
cmp | Compare two files byte by byte |
diff | Compare files line by line |
patch | Apply a diff patch |
iconv | Character encoding conversion |
File Operations
| Command | Notes |
|---|
cp | Copy files |
mv | Move/rename files |
rm | Remove files |
mkdir | Create directories |
touch | Create empty file or update timestamp |
ln | Create symbolic links |
tee | Write stdin to file and stdout |
mktemp | Create temporary file |
split | Split file into pieces |
csplit | Split file by context |
Path Utilities
| Command | Notes |
|---|
basename | Strip directory from path |
dirname | Strip filename from path |
realpath | Resolve path |
readlink | Print symbolic link target |
ls | List directory contents |
Compression
| Command | Notes |
|---|
gzip | Compress files |
gunzip | Decompress gzip files |
zip | Create zip archives |
unzip | Extract zip archives |
tar | Archive files |
zcat | Cat compressed files |
zgrep | Grep compressed files |
Encoding
| Command | Notes |
|---|
base64 | Base64 encode/decode |
| Format | Extension | Variants |
|---|
| Parquet | .parquet | cat, head, tail, wc, stat, cut, grep, ls, file |
| Feather | .feather | cat, head, tail, wc, stat, cut, grep, ls, file |
| ORC | .orc | cat, head, tail, wc, stat, cut, grep, ls, file |
| HDF5 | .hdf5 | cat, head, tail, wc, stat, cut, grep, ls, file |
Audio Support (Optional)
Audio commands are opt-in and require sherpa-onnx with a Whisper model.
They transcribe audio to text, enabling cat, head, tail, grep,
and stat on audio files.
| Format | Extension | Commands |
|---|
| WAV | .wav | cat, head, tail, grep, stat |
| MP3 | .mp3 | cat, head, tail, grep, stat |
| OGG | .ogg | cat, head, tail, grep, stat |
from mirage.commands.audio import AUDIO_COMMANDS
from mirage.commands.audio.utils import configure
configure(model_dir="path/to/sherpa-onnx-whisper-base")
for cmd in AUDIO_COMMANDS:
ws.register(cmd)
Use Cases
- AI agents accessing cloud data: Mount S3 buckets for agents to read and process remote datasets
- Data pipelines: Read and write S3 objects with shell-like commands
- Sandboxed cloud storage access: Restrict agent operations to a specific bucket and prefix
- FUSE mounting: Expose S3 buckets through a virtual FUSE mount for external tools
Home
Python
TS
