HF Buckets

The HF Buckets VFS mounts a Hugging Face Bucket at some prefix such as /hf/. Listing and writes use OpenDAL (the hf-buckets extra). Reads stream from the Hub. For credential setup, see HF Buckets Setup.

Install

uv add "mirage-ai[hf-buckets]"

Config

import os

from mirage import MountMode, Workspace
from mirage.vfs.hf_buckets import HfBucketsConfig, HfBucketsVFS

config = HfBucketsConfig(
    bucket=os.environ["HF_BUCKET_NAME"],  # "namespace/bucket-name"
    token=os.environ["HF_TOKEN"],
    # Optional:
    # endpoint="https://huggingface.co",
    # timeout=30,
    # key_prefix="data/",
)
vfs = HfBucketsVFS(config)
ws = Workspace({"/hf": vfs}, mode=MountMode.READ)

HfBucketsVFS(config) takes an HfBucketsConfig object with the bucket in namespace/bucket-name form plus an optional access token. timeout is how many seconds one Hub request may go without progress: connecting, waiting for the answer, or between two chunks of a download. A large file that keeps arriving is never cut off, and 0 turns the bound off. Both READ and WRITE modes are supported out of the box.

Field	Description
`timeout` default `30`	Seconds a request may wait to connect, or between two reads of its answer, before it fails; a download that keeps flowing is never cut off.
`bucket` required	Bucket id in `namespace/name` form.
`token` secret	Hugging Face access token.
`endpoint` default `'https://huggingface.co'`	Hub origin requests go to, for a mirror or a self-hosted Hub.
`key_prefix`	Path inside the bucket that becomes the mount root; files outside it are not listed.

Fields marked secret are redacted in snapshots.

VFS.md

The VFS.md an agent reads with HF Buckets mounted at /<path>.

Rendered
Raw

Virtual Filesystem

Path	Description
`/<path>`	`hf_buckets`, write

Supported Unix Commands for VFS

Unix commands work on each path as on a local filesystem, as far as the path supports them:

Path	Supported
`/<path>`	Almost all Unix commands.

Limitations of Unix Commands for VFS

Where a path differs from a local filesystem:

Path	Limitations
`/<path>`	Not supported: `mv`, `rmdir`. `grep -r`, `find` and `du` stop after 1000 files; a glob matching more, `tar -c`, `cp -r` and `rm -r` are refused.

Run man <cmd> or <cmd> --help for one command’s usage.

---
name: workspace-filesystem
description: The virtual filesystem layout of this workspace.
---

# Virtual Filesystem

| Path | Description |
| --- | --- |
| `/<path>` | `hf_buckets`, write |

## Supported Unix Commands for VFS

Unix commands work on each path as on a local filesystem, as far as the path supports them:

| Path | Supported |
| --- | --- |
| `/<path>` | Almost all Unix commands. |

## Limitations of Unix Commands for VFS

Where a path differs from a local filesystem:

| Path | Limitations |
| --- | --- |
| `/<path>` | Not supported: `mv`, `rmdir`. `grep -r`, `find` and `du` stop after 1000 files; a glob matching more, `tar -c`, `cp -r` and `rm -r` are refused. |

Run `man <cmd>` or `<cmd> --help` for one command's usage.

The HF Buckets VFS maps bucket object keys to virtual paths under the mount prefix. For example, if bucket your-user/my-data contains:

data/file.txt
data/config.json
reports/q1.csv
reports/q2.csv

Then mounting at /hf/ exposes:

/hf/
  data/
    file.txt
    config.json
  reports/
    q1.csv
    q2.csv

Path mapping: virtual /hf/data/file.txt maps to bucket key data/file.txt.

Cache

The HF Buckets VFS uses IndexCacheStore with index_ttl = 600 (10 minutes). Directory listings are cached and populate file-size/type entries that stat reads via a fast path, so a readdir followed by per-entry stat calls (which is what ls, FUSE getattr, and most shell commands trigger) costs one HTTP request instead of N. A configured index: replaces this lifetime with its own ttl, and the mount’s ttl: caps it.

Integration

What Hugging Face buckets answers through its own API, so fewer files are read. Integration explains each command.

Command	On Hugging Face buckets
`find`, `du`, `ls -R`, `tree`	One recursive listing

Example

import asyncio
import os

from dotenv import load_dotenv

from mirage import MountMode, Workspace
from mirage.vfs.hf_buckets import HfBucketsConfig, HfBucketsVFS

load_dotenv(".env.development")

config = HfBucketsConfig(
    bucket=os.environ["HF_BUCKET_NAME"],
    token=os.environ["HF_TOKEN"],
)

vfs = HfBucketsVFS(config)


async def main() -> None:
    ws = Workspace({"/hf/": vfs}, mode=MountMode.READ)

    r = await ws.shell("ls /hf/")
    print(await r.stdout_str())

    r = await ws.shell("cat /hf/data/file.txt")
    print(await r.stdout_str())

    r = await ws.shell("tree /hf/")
    print(await r.stdout_str())

    r = await ws.shell("find /hf/ -name '*.json'")
    print(await r.stdout_str())

    r = await ws.shell("grep example /hf/data/config.json")
    print(await r.stdout_str())

    r = await ws.shell("stat /hf/data/file.txt")
    print(await r.stdout_str())


if __name__ == "__main__":
    asyncio.run(main())

Shell Commands

The HF Buckets VFS supports the full set of shell commands since it operates on real file content (text, binary, JSON, CSV, etc.). Large files benefit from range reads to avoid downloading entire objects.

Read Commands

Command	Notes
`cat`	Read file content
`head` / `tail`	First/last N lines
`grep` / `rg`	Pattern search (file or directory level)
`jq`	Query JSON fields
`wc`	Line/word/byte counts
`stat`	File metadata (name, size, type, modified)
`find`	Recursive search with `-name`, `-maxdepth`
`tree`	Directory tree view
`nl`	Number lines
`du`	Disk usage summary
`file`	Detect file type
`strings`	Extract printable strings from binary
`xxd`	Hex dump
`md5`	MD5 checksum
`sha256sum`	SHA-256 checksum

Text Processing

Command	Notes
`awk`	Pattern scanning and processing
`sed`	Stream editor
`tr`	Translate or delete characters
`sort`	Sort lines
`uniq`	Remove duplicate lines
`cut`	Extract fields/columns
`join`	Join lines on a common field
`paste`	Merge lines side by side
`column`	Columnate output
`fold`	Wrap lines to a specified width
`expand`	Convert tabs to spaces
`unexpand`	Convert spaces to tabs
`fmt`	Simple text formatter
`rev`	Reverse lines
`tac`	Concatenate and print in reverse
`look`	Display lines beginning with a given string
`shuf`	Shuffle lines
`tsort`	Topological sort
`comm`	Compare two sorted files
`cmp`	Compare two files byte by byte
`diff`	Compare files line by line
`iconv`	Character encoding conversion

File Operations

Command	Notes
`rm`	Remove files; `-r` removes a subtree
`mkdir`	No-op: a directory exists only while it holds a file
`touch`	Create empty file or update timestamp
`mktemp`	Create temporary file
`split`	Split file into pieces
`csplit`	Split file by context

Path Utilities

Command	Notes
`basename`	Strip directory from path
`dirname`	Strip filename from path
`realpath`	Resolve path
`readlink`	Print symbolic link target
`ls`	List directory contents

Compression

Command	Notes
`gzip`	Compress files
`gunzip`	Decompress gzip files
`zip`	Create zip archives
`unzip`	Extract zip archives
`tar`	Archive files
`zcat`	Cat compressed files
`zgrep`	Grep compressed files

Encoding

Command	Notes
`base64`	Base64 encode/decode

Use Cases

AI agents accessing HF datasets: Mount HF Buckets for agents to read and process datasets stored on the Hub
Data pipelines: Read and write HF bucket objects with shell-like commands
Sandboxed bucket access: Restrict agent operations to a specific bucket and prefix
FUSE mounting: Expose HF Buckets through a virtual FUSE mount for external tools

Scoping a VFS to a key prefix

Pass key_prefix: str | None = None to HfBucketsConfig to transparently scope every operation to a subpath of the bucket:

HfBucketsVFS(HfBucketsConfig(
    bucket="your-user/app-data",
    token=hf_token,
    key_prefix=f"users/{user_id}/",
))

When set, every read/write/list/stat operation is transparently scoped to that bucket subpath. Agents see clean paths like /data/notes.md; the underlying bucket key is users/{user_id}/data/notes.md. Useful for multi-tenant systems. Normalization: leading slashes are stripped and a trailing slash is added automatically. Both None and an empty string are treated as “no prefix.”

Getting Started

Access

AI Agents

VFS

CLIs

Runtimes

Install

Config

VFS.md

Virtual Filesystem

Supported Unix Commands for VFS

Limitations of Unix Commands for VFS

Cache

Integration

Example

Shell Commands

Read Commands

Text Processing

File Operations

Path Utilities

Compression

Encoding

Use Cases

Scoping a VFS to a key prefix

​Install

​Config

​VFS.md

​Virtual Filesystem

​Supported Unix Commands for VFS

​Limitations of Unix Commands for VFS

​Cache

​Integration

​Example

​Shell Commands

​Read Commands

​Text Processing

​File Operations

​Path Utilities

​Compression

​Encoding

​Use Cases

​Scoping a VFS to a key prefix

Install

Config

VFS.md

Virtual Filesystem

Supported Unix Commands for VFS

Limitations of Unix Commands for VFS

Cache

Integration

Example

Shell Commands

Read Commands

Text Processing

File Operations

Path Utilities

Compression

Encoding

Use Cases

Scoping a VFS to a key prefix