> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mirage.strukto.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# HF Buckets

> Virtualize a Hugging Face Storage Bucket from Node.

`HfBucketsVFS` mounts one
[Hugging Face Storage Bucket](https://huggingface.co/docs/hub/storage-buckets):
every file in it becomes a file and every `/` in its path a folder. Set up the bucket and token as in [HF Buckets setup](https://docs.mirage.strukto.ai/home/setup/hf_buckets.md).

## Install

```bash theme={null}
npm install @struktoai/mirage-node opendal
```

`opendal` is a native binding, so the mount runs in Node only, not in the
browser.

## Configure

```ts theme={null}
import { HfBucketsVFS, MountMode, Workspace } from '@struktoai/mirage-node'

const hf = new HfBucketsVFS({
  bucket: process.env.HF_BUCKET_NAME!,
  token: process.env.HF_TOKEN!,
})
const ws = new Workspace({ '/hf': hf }, { mode: MountMode.WRITE })

const result = await ws.shell("ls /hf && find /hf -name '*.json'")
console.log(result.stdoutText)
await ws.close()
```

`bucket` is the bucket's id, `<owner>/<name>`. `endpoint` points the mount at
a mirror or a self-hosted Hub. A mount takes `MountMode.READ`, `WRITE` or
`EXEC`.

### Fields

**bucket** `string` *required*

Bucket id in `namespace/name` form.

**token** `string`

Hugging Face access token.

**endpoint** `string` (default: '[https://huggingface.co](https://huggingface.co)')

Hub origin requests go to, for a mirror or a self-hosted Hub.

**timeout** `number` (default: DEFAULT\_TIMEOUT)

Seconds a request may wait to connect, or between two reads of its answer, before it fails; a download that keeps flowing is never cut off.

**keyPrefix** `string`

Path inside the bucket that becomes the mount root; files outside it are not listed.

Fields marked secret are redacted in snapshots.

## Authentication

`token` is a Hugging Face access token: **Read** for a read mount, **Write** to
upload. A fine-grained token needs read access, or write access to upload, on
the bucket's namespace. A bucket the token cannot reach, or one that does not
exist, answers `Permission denied`.

## Layout

```text theme={null}
/hf/
  checkpoints/
    step-1000/
      config.json
    step-2000/
      config.json
  logs/
    train.log
```

`/hf/logs/train.log` is the file `logs/train.log` in the bucket. Folders come
from the `/` in file paths.

## Behavior

* **Key prefix:** `keyPrefix: 'users/42/'` mounts only that part of the bucket, so
  `/hf/notes.md` is the file `users/42/notes.md`. A leading `/` is dropped and
  a trailing `/` added.
* **Folders:** a folder exists only while it holds a file, so `mkdir` creates
  nothing.
* **No move:** `mv` and `rmdir` are not supported.
* **No history:** a bucket keeps no history, so a removed file is gone for
  good.

## Integration

HF Buckets answers these commands through its own API, so fewer files are read.
See [Integration](https://docs.mirage.strukto.ai/home/integration/overview.md).

* `find`: one recursive listing.
* `du`: one recursive listing.
* `ls -R` and `tree`: one recursive listing.

## VFS.md

The [VFS.md](https://docs.mirage.strukto.ai/home/setup/vfs-md.md) an agent reads with HF Buckets mounted at `/<path>`.
`<mode>` is the mount's mode: `read`, `write` or `exec`. The commands shown are
those of a `write` mount.

> [tab] Rendered
>
> **Virtual Filesystem**
>
> | Path | Description |
> | - | - |
> | `/<path>` | `<mode>` |
>
> **Supported Unix Commands for VFS**
>
> Unix commands work on each path as on a local filesystem, as far as the path supports them:
>
> | Path | Supported |
> | - | - |
> | `/<path>` | Almost all Unix commands. |
>
> **Limitations of Unix Commands for VFS**
>
> Where a path differs from a local filesystem:
>
> | Path | Limitations |
> | - | - |
> | `/<path>` | Not supported: `mv`, `rmdir`. `grep -r`, `find` and `du` stop after 1000 files; a glob matching more, `tar -c`, `cp -r` and `rm -r` are refused. |
>
> Run `man <cmd>` or `<cmd> --help` for one command's usage.

> [tab] Raw
>
> ```markdown theme={null}
> ---
> name: workspace-filesystem
> description: The virtual filesystem layout of this workspace.
> ---
>
> # Virtual Filesystem
>
> | Path | Description |
> | --- | --- |
> | `/<path>` | `<mode>` |
>
> ## Supported Unix Commands for VFS
>
> Unix commands work on each path as on a local filesystem, as far as the path supports them:
>
> | Path | Supported |
> | --- | --- |
> | `/<path>` | Almost all Unix commands. |
>
> ## Limitations of Unix Commands for VFS
>
> Where a path differs from a local filesystem:
>
> | Path | Limitations |
> | --- | --- |
> | `/<path>` | Not supported: `mv`, `rmdir`. `grep -r`, `find` and `du` stop after 1000 files; a glob matching more, `tar -c`, `cp -r` and `rm -r` are refused. |
>
> Run `man <cmd>` or `<cmd> --help` for one command's usage.
> ```


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.