> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mirage.strukto.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# HF Datasets

> Virtualize a Hugging Face dataset repository from Node.

`HfDatasetsVFS` mounts one [Hugging Face dataset](https://huggingface.co/datasets)
repository, read-only: every file in the repo becomes a file, read lazily, so
only the bytes a command reads are transferred. Set up the tokens for gated and private datasets as in [HF Datasets setup](https://docs.mirage.strukto.ai/home/setup/hf_datasets.md).

## Install

```bash theme={null}
npm install @struktoai/mirage-node
```

## Configure

```ts theme={null}
import { HfDatasetsVFS, MountMode, Workspace } from '@struktoai/mirage-node'

const dataset = new HfDatasetsVFS({
  repoId: 'AlienKevin/SWE-ZERO-12M-trajectories',
  token: process.env.HF_TOKEN,
})
const ws = new Workspace({ '/data': dataset }, { mode: MountMode.READ })

const result = await ws.shell("ls /data && find /data -name '*.parquet' | head -n 5")
console.log(result.stdoutText)
await ws.close()
```

`revision` pins a branch, tag or commit (default `main`), and `keyPrefix`
mounts one folder of the repo. `endpoint` points at a mirror or a self-hosted
Hub.

### Fields

**repoId** `string` *required*

Repository id as `namespace/name`, or a bare `name` that resolves against the token's owner.

**token** `string`

Hugging Face access token; unset sends anonymous requests, which reach public repositories only.

**endpoint** `string` (default: '[https://huggingface.co](https://huggingface.co)')

Hub origin requests go to, for a mirror or a self-hosted Hub.

**timeout** `number` (default: DEFAULT\_TIMEOUT)

Seconds a request may wait to connect, or between two reads of its answer, before it fails; a download that keeps flowing is never cut off.

**keyPrefix** `string`

Path inside the repository that becomes the mount root; files outside it are not listed.

**revision** `string`

Branch, tag or commit the mount reads; unset reads `main`.

**expandCommits** `boolean`

Fetch each file's last commit so listings carry modification times, at 50 entries per request instead of 1000. Unset does so only when the whole repository fits in one page.

Fields marked secret are redacted in snapshots.

## Authentication

A public dataset needs no token. A gated or private one needs `token`: a `read`
token, or a fine-grained token with read access to that dataset; a gated
dataset also needs its conditions accepted on the Hub. A repository or revision
the token cannot see, or that does not exist, answers `Permission denied`.

## Layout

```text theme={null}
/data/
  README.md
  assets/
    marin8b_swe_bench_scaling.png
  data/
    train-00000.parquet
    train-00001.parquet
```

Files keep their repo paths: `/data/data/train-00000.parquet` is
`data/train-00000.parquet` in the repo. `repoId` is `namespace/name`, or a
bare name under the token's owner.

## Behavior

* **Read-only:** a Hub write is a commit, and a file write cannot say where a
  commit ends, so `echo >`, `rm`, `cp` and `mv` are refused rather than making
  one commit per file. The [`hf` CLI](https://docs.mirage.strukto.ai/typescript/cli/hf) is the write half:
  `hf download --local-dir` puts a copy on a writable mount, and `hf upload`
  sends it back as one commit.
* **Lazy reads:** `ls` and `stat` never download a file, and `head` on a large
  shard reads only its first bytes.
* **Fresh reads:** under `read: fresh`, each command checks once which commit
  `revision` points at, even when `revision` is a full commit sha, since a
  branch or tag can carry the same name.

## Integration

The Hub answers this command through its own API, so fewer files are read. See
[Integration](https://docs.mirage.strukto.ai/home/integration/overview.md).

* `sha256sum`: the LFS sha256 the Hub keeps; other files are read.

## VFS.md

The [VFS.md](https://docs.mirage.strukto.ai/home/setup/vfs-md.md) an agent reads with HF Datasets mounted at `/<path>`.

> [tab] Rendered
>
> **Virtual Filesystem**
>
> | Path | Description |
> | - | - |
> | `/<path>` | read |
>
> **Supported Unix Commands for VFS**
>
> Unix commands work on each path as on a local filesystem, as far as the path supports them:
>
> | Path | Supported |
> | - | - |
> | `/<path>` | Commands that read files (`ls`, `cat`, `grep`, `find`, `jq` and the rest). |
>
> **Limitations of Unix Commands for VFS**
>
> Where a path differs from a local filesystem:
>
> | Path | Limitations |
> | - | - |
> | `/<path>` | Read only: no command changes files. `grep -r`, `find` and `du` stop after 1000 files; a glob matching more, `tar -c`, `cp -r` and `rm -r` are refused. |
>
> Run `man <cmd>` or `<cmd> --help` for one command's usage.

> [tab] Raw
>
> ```markdown theme={null}
> ---
> name: workspace-filesystem
> description: The virtual filesystem layout of this workspace.
> ---
>
> # Virtual Filesystem
>
> | Path | Description |
> | --- | --- |
> | `/<path>` | read |
>
> ## Supported Unix Commands for VFS
>
> Unix commands work on each path as on a local filesystem, as far as the path supports them:
>
> | Path | Supported |
> | --- | --- |
> | `/<path>` | Commands that read files (`ls`, `cat`, `grep`, `find`, `jq` and the rest). |
>
> ## Limitations of Unix Commands for VFS
>
> Where a path differs from a local filesystem:
>
> | Path | Limitations |
> | --- | --- |
> | `/<path>` | Read only: no command changes files. `grep -r`, `find` and `du` stop after 1000 files; a glob matching more, `tar -c`, `cp -r` and `rm -r` are refused. |
>
> Run `man <cmd>` or `<cmd> --help` for one command's usage.
> ```


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.