HF Datasets

HfDatasetsResource mounts a Hugging Face Dataset repo at a prefix such as /ds/. All reads are lazy: only the bytes you actually cat/head get transferred. The TypeScript backend mirrors the Python one and returns identical results. For credential setup, see HF Datasets Setup.

Node only: the resource ships in @struktoai/mirage-node. It needs no native binding, though, because it speaks the Hub API over the runtime’s own fetch. The opendal binding belongs to HF Buckets, which is a different product.

Node

pnpm add @struktoai/mirage-node

import { HfDatasetsResource, MountMode, Workspace } from '@struktoai/mirage-node'

const dataset = new HfDatasetsResource({
  repoId: 'AlienKevin/SWE-ZERO-12M-trajectories', // "namespace/dataset-name"
  token: process.env.HF_TOKEN, // optional for public datasets
  // Optional:
  // endpoint: 'https://huggingface.co',
  // revision: 'main',
  // keyPrefix: 'train/',
})

const ws = new Workspace({ '/ds/': dataset }, { mode: MountMode.READ })

console.log((await ws.execute('ls -lh /ds/')).stdoutText)
console.log((await ws.execute("find /ds/ -name '*.parquet' | wc -l")).stdoutText)
console.log((await ws.execute('cat /ds/README.md | head -n 20')).stdoutText)

HfDatasetsResource takes repoId in namespace/dataset-name form plus an optional access token. Public datasets need no token.

Reading, not writing

This mount is read-only, the way a github mount is. A Hub write is a commit, and a POSIX write cannot say where a commit ends, so echo >, rm, cp and mv are refused here rather than silently making one commit per file. The hf CLI is the write half: hf download --local-dir puts a copy on a ram or disk mount, which is an ordinary writable filesystem, and hf upload sends it back as a single commit.

Filesystem Layout

Maps dataset repo files (README, parquet shards, etc.) to virtual paths under the mount prefix. Parquet shards stream lazily via byte-range reads.

Shell Commands

Every read command in HF Buckets’ set works here, as do the text processing and path utilities, which only read. What does not is the File Operations group: this mount is read-only, so rm and touch are refused, as is any command asked to write into it.

Getting Started

AI Agents

Resources

CLIs

Watch

Runtimes

Node

Reading, not writing

Filesystem Layout

Shell Commands

​Node

​Reading, not writing

​Filesystem Layout

​Shell Commands

Node

Reading, not writing

Filesystem Layout

Shell Commands