HfDatasetsVFS mounts one Hugging Face dataset
repository, read-only: every file in the repo becomes a file, read lazily, so
only the bytes a command reads are transferred. Set up the tokens for gated and private datasets as in HF Datasets setup.
Install
Configure
revision pins a branch, tag or commit (default main), and keyPrefix
mounts one folder of the repo. endpoint points at a mirror or a self-hosted
Hub.
Fields
Repository id as
namespace/name, or a bare name that resolves against the token’s owner.Hugging Face access token; unset sends anonymous requests, which reach public repositories only.
Hub origin requests go to, for a mirror or a self-hosted Hub.
Seconds a request may wait to connect, or between two reads of its answer, before it fails; a download that keeps flowing is never cut off.
Fetch each file’s last commit so listings carry modification times, at 50 entries per request instead of 1000. Unset does so only when the whole repository fits in one page.
Authentication
A public dataset needs no token. A gated or private one needstoken: a read
token, or a fine-grained token with read access to that dataset; a gated
dataset also needs its conditions accepted on the Hub. A repository or revision
the token cannot see, or that does not exist, answers Permission denied.
Layout
/data/data/train-00000.parquet is
data/train-00000.parquet in the repo. repoId is namespace/name, or a
bare name under the token’s owner.
Behavior
- Read-only: a Hub write is a commit, and a file write cannot say where a
commit ends, so
echo >,rm,cpandmvare refused rather than making one commit per file. ThehfCLI is the write half:hf download --local-dirputs a copy on a writable mount, andhf uploadsends it back as one commit. - Lazy reads:
lsandstatnever download a file, andheadon a large shard reads only its first bytes. - Fresh reads: under
read: fresh, each command checks once which commitrevisionpoints at, even whenrevisionis a full commit sha, since a branch or tag can carry the same name.
Integration
The Hub answers this command through its own API, so fewer files are read. See Integration.sha256sum: the LFS sha256 the Hub keeps; other files are read.
VFS.md
The VFS.md an agent reads with HF Datasets mounted at/<path>.
- Rendered
- Raw
Virtual Filesystem
Supported Unix Commands for VFSUnix commands work on each path as on a local filesystem, as far as the path supports them:
Limitations of Unix Commands for VFSWhere a path differs from a local filesystem:
Run
man <cmd> or <cmd> --help for one command’s usage.