GCSVFS mounts one Google Cloud Storage bucket: every object becomes a file and
every / in a key a folder. It reaches GCS through the S3-compatible XML API at
https://storage.googleapis.com, signed with an HMAC key. Set up credentials as in GCS setup.
Install
Configure
endpoint defaults to https://storage.googleapis.com and region to auto:
one host serves every region. A mount takes MountMode.READ, WRITE or EXEC.
Fields
Named profile in the shared AWS config and credentials files that supplies credentials when no access key pair is given.
Address the bucket in the URL path (
<endpoint>/<bucket>/<key>) instead of as a subdomain of the endpoint.Key prefix inside the bucket that the mount treats as its root; keys outside it are not visible. Leading slashes are dropped and a trailing
/ is added.Seconds a request may wait to connect, or between two reads of its answer, before it fails; a download that keeps flowing is never cut off.
Authentication
Credentials come fromaccessKeyId and secretAccessKey, a service account’s
HMAC key, with sessionToken for temporary credentials. Without them, the AWS
SDK reads its own chain: environment variables, ~/.aws/credentials, then the
instance or container role.
The service account needs Storage Object Viewer on the bucket, or Storage
Object User (or Storage Object Admin) to write. A new key can take up to a
minute to work. A request the credentials may not make answers
Permission denied.
Layout
/gcs/data/file.txt is the key data/file.txt. Folders come from the / in
keys; GCS stores no folder objects.
Behavior
- Key prefix:
keyPrefix: 'users/42/'mounts only that part of the bucket, so/gcs/notes.mdis the keyusers/42/notes.md.
Browser
The browserGCSVFS from @struktoai/mirage-browser keeps credentials off the
page: a server signs a URL for each S3 call with the GCS keys, and the browser
fetches it.
- The bucket’s CORS rules must allow the page’s origin, exactly as served.
- A browser mount cannot declare
write: conditional, and any HTTP 403 from a signed URL, an expired one included, answersPermission denied. keyPrefixset in the browser is no security boundary: enforce the prefix in the signing server.endpointonly names the mount’s storage location: requests go to the host each signed URL names.
Signing server
Signing server
An Express route that signs URLs:Google’s own signer (
getSignedUrl in @google-cloud/storage) works too: the
mount only needs a URL that fetch() accepts, and native signing needs no HMAC
key on the server.Sign only the operations the app needs: a viewer needs GET, HEAD and LIST.Bucket CORS rules
Bucket CORS rules
GCS refuses the S3 GCS lets every request header through, so the rules name none.
PutBucketCors call, since its CORS schema differs. Set the
rules with gsutil:Browser fields
Browser fields
Function
(path, operation, options) that returns a signed URL for each S3 operation; your server signs it, so the browser holds no keys.Endpoint URL of the GCS XML API. Requests go to the host each presigned URL names; the mount uses it to name its storage location.
Key prefix the mount treats as its root. The browser can change it before a URL is signed, so enforce the prefix on the signing server.
Seconds a request may wait to connect, or between two reads of its answer, before it fails; a download that keeps flowing is never cut off.
Integration
GCS answers these commands through its own API, so fewer objects are read. See Integration.md5sum: the object’s ETag, when it is the MD5 (a single-part upload, unencrypted or under SSE-S3).find: one recursive listing.du: one recursive listing.ls -Randtree: one recursive listing.
VFS.md
The VFS.md an agent reads with GCS mounted at/<path>.
<mode> is the mount’s mode: read, write or exec. The commands shown are
those of a write mount.
- Rendered
- Raw
Virtual Filesystem
Supported Unix Commands for VFSUnix commands work on each path as on a local filesystem, as far as the path supports them:
Limitations of Unix Commands for VFSWhere a path differs from a local filesystem:
Run
man <cmd> or <cmd> --help for one command’s usage.