> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mirage.strukto.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# S3

> Virtualize an Amazon S3 or S3-compatible bucket from Node or the browser.

`S3VFS` mounts one bucket: every object becomes a file and every `/` in a key a
folder. It works with Amazon S3 and any S3-compatible server, such as MinIO,
Ceph or SeaweedFS. Set up credentials as in [S3 setup](https://docs.mirage.strukto.ai/home/setup/s3.md).

## Install

```bash theme={null}
npm install @struktoai/mirage-node
```

## Configure

```ts theme={null}
import { MountMode, S3VFS, Workspace } from '@struktoai/mirage-node'

const s3 = new S3VFS({
  bucket: process.env.AWS_S3_BUCKET!,
  region: process.env.AWS_DEFAULT_REGION!,
  accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
  secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
})
const ws = new Workspace({ '/s3': s3 }, { mode: MountMode.WRITE })

const result = await ws.shell("ls /s3 && find /s3 -name '*.csv'")
console.log(result.stdoutText)
await ws.close()
```

For MinIO or another S3-compatible server, add `endpoint` and, if the server
needs it, `forcePathStyle: true`. A mount takes `MountMode.READ`, `WRITE` or
`EXEC`.

### Fields

**bucket** `string` *required*

Name of the bucket to mount.

**region** `string`

AWS region requests are signed for and sent to. Unset leaves it to the AWS SDK's environment variables and profile.

**endpoint** `string`

Endpoint URL of an S3-compatible server. Unset, the node mount uses `AWS_ENDPOINT_URL_S3` or `AWS_ENDPOINT_URL` when set, else AWS's endpoint for `region`; an endpoint set only in an AWS profile is not read.

**accessKeyId** `string`

Access key ID; set it with `secretAccessKey`. Unset uses the AWS SDK credential chain.

**secretAccessKey** `string`

Secret access key paired with `accessKeyId`.

**sessionToken** `string`

Session token for temporary credentials, sent with the access key pair.

**forcePathStyle** `boolean`

Address the bucket in the URL path (`<endpoint>/<bucket>/<key>`) instead of as a subdomain of the endpoint.

**timeout** `number` (default: DEFAULT\_TIMEOUT)

Seconds a request may wait to connect, or between two reads of its answer, before it fails; a download that keeps flowing is never cut off.

**presignedUrlProvider** `function`

Function `(path, operation, options)` that returns a signed URL for each S3 operation. When set, the mount fetches those URLs and needs no credentials.

**httpAgentProvider** `function`

Function that returns the `httpAgent` and `httpsAgent` for one S3 client, called once per client. The node mount builds it from `proxy` when that is set.

**defaultContentType** `string`

`Content-Type` stored with every object the mount writes. Unset sends none, so the store applies its own default; a write through `presignedUrlProvider` sends `application/octet-stream`.

**keyPrefix** `string`

Key prefix inside the bucket that the mount treats as its root; keys outside it are not visible. Leading slashes are dropped and a trailing `/` is added.

**profile** `string`

Named profile in the shared AWS config and credentials files that supplies credentials when no access key pair is given.

**proxy** `string`

Proxy URL for every HTTP and HTTPS request to the endpoint.

Fields marked secret are redacted in snapshots.

## Authentication

Credentials come from `accessKeyId` and `secretAccessKey`, with `sessionToken`
for temporary credentials. Without them, the AWS SDK reads its own chain:
environment variables, `~/.aws/credentials`, then the instance or container
role.

The mount needs `s3:ListBucket` on the bucket, `s3:GetObject` to read, and
`s3:PutObject` and `s3:DeleteObject` to write. An object the credentials cannot
read answers `Permission denied`. Without `s3:ListBucket`, a missing file
answers `Permission denied` too, since S3 then hides which keys exist.

## Layout

```text theme={null}
/s3/
  data/
    config.json
    file.txt
  reports/
    q1.csv
    q2.csv
```

`/s3/data/file.txt` is the key `data/file.txt`. Folders come from the `/` in
keys; S3 stores no folder objects.

## Behavior

* **Key prefix:** `keyPrefix: 'users/42/'` mounts only that part of the bucket,
  so `/s3/notes.md` is the key `users/42/notes.md`.

## Browser

The browser `S3VFS` from `@struktoai/mirage-browser` keeps credentials off the
page: a server signs a URL for each S3 call, and the browser fetches it.

```bash theme={null}
npm install @struktoai/mirage-browser
```

```ts theme={null}
import { MountMode, S3VFS, Workspace } from '@struktoai/mirage-browser'

const s3 = new S3VFS({
  bucket: 'my-bucket',
  presignedUrlProvider: async (path, op, opts) => {
    const r = await fetch('/presign', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ path, op, opts }),
    })
    return ((await r.json()) as { url: string }).url
  },
})
const ws = new Workspace({ '/s3': s3 }, { mode: MountMode.READ })
```

* The bucket's CORS rules must allow the page's origin, exactly as served.
* A browser mount cannot declare `write: conditional`, and any HTTP 403 from a
  signed URL, an expired one included, answers `Permission denied`.
* `keyPrefix` set in the browser is no security boundary: enforce the prefix in
  the signing server and its IAM policy.

> [accordion] Signing server
>
> An Express route that signs URLs:
>
> ```ts theme={null}
> import {
>   CopyObjectCommand,
>   DeleteObjectCommand,
>   GetObjectCommand,
>   HeadObjectCommand,
>   ListObjectsV2Command,
>   PutObjectCommand,
>   S3Client,
> } from '@aws-sdk/client-s3'
> import { getSignedUrl } from '@aws-sdk/s3-request-presigner'
>
> const client = new S3Client({ region: process.env.AWS_DEFAULT_REGION })
> const BUCKET = process.env.AWS_S3_BUCKET!
>
> app.post('/presign', async (req, res) => {
>   const { path, op, opts } = req.body as {
>     path: string
>     op: 'GET' | 'PUT' | 'HEAD' | 'DELETE' | 'LIST' | 'COPY'
>     opts?: Record<string, unknown>
>   }
>   const key = path.replace(/^\/+/, '')
>   const ttl = typeof opts?.ttlSec === 'number' ? opts.ttlSec : 300
>   let cmd
>   switch (op) {
>     case 'GET':    cmd = new GetObjectCommand({ Bucket: BUCKET, Key: key }); break
>     case 'PUT':    cmd = new PutObjectCommand({ Bucket: BUCKET, Key: key, ContentType: opts?.contentType as string | undefined }); break
>     case 'HEAD':   cmd = new HeadObjectCommand({ Bucket: BUCKET, Key: key }); break
>     case 'DELETE': cmd = new DeleteObjectCommand({ Bucket: BUCKET, Key: key }); break
>     case 'LIST':   cmd = new ListObjectsV2Command({
>       Bucket: BUCKET,
>       Prefix: opts?.listPrefix as string | undefined,
>       Delimiter: opts?.listDelimiter as string | undefined,
>       ContinuationToken: opts?.listContinuationToken as string | undefined,
>     }); break
>     case 'COPY':   cmd = new CopyObjectCommand({
>       Bucket: BUCKET,
>       Key: key,
>       CopySource: `${BUCKET}/${opts?.copySource as string}`,
>     }); break
>   }
>   const url = await getSignedUrl(client, cmd, { expiresIn: ttl })
>   res.json({ url })
> })
> ```
>
> Sign only the operations the app needs: a viewer needs `GET`, `HEAD` and `LIST`.

> [accordion] Bucket CORS rules
>
> ```json theme={null}
> [
>   {
>     "AllowedOrigins": ["http://localhost:5173", "https://app.example.com"],
>     "AllowedMethods": ["GET", "PUT", "HEAD", "DELETE", "POST"],
>     "AllowedHeaders": ["*"],
>     "ExposeHeaders": ["ETag", "Content-Length", "Content-Type", "Last-Modified"],
>     "MaxAgeSeconds": 3000
>   }
> ]
> ```
>
> ```bash theme={null}
> aws s3api put-bucket-cors --bucket $AWS_S3_BUCKET --cors-configuration file://cors.json
> ```
>
> ```bash theme={null}
> aws s3api put-bucket-cors --bucket $AWS_S3_BUCKET --cors-configuration file://cors.json
> ```

> [accordion] Browser fields
>
> **bucket** `string` *required*
>
> Name of the bucket to mount.
>
> **region** `string`
>
> AWS region requests are signed for and sent to. Unset leaves it to the AWS SDK's environment variables and profile.
>
> **endpoint** `string`
>
> Endpoint URL of an S3-compatible server. Unset, the node mount uses `AWS_ENDPOINT_URL_S3` or `AWS_ENDPOINT_URL` when set, else AWS's endpoint for `region`; an endpoint set only in an AWS profile is not read.
>
> **timeout** `number` (default: DEFAULT\_TIMEOUT)
>
> Seconds a request may wait to connect, or between two reads of its answer, before it fails; a download that keeps flowing is never cut off.
>
> **presignedUrlProvider** `function`
>
> Function `(path, operation, options)` that returns a signed URL for each S3 operation. When set, the mount fetches those URLs and needs no credentials.
>
> **defaultContentType** `string`
>
> `Content-Type` stored with every object the mount writes. Unset sends none, so the store applies its own default; a write through `presignedUrlProvider` sends `application/octet-stream`.
>
> **keyPrefix** `string`
>
> Key prefix inside the bucket that the mount treats as its root; keys outside it are not visible. Leading slashes are dropped and a trailing `/` is added.
>
> Fields marked secret are redacted in snapshots.
>
> Without `presignedUrlProvider`, the browser `S3VFS` signs requests itself with
> the Node fields' credentials, which puts the keys in the page.

## Integration

S3 answers these commands through its own API, so fewer objects are read. See
[Integration](https://docs.mirage.strukto.ai/home/integration/overview.md).

* `md5sum`: the object's ETag, when it is the MD5 (a single-part upload,
  unencrypted or under SSE-S3).
* `find`: one recursive listing.
* `du`: one recursive listing.
* `ls -R` and `tree`: one recursive listing.

## VFS.md

The [VFS.md](https://docs.mirage.strukto.ai/home/setup/vfs-md.md) an agent reads with S3 mounted at `/<path>`.
`<mode>` is the mount's mode: `read`, `write` or `exec`. The commands shown are
those of a `write` mount.

> [tab] Rendered
>
> **Virtual Filesystem**
>
> | Path | Description |
> | - | - |
> | `/<path>` | `<mode>` |
>
> **Supported Unix Commands for VFS**
>
> Unix commands work on each path as on a local filesystem, as far as the path supports them:
>
> | Path | Supported |
> | - | - |
> | `/<path>` | Almost all Unix commands. |
>
> **Limitations of Unix Commands for VFS**
>
> Where a path differs from a local filesystem:
>
> | Path | Limitations |
> | - | - |
> | `/<path>` | `grep -r`, `find` and `du` stop after 1000 files; a glob matching more, `tar -c`, `cp -r` and `rm -r` are refused. |
>
> Run `man <cmd>` or `<cmd> --help` for one command's usage.

> [tab] Raw
>
> ```markdown theme={null}
> ---
> name: workspace-filesystem
> description: The virtual filesystem layout of this workspace.
> ---
>
> # Virtual Filesystem
>
> | Path | Description |
> | --- | --- |
> | `/<path>` | `<mode>` |
>
> ## Supported Unix Commands for VFS
>
> Unix commands work on each path as on a local filesystem, as far as the path supports them:
>
> | Path | Supported |
> | --- | --- |
> | `/<path>` | Almost all Unix commands. |
>
> ## Limitations of Unix Commands for VFS
>
> Where a path differs from a local filesystem:
>
> | Path | Limitations |
> | --- | --- |
> | `/<path>` | `grep -r`, `find` and `du` stop after 1000 files; a glob matching more, `tar -c`, `cp -r` and `rm -r` are refused. |
>
> Run `man <cmd>` or `<cmd> --help` for one command's usage.
> ```


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.