> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mirage.strukto.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Tracing

> Send each line Mirage runs to an OpenTelemetry backend.

Mirage can send each line it runs to an OpenTelemetry backend as a trace. Use
traces to watch many machines in one place, such as Datadog, Honeycomb,
Grafana or Langfuse. [Command logs](https://docs.mirage.strukto.ai/home/infrastructure/command-logs.md#logs-and-traces)
compares traces with the logs Mirage keeps itself.

A session runs a line

↓→

Miragewrites spans

↓→

OTel SDKin your process

↓→

OTLP backendsuch as Datadog

## Setup

Mirage sends nothing until your process sets up an OpenTelemetry SDK. Mirage
depends only on the OpenTelemetry API, so install the SDK and an exporter
yourself. The exporter picks the backend. Set up the SDK before the
workspace runs its first line.

> [tab] Python
>
> **Set up the SDK in code.**
>
> ```python agent.py theme={null}
> import asyncio
>
> from opentelemetry import trace
> from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
> from opentelemetry.sdk.trace import TracerProvider
> from opentelemetry.sdk.trace.export import BatchSpanProcessor
>
> from mirage import RAMVFS, MountMode, Workspace
>
> # Set up the OpenTelemetry SDK before the workspace runs a line.
> provider = TracerProvider()
> provider.add_span_processor(
>     BatchSpanProcessor(OTLPSpanExporter(endpoint="https://otlp.example.com/v1/traces"))
> )
> trace.set_tracer_provider(provider)
>
>
> # No tracing code here. Mirage finds the SDK on its own.
> async def main():
>     ws = Workspace({"/data": RAMVFS()}, mode=MountMode.WRITE, workspace_id="demo")
>     await ws.shell("echo hi > /data/a.txt; cat /data/a.txt")
>     await ws.close()
>
>
> asyncio.run(main())
> # Send the last spans before the process exits.
> provider.shutdown()
> ```
>
> ```bash theme={null}
> pip install mirage-ai opentelemetry-sdk opentelemetry-exporter-otlp-proto-http
> python agent.py
> ```
>
> **Or run without code changes.** `opentelemetry-instrument` sets up the
> SDK for you.
>
> ```python agent.py theme={null}
> # No tracing code here. opentelemetry-instrument sets up the SDK.
> import asyncio
>
> from mirage import RAMVFS, MountMode, Workspace
>
>
> async def main():
>     ws = Workspace({"/data": RAMVFS()}, mode=MountMode.WRITE, workspace_id="demo")
>     await ws.shell("echo hi > /data/a.txt; cat /data/a.txt")
>     await ws.close()
>
>
> asyncio.run(main())
> ```
>
> ```bash theme={null}
> pip install mirage-ai opentelemetry-distro opentelemetry-exporter-otlp
>
> # opentelemetry-instrument sets up the SDK from the OTEL_* variables
> # before agent.py starts. Without OTEL_EXPORTER_OTLP_PROTOCOL, the
> # exporter uses gRPC instead of HTTP.
> OTEL_TRACES_EXPORTER=otlp \
> OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf \
> OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.example.com \
>   opentelemetry-instrument python agent.py
> ```

> [tab] TypeScript
>
> ```typescript agent.ts theme={null}
> import { trace } from '@opentelemetry/api'
> import { BatchSpanProcessor, TracerProvider } from '@opentelemetry/sdk-trace'
> import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http'
> import { MountMode, RAMVFS, Workspace } from '@struktoai/mirage-node'
>
> // Set up the OpenTelemetry SDK before the workspace runs a line.
> const provider = new TracerProvider({
>   spanProcessors: [
>     new BatchSpanProcessor({
>       exporter: new OTLPTraceExporter({ url: 'https://otlp.example.com/v1/traces' }),
>     }),
>   ],
> })
> trace.setGlobalTracerProvider(provider)
>
> // No tracing code here. Mirage finds the SDK on its own.
> const ws = new Workspace({ '/data': new RAMVFS() }, { mode: MountMode.WRITE, workspaceId: 'demo' })
> await ws.shell('echo hi > /data/a.txt; cat /data/a.txt')
> await ws.close()
> // Send the last spans before the process exits.
> await provider.shutdown()
> ```
>
> ```bash theme={null}
> # Mirage needs @opentelemetry/api 1.10 or later. Use the 3.x @opentelemetry/sdk-trace:
> # the older @opentelemetry/sdk-trace-base 2.x allows only API versions below 1.10.
> npm install @struktoai/mirage-node @opentelemetry/api @opentelemetry/sdk-trace @opentelemetry/exporter-trace-otlp-http
>
> # Node 22.20 or later runs TypeScript files directly.
> node agent.ts
> ```

## Example

Run one line in a workspace that mounts a RAM folder at `/data`:

```bash theme={null}
cd /data; wc -l notes.txt
```

Mirage sends one trace with three spans. The `command` span covers the whole
line. The other two sit under it.

```text theme={null}
command    cd /data; wc -l notes.txt     exit 0, 2 ms
├─ vfs.read    /data/notes.txt           6 bytes
└─ process     cd /data; wc -l notes.txt exit 0
```

These are the `command` span's attributes:

```json theme={null}
{
  "mirage.line.id": "line_01a1c8d0-d172-7069-a48a-affb2a2dda8a",
  "mirage.session.id": "01a1c8d0-d16e-7047-a1ed-eb23fe176473",
  "mirage.command": "cd /data; wc -l notes.txt",
  "mirage.timestamp": 1794370490739,
  "mirage.workspace.id": "demo",
  "host.id": "host:my-laptop",
  "mirage.cwd": "/",
  "mirage.exit_code": 0,
  "mirage.machine.ids": ["host:my-laptop"],
  "mirage.duration_ms": 2,
  "mirage.stdout": "3 notes.txt\n",
  "mirage.stderr": ""
}
```

The `vfs.read` span names the file, the mount that served it and the bytes
read:

```json theme={null}
{
  "mirage.vfs.path": "/data/notes.txt",
  "mirage.vfs.source": "ram",
  "mirage.vfs.bytes": 6,
  "mirage.vfs.timestamp": 1794370490741,
  "mirage.vfs.duration_ms": 0
}
```

## Spans

Each trace covers one line and everything it started. Use the span and
attribute names in your backend's queries, dashboards and alerts.

| To find | Filter on |
| - | - |
| Failed commands | `command` spans where `mirage.exit_code` is not 0 |
| Slow mounts | The duration of `vfs.<call>` spans, grouped by `mirage.vfs.source` |
| Data read from a mount | The sum of `mirage.vfs.bytes` on `vfs.read` spans, by `mirage.vfs.source` |
| One agent's run | Any span with a given `mirage.session.id` |
| Killed programs | `process` spans with a `mirage.signal` |
| Traces missing file calls | `command` spans where `mirage.vfs.dropped` is above 0 |

> [accordion] All spans and attributes
>
> | Span | Sent | Attributes |
> | - | - | - |
> | `command` | When the line ends | `mirage.line.id`, `mirage.workspace.id`, `mirage.session.id`, `host.id`, `mirage.machine.ids`, `mirage.command`, `mirage.cwd`, `mirage.exit_code`, `mirage.timestamp`, `mirage.duration_ms`, `mirage.stdout`, `mirage.stderr`, `mirage.vfs.dropped` |
> | `vfs.<call>` | For each call the line makes to a mount, such as `vfs.read` | `mirage.vfs.path`, `mirage.vfs.source`, `mirage.vfs.bytes`, `mirage.vfs.timestamp`, `mirage.vfs.duration_ms` |
> | `job` | When an `&` job ends | `mirage.job.id`, `mirage.job.number`, `mirage.line.id`, `mirage.workspace.id`, `mirage.session.id`, `host.id`, `mirage.machine.id`, `process.pid`, `mirage.command`, `mirage.cwd`, `mirage.exit_code`, `mirage.signal` |
> | `process` | When a process the line started ends | `process.pid`, `process.parent_pid`, `process.group_leader.pid`, `mirage.line.id`, `mirage.workspace.id`, `mirage.session.id`, `host.id`, `mirage.machine.id`, `process.command_line`, `mirage.cwd`, `mirage.runtime`, `mirage.machine.pid`, `process.exit.code`, `mirage.signal`, `mirage.error` |

## Attributes to know

| Attribute | Meaning |
| - | - |
| `host.id` | The host that runs Mirage, such as `host:my-laptop`. |
| `mirage.machine.id` | The [machine](https://docs.mirage.strukto.ai/home/setup/machine/overview.md) a job or process ran on, such as `docker:tools`. |
| `mirage.machine.ids` | The machines the line's processes ran on by the time the line ended. A background job that starts a program later names its machine on that program's `process` span. |
| `mirage.cwd` | The directory the line started in. |
| `mirage.timestamp` | The span's start, in milliseconds. It repeats the span's own time for tracers that can't set it, such as Cloudflare Workers custom spans. `mirage.duration_ms` does the same for its length. |
| `mirage.vfs.timestamp` | When the mount call ended, in milliseconds. |

Traces cover what runs on the workspace's machine and the runtimes it
reaches. A program that a runtime runs on another host sends no spans of its
own.

## Limits

| Setting | Default | Effect |
| - | - | - |
| `MIRAGE_TRACE_CONTENT_MAX_BYTES` | 16 KiB | How much of the output `mirage.stdout` and `mirage.stderr` each keep, as UTF-8 text. The default fits the 64 KB that a Cloudflare Workers span allows for all its attributes. |
| `MIRAGE_TRACE_MAX_SPANS` | 1000 | The most `vfs.<call>` spans one line sends, so a walk over a large tree can't flood the exporter. `mirage.vfs.dropped` counts the rest. Command logs still keep every call. |
| `MIRAGE_TRACE_CAPTURE_CONTENT` | `true` | Set to `false` to keep output and mount calls out of traces. `command` spans then carry no output, and Mirage sends no `vfs.<call>` spans. |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.