Tracing

Mirage can send each line it runs to an OpenTelemetry backend as a trace. Use traces to watch many machines in one place, such as Datadog, Honeycomb, Grafana or Langfuse. Command logs compares traces with the logs Mirage keeps itself.

A session runs a line

↓→

Mirage

writes spans

↓→

OTel SDK

in your process

↓→

OTLP backend

such as Datadog

Setup

Mirage sends nothing until your process sets up an OpenTelemetry SDK. Mirage depends only on the OpenTelemetry API, so install the SDK and an exporter yourself. The exporter picks the backend. Set up the SDK before the workspace runs its first line.

Python
TypeScript

Set up the SDK in code.

agent.py

import asyncio

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

from mirage import RAMVFS, MountMode, Workspace

# Set up the OpenTelemetry SDK before the workspace runs a line.
provider = TracerProvider()
provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="https://otlp.example.com/v1/traces"))
)
trace.set_tracer_provider(provider)


# No tracing code here. Mirage finds the SDK on its own.
async def main():
    ws = Workspace({"/data": RAMVFS()}, mode=MountMode.WRITE, workspace_id="demo")
    await ws.shell("echo hi > /data/a.txt; cat /data/a.txt")
    await ws.close()


asyncio.run(main())
# Send the last spans before the process exits.
provider.shutdown()

pip install mirage-ai opentelemetry-sdk opentelemetry-exporter-otlp-proto-http
python agent.py

Or run without code changes. opentelemetry-instrument sets up the SDK for you.

agent.py

# No tracing code here. opentelemetry-instrument sets up the SDK.
import asyncio

from mirage import RAMVFS, MountMode, Workspace


async def main():
    ws = Workspace({"/data": RAMVFS()}, mode=MountMode.WRITE, workspace_id="demo")
    await ws.shell("echo hi > /data/a.txt; cat /data/a.txt")
    await ws.close()


asyncio.run(main())

pip install mirage-ai opentelemetry-distro opentelemetry-exporter-otlp

# opentelemetry-instrument sets up the SDK from the OTEL_* variables
# before agent.py starts. Without OTEL_EXPORTER_OTLP_PROTOCOL, the
# exporter uses gRPC instead of HTTP.
OTEL_TRACES_EXPORTER=otlp \
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf \
OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.example.com \
  opentelemetry-instrument python agent.py

agent.ts

import { trace } from '@opentelemetry/api'
import { BatchSpanProcessor, TracerProvider } from '@opentelemetry/sdk-trace'
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http'
import { MountMode, RAMVFS, Workspace } from '@struktoai/mirage-node'

// Set up the OpenTelemetry SDK before the workspace runs a line.
const provider = new TracerProvider({
  spanProcessors: [
    new BatchSpanProcessor({
      exporter: new OTLPTraceExporter({ url: 'https://otlp.example.com/v1/traces' }),
    }),
  ],
})
trace.setGlobalTracerProvider(provider)

// No tracing code here. Mirage finds the SDK on its own.
const ws = new Workspace({ '/data': new RAMVFS() }, { mode: MountMode.WRITE, workspaceId: 'demo' })
await ws.shell('echo hi > /data/a.txt; cat /data/a.txt')
await ws.close()
// Send the last spans before the process exits.
await provider.shutdown()

# Mirage needs @opentelemetry/api 1.10 or later. Use the 3.x @opentelemetry/sdk-trace:
# the older @opentelemetry/sdk-trace-base 2.x allows only API versions below 1.10.
npm install @struktoai/mirage-node @opentelemetry/api @opentelemetry/sdk-trace @opentelemetry/exporter-trace-otlp-http

# Node 22.20 or later runs TypeScript files directly.
node agent.ts

Example

Run one line in a workspace that mounts a RAM folder at /data:

cd /data; wc -l notes.txt

Mirage sends one trace with three spans. The command span covers the whole line. The other two sit under it.

command    cd /data; wc -l notes.txt     exit 0, 2 ms
├─ vfs.read    /data/notes.txt           6 bytes
└─ process     cd /data; wc -l notes.txt exit 0

These are the command span’s attributes:

{
  "mirage.line.id": "line_01a1c8d0-d172-7069-a48a-affb2a2dda8a",
  "mirage.session.id": "01a1c8d0-d16e-7047-a1ed-eb23fe176473",
  "mirage.command": "cd /data; wc -l notes.txt",
  "mirage.timestamp": 1794370490739,
  "mirage.workspace.id": "demo",
  "host.id": "host:my-laptop",
  "mirage.cwd": "/",
  "mirage.exit_code": 0,
  "mirage.machine.ids": ["host:my-laptop"],
  "mirage.duration_ms": 2,
  "mirage.stdout": "3 notes.txt\n",
  "mirage.stderr": ""
}

The vfs.read span names the file, the mount that served it and the bytes read:

{
  "mirage.vfs.path": "/data/notes.txt",
  "mirage.vfs.source": "ram",
  "mirage.vfs.bytes": 6,
  "mirage.vfs.timestamp": 1794370490741,
  "mirage.vfs.duration_ms": 0
}

Spans

Each trace covers one line and everything it started. Use the span and attribute names in your backend’s queries, dashboards and alerts.

To find	Filter on
Failed commands	`command` spans where `mirage.exit_code` is not 0
Slow mounts	The duration of `vfs.<call>` spans, grouped by `mirage.vfs.source`
Data read from a mount	The sum of `mirage.vfs.bytes` on `vfs.read` spans, by `mirage.vfs.source`
One agent’s run	Any span with a given `mirage.session.id`
Killed programs	`process` spans with a `mirage.signal`
Traces missing file calls	`command` spans where `mirage.vfs.dropped` is above 0

All spans and attributes

Span	Sent	Attributes
`command`	When the line ends	`mirage.line.id`, `mirage.workspace.id`, `mirage.session.id`, `host.id`, `mirage.machine.ids`, `mirage.command`, `mirage.cwd`, `mirage.exit_code`, `mirage.timestamp`, `mirage.duration_ms`, `mirage.stdout`, `mirage.stderr`, `mirage.vfs.dropped`
`vfs.<call>`	For each call the line makes to a mount, such as `vfs.read`	`mirage.vfs.path`, `mirage.vfs.source`, `mirage.vfs.bytes`, `mirage.vfs.timestamp`, `mirage.vfs.duration_ms`
`job`	When an `&` job ends	`mirage.job.id`, `mirage.job.number`, `mirage.line.id`, `mirage.workspace.id`, `mirage.session.id`, `host.id`, `mirage.machine.id`, `process.pid`, `mirage.command`, `mirage.cwd`, `mirage.exit_code`, `mirage.signal`
`process`	When a process the line started ends	`process.pid`, `process.parent_pid`, `process.group_leader.pid`, `mirage.line.id`, `mirage.workspace.id`, `mirage.session.id`, `host.id`, `mirage.machine.id`, `process.command_line`, `mirage.cwd`, `mirage.runtime`, `mirage.machine.pid`, `process.exit.code`, `mirage.signal`, `mirage.error`

Attributes to know

Attribute	Meaning
`host.id`	The host that runs Mirage, such as `host:my-laptop`.
`mirage.machine.id`	The machine a job or process ran on, such as `docker:tools`.
`mirage.machine.ids`	The machines the line’s processes ran on by the time the line ended. A background job that starts a program later names its machine on that program’s `process` span.
`mirage.cwd`	The directory the line started in.
`mirage.timestamp`	The span’s start, in milliseconds. It repeats the span’s own time for tracers that can’t set it, such as Cloudflare Workers custom spans. `mirage.duration_ms` does the same for its length.
`mirage.vfs.timestamp`	When the mount call ended, in milliseconds.

Traces cover what runs on the workspace’s machine and the runtimes it reaches. A program that a runtime runs on another host sends no spans of its own.

Limits

Setting	Default	Effect
`MIRAGE_TRACE_CONTENT_MAX_BYTES`	16 KiB	How much of the output `mirage.stdout` and `mirage.stderr` each keep, as UTF-8 text. The default fits the 64 KB that a Cloudflare Workers span allows for all its attributes.
`MIRAGE_TRACE_MAX_SPANS`	1000	The most `vfs.<call>` spans one line sends, so a walk over a large tree can’t flood the exporter. `mirage.vfs.dropped` counts the rest. Command logs still keep every call.
`MIRAGE_TRACE_CAPTURE_CONTENT`	`true`	Set to `false` to keep output and mount calls out of traces. `command` spans then carry no output, and Mirage sends no `vfs.<call>` spans.

Getting Started

Examples

Agents

VFS

Integration

CLIs

Mirage Shell

Agent Profiles

Policy

Runtimes

Machines

Infrastructure

Configuration

Notifications

Help

Setup

Example

Spans

Attributes to know

Limits

​Setup

​Example

​Spans

​Attributes to know

​Limits

Setup

Example

Spans

Attributes to know

Limits