> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mirage.strukto.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Runtime Routing

> Route each line with a policy script, first a fixed one and then one that asks an LLM, so plain Python stays in-process and only GPU work reaches a Daytona GPU sandbox.

Mirage picks a machine for every command line. Most Python an agent writes
runs on [Monty](https://docs.mirage.strukto.ai/home/setup/runtime/monty.md), at no infrastructure
cost. A line that needs a GPU has to run on one, and a GPU is
billed while it runs. A [route policy](https://docs.mirage.strukto.ai/home/policy/route-machine.md) makes that
choice per line. This example sends GPU work to a
[Daytona](https://docs.mirage.strukto.ai/home/setup/machine/daytona.md) GPU sandbox and keeps
everything else on Monty: first with a fixed script, then with one that asks
an LLM.

The LLM step builds on the LLM routing example in
[Python](https://github.com/strukto-ai/mirage/tree/main/examples/python/policy/llm_routing)
and
[TypeScript](https://github.com/strukto-ai/mirage/tree/main/examples/typescript/policy/llm_routing),
which routes between Monty and the host's Python.

## The GPU sandbox

Create a GPU sandbox in Daytona from a snapshot with PyTorch and CUDA, and
note its id. Daytona sets the GPU type on the snapshot, and a GPU sandbox is
ephemeral. Mirage runs programs in it and leaves its lifecycle to Daytona.

## The workspace

Two machines capture `python3`: `monty` first, a `worker` that runs it on
Monty, then `gpu`, the Daytona sandbox. The route policy is a script that
runs on Monty, read from `route.py`:

```yaml workspace.yaml theme={null}
mode: exec

mounts:
  /data:
    vfs: ram

machines:
  monty:
    provider: worker
  gpu:
    provider: daytona
    sandbox_id: ${DAYTONA_GPU_SANDBOX_ID}
    captures: ["python3"]

route_policy:
  script: route.py
  runtime: monty
```

## Route by a fixed script

A policy script reads the line and returns where it runs. This one sends
any line that names PyTorch or CUDA to the GPU:

```python route.py theme={null}
def pre_execute(ctx):
    if any(word in ctx["line"] for word in ("torch", "cuda")):
        return {"kind": "route", "machine": "gpu", "reason": "the line uses PyTorch or CUDA"}
```

A line the script stays silent on runs on the first machine that captures
it, so `python3 -c "print(sum(range(1, 101)))"` runs on Monty. The script costs
microseconds, and it misses what it does not name: a script file that
imports `torch`, or a GPU library under another name.

## Route by an LLM

An LLM reads the line the way an engineer would. This `route.py` describes
each machine and asks which one the line needs. A program on Monty has no
network, so the LLM call is a
[host function](https://docs.mirage.strukto.ai/home/policy/scripts.md#host-functions), `parse_machine`, that
the workspace hands it:

```python route.py theme={null}
TASK = (
    "Pick the machine that can run the Python code in this line, monty "
    "when it can."
)

MACHINES = {
    "monty": (
        "Mirage's sandboxed Python, free: a subset of the "
        "language and part of the standard library, with no packages, GPU, "
        "network or subprocesses"
    ),
    "gpu": (
        "a cloud sandbox with an NVIDIA GPU, CUDA, PyTorch and the full "
        "standard library, billed while it runs"
    ),
}


async def pre_execute(ctx):
    if any(c["command"] == "python3" for c in ctx["commands"]):
        choice = await parse_machine(TASK, MACHINES, ctx["line"])
        return {"kind": "route", **choice}
```

A line with no `python3` gets no answer, and runs where its commands'
captures put it.

`parse_machine` asks the LLM for a machine and a one-sentence reason. Both
`explain` and the run ask the policy, so it keeps each line's answer and the
preview and the run agree:

> [tab] Python
>
> ```python theme={null}
> from typing import Literal
>
> from openai import AsyncOpenAI
> from pydantic import BaseModel
>
> from mirage import Workspace
> from mirage.config import load_config
>
>
> class MachineChoice(BaseModel):
>     machine: Literal["monty", "gpu"]
>     reason: str
>
>
> CHOICES: dict[str, dict[str, str]] = {}
>
>
> async def parse_machine(task: str, machines: dict[str, str], line: str) -> dict[str, str]:
>     if line not in CHOICES:
>         prompt = f"{task} Say why in one sentence.\n" + "\n".join(
>             f"{name}: {about}" for name, about in machines.items()
>         )
>         done = await AsyncOpenAI(timeout=10).responses.parse(
>             model="gpt-6-luna", instructions=prompt, input=line, text_format=MachineChoice
>         )
>         CHOICES[line] = done.output_parsed.model_dump()
>     return CHOICES[line]
>
>
> ws = Workspace(
>     **load_config("workspace.yaml").to_workspace_kwargs(),
>     policy_functions={"parse_machine": parse_machine},
> )
> ```

> [tab] TypeScript
>
> ```typescript theme={null}
> import OpenAI from 'openai'
> import { zodTextFormat } from 'openai/helpers/zod'
> import { z } from 'zod'
> import { Workspace, configToWorkspaceArgs, loadWorkspaceConfigFile } from '@struktoai/mirage-node'
>
> const MachineChoice = z.object({ machine: z.enum(['monty', 'gpu']), reason: z.string() })
> type Choice = z.infer<typeof MachineChoice>
>
> const choices = new Map<string, Choice>()
>
> async function parseMachine(task: string, machines: Record<string, string>, line: string): Promise<Choice> {
>   const known = choices.get(line)
>   if (known !== undefined) return known
>   const prompt = [
>     `${task} Say why in one sentence.`,
>     ...Object.entries(machines).map(([name, about]) => `${name}: ${about}`),
>   ].join('\n')
>   const done = await new OpenAI({ timeout: 10_000 }).responses.parse({
>     model: 'gpt-6-luna',
>     instructions: prompt,
>     input: line,
>     text: { format: zodTextFormat(MachineChoice, 'machine_choice') },
>   })
>   if (done.output_parsed === null) throw new Error('the model chose no machine')
>   choices.set(line, done.output_parsed)
>   return done.output_parsed
> }
>
> const { mounts, options } = await configToWorkspaceArgs(loadWorkspaceConfigFile('workspace.yaml'))
> const ws = new Workspace(mounts, { ...options, policyFunctions: { parse_machine: parseMachine } })
> ```

## Run it

`ws.explain(line)` shows where a line will run and why, before it runs, so
no GPU starts for a line that does not need one:

> [tab] Python
>
> ```python theme={null}
> for line in (
>     'python3 -c "print(sum(range(1, 101)))"',
>     'python3 -c "import torch; print(torch.cuda.get_device_name(0))"',
> ):
>     plan = await ws.explain(line)
>     for answer in plan.answers:
>         if answer.kind == "route":
>             print(line, "->", answer.machine, answer.reason)
>     await ws.shell(line)
> ```

> [tab] TypeScript
>
> ```typescript theme={null}
> for (const line of [
>   'python3 -c "print(sum(range(1, 101)))"',
>   'python3 -c "import torch; print(torch.cuda.get_device_name(0))"',
> ]) {
>   const plan = await ws.explain(line)
>   for (const answer of plan.answers) {
>     if (answer.kind === 'route') console.log(line, '->', answer.machine, answer.reason)
>   }
>   await ws.shell(line)
> }
> ```

The sum stays on Monty and prints `5050`; only the `torch` line
reaches the GPU.

## Notes

* Daytona GPU sandboxes cannot mount FUSE, so a Mirage inside one cannot
  serve the workspace's files. Lines routed there should carry their code
  (`python3 -c`, or a script piped on stdin) and fetch their data, such as
  model weights from Hugging Face, themselves.
* The script has 10 seconds, the LLM call included. A script that fails or
  runs past them fails the line with a routing error rather than guessing.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.