Documentation Index
Fetch the complete documentation index at: https://docs.mirage.strukto.ai/llms.txt
Use this file to discover all available pages before exploring further.
Mirage supports most common shell features. The following are intentionally
not supported due to the async streaming architecture or because they
don’t apply to a virtual filesystem.
How streaming works
Pipes are demand-driven: when downstream stops reading, upstream
stops being asked for chunks. The exception is the cache-drain task,
when a file is flagged for caching, a background task continues
downloading even if downstream exited early, so the cache entry is
complete for the next read. This is a deliberate trade-off between
bandwidth and cache fill rate, not a bug.
Unsupported Shell Syntax
| Feature | Workaround |
|---|
>(cmd) output process substitution | Use pipes: cmd1 | cmd2 |
for ((i=0; i<10; i++)) C-style loops | for i in $(seq 0 9) |
Array indexing ${arr[0]}, ${arr[@]} | Use separate variables |
trap signal handling | No-op (no real processes) |
tail -f follow mode | tail -n for last N lines (infinite producers need a cancellation protocol mirage doesn’t have yet) |
Background cache drain after partial read (cat file | head -n 1) | Drain runs to completion to populate cache; sequential execute() calls naturally observe the drain finishing |
for f in $(find ...) materializes full output before iterating | For very large outputs, pipe instead: find ... | while read f; do ...; done |
while read line loops capped at 10000 iterations | Process larger streams with awk/grep instead of shell-level loops |
echo /data/*.json no glob expansion | ls /data/*.json |
rm -i interactive prompt | rm (non-interactive by default) |
Backtick substitution `cmd` | $(cmd) |
Brace expansion {a,b,c} | List items explicitly |
Parser Limitations
| Feature | Workaround |
|---|
grep -e pattern file | cat file | grep -e pattern or grep "pattern" file |
awk -e program file | awk 'program' file |
sed -f script file | sed 's/foo/bar/' file |
Cross-Mount Limitations
| Feature | Workaround |
|---|
cp/mv between S3 mounts | Downloads then re-uploads (no server-side copy) |
diff -r on S3 | diff individual files |
cp into a read-only resource (e.g. cp X /gdrive/...) | Read-only resources can be source only; pick a writable destination |
mv touching a read-only resource on either side | mv requires delete on source and write on destination; use cp from the read-only side instead |
cp/mv writes to GCS via the S3-compat endpoint | GCS rejects boto3’s streaming-payload signature with SignatureDoesNotMatch; reads work fine. Use the native GCS endpoint or write through a different mount |
Home
Python
TS