qalarc.com / projects / gmux
Gmux — Gesture Terminal Multiplexer
Tauri desktop app + gesture engine + phone PWA for running fleets of AI agents. tmux lives inside it. Swipe to switch agents. Speak to them. Grant permissions from your phone.
Gmux is a full HCI stack built for the AI-agent era. When you run 10+ Claude/Codex agents in parallel, keyboard shortcuts and split terminals aren't enough — you need visibility, multimodal control, and a remote. js PTY, gesture and voice overlays sit on top of the exact terminal area, and a right sidebar shows every agent's live state, todo progress, and pending permission requests.
py, 916 lines) connects the Rust PTY, gesture engine, voice daemon, and phone. py) intercepts physical volume keys via a MediaSession hack to cycle agents without touching the screen — and shows live per-agent todo progress bars, tool chips, and permission Accept/Deny buttons.
The gesture engine (674 lines, ported from the castle-defense game) classifies 12 gestures including swipes, pinch, fist, thumbs up/down, and two-hand combos with landmark smoothing, velocity tracking, and confirm-frame debouncing. All three pieces (Tauri app, Python backend, phone UI) were confirmed working end-to-end in April 2026.
Capabilities
tmux runs inside Tauri via real PTY (xterm.js + portable-pty Rust crate) — no overlay alignment hacks
674-line gesture engine: classifies 12 gestures with landmark smoothing, velocity, confirm frames, two-hand state
1,185-line gesture renderer: skeleton, wrist trails, confidence rings, pinch indicators
Swipe left/right to switch tmux windows; three-finger-left jumps to next WAITING agent
Thumbs up/down accepts/denies agent permission requests from a hand wave
Phone PWA (873 lines) served over Tailscale: per-agent cards with state, todo bars, tool chips, permission buttons
Volume key interception on phone via MediaSession hack (silent WAV loop) — vol↓ cycles agents, vol↑ push-to-talk
Glasses mode (?mode=glasses): auto-cycles to next non-idle agent every 4s, larger fonts — designed for AR passthrough
Voice drawer with mic level visualiser, transcript history, command routing to focused tmux pane
Camera PiP in corner + gesture HUD with confidence bar
Calibration view: live 320×240 camera with per-gesture sliders, saves to ~/.config/gmux/gesture_calibration.json
Drop a project folder onto the window — opens new tmux window with qc ready in that directory
Python sidecars auto-spawned on startup with port-in-use checks (bridge.py :8767, monitor.py, voice_daemon :8765)
Per-window state aggregation: worst-state across all panes wins, shown in window tab badges
Tags
Status: active · First built: 2026-04-21 · Last updated: 2026-04-21