Unattended Client-Driving Harness (VM / headless) — research & plan

Question: can the AI drive the dev game client (capture + clicks) fully unattended, isolated from the user's desktop? Answer: yes — best via a headless Linux box running RuneLite under Xvfb, driven over SSH. A Windows VM with the Native client is possible but heavier and not fully self-provisioning. The server already runs headless on the host (driven via CLI + the debug TCP channel :43595); only the visual client needs isolating.

Status: PLAN ONLY (not yet built). We continue dev on the host meanwhile; the PrintWindow capture fix already makes reads/commands non-intrusive, so only real mouse-clicks touch the host desktop.

Why the current setup interrupts the desktop

• Capturing the host client by screen-rect grabs whatever is visually on top; clicking it requires foregrounding + moving the real mouse. (Reads are now fixed via PrintWindow — see scripts/ui.ps1 — but clicks still foreground.)
• Goal: a place where the client lives that the AI fully owns, so capture+input never touch the user.

Option A — Headless Linux + RuneLite (RECOMMENDED)

RuneLite is Java and connects to our rev-233 server via RSProx version-pinning (verified by the user). RSProx is Java too. So the entire client stack runs on Linux, and Linux has mature headless GUI automation that needs no real display, GPU, or focus:

• Xvfb — virtual X display (Xvfb :99 -screen 0 1280x1000x24); the client renders into it.
• xdotool — synthesizes mouse/keyboard events to a window on that display (no focus steal, no real mouse movement).
• scrot / ImageMagick import — capture the display (or a window) to PNG.
• OpenSSH — the AI invokes all of the above from the host shell: ssh box "DISPLAY=:99 xdotool ...".
• Software GL (Mesa llvmpipe) — set RuneLite to its CPU/software renderer (disable the GPU plugin) so no GPU is required. Slower frames, fine for dev.

Networking: the client box reaches the host server at host-ip:43594 (NAT/bridged). The AI keeps driving the server from the host as today (:43594 / :43595).

What the user bootstraps (one-time, needs elevation/hypervisor)

Pick ONE host for the Linux env:

1. WSL2 Ubuntu (lightest; on the user's Win11): wsl --install -d Ubuntu (admin, one reboot).
2. Hyper-V Linux VM (native on Win11 Pro): create an Ubuntu VM (ISO or cloud image + cloud-init).
3. Any spare Linux box / cloud VM the user has.

What the AI can then automate (unattended provisioning script)

Once SSH access exists, a single provisioning script installs and configures everything:

sudo apt-get update
sudo apt-get install -y openjdk-17-jre xvfb x11-utils xdotool scrot imagemagick openssh-server fonts-dejavu
# fetch RuneLite (jar) + the project's RSProx; copy proxy-targets.yaml (host IP, rev 233, modulus from
# server/.data/client.key); set RuneLite renderer = software.
# systemd units: xvfb@:99, rsprox, runelite (DISPLAY=:99) so they autostart.

Then a Linux port of the harness (mirror of scripts/ui.ps1) provides find|shot|click|type|key via xdotool + scrot against DISPLAY=:99, callable over SSH. Login/spawn/verify loops work exactly as today, but headless and non-intrusive.

Pros / cons

• ➕ Fully scriptable post-bootstrap; no GPU passthrough; no focus steal; cheap; SSH-native.
• ➖ RuneLite ≠ the Native client (fine — both connect); software GL is slower; one-time bootstrap.

Option B — Windows VM + Native client

• Hyper-V (native on Win11 Pro) VM running RSProx + the Native client; OpenSSH Server in the guest for control; the AI runs the existing ui.ps1 inside the guest over SSH.
• Blockers to full unattended setup: enabling Hyper-V needs admin + reboot; an unattended guest Windows install needs an ISO + autounattend.xml; the Native client's OpenGL needs GPU partitioning (GPU-P) in the guest (fiddly) or a software-GL fallback.
• ➕ Same Native client we use now; ➖ heavy, not self-provisioning, GPU is a real risk.

Verdict: only worth it if we specifically need the Native client. For automation, Option A wins.

Recommendation

1. Keep developing on the host now (PrintWindow reads + dbg.ps1 input are already non-intrusive; only real clicks foreground).
2. When click-heavy testing (combat/skilling) makes that annoying, stand up Option A: user does the one-time bootstrap (WSL2 Ubuntu is lowest-friction), then the AI provisions RuneLite+Xvfb+xdotool+SSH and ports the harness. That yields fully unattended, desktop-free client driving.

Open questions to resolve before building

• WSL2 vs Hyper-V VM vs separate box (resource/permission trade-offs).
• RSProx headless launch on Linux (it's a Gradle proxy task / GUI launcher — confirm a headless/ CLI launch path, or run its launcher under Xvfb too).
• RuneLite software-renderer config + any RSProx plugin-hub constraints under version-pinning.