# hyprtile-remoteviewer


## Encoder capabilities

The host can now expose encoder capabilities for launchers such as HyprTile:

```sh
hyprtile-remoteviewer list-encoders
hyprtile-remoteviewer list-encoders --json
```

`usable=true` means the encoder can be selected with `--encoder` in this build.
`detected=true` means matching hardware or system hints were found, but the backend may
still be unavailable until implemented. JSON output also includes `resolved_id` for
launchers that expose `auto`, plus `device` and `driver` when a hardware encoder has
a resolved render node. This build adds an optimized `vp9-vaapi` backend. When `ffmpeg`
exposes `vp9_vaapi` and a `/dev/dri/renderD*` node passes the smoke test, `vp9-vaapi`
becomes `usable=true` and `auto` resolves to it; otherwise `auto` falls back to
`vp9-cpu`. The VAAPI path avoids an extra repack copy when frame stride permits and
uses VAAPI VPP for BGRA/BGR0 to NV12 conversion before hardware encoding.

This build also performs diagnostic NVENC capability probing for future optional
backends. `hevc-nvenc` and `av1-nvenc` may be listed as detected, but the live browser
stream remains deliberately VP9-only.

```sh
hyprtile-remoteviewer host --output DP-1 --encoder auto --web-port 7780
```


`hyprtile-remoteviewer` is a C MVP for a modern Wayland-native remote viewer:

- direct wlroots-style screen capture, planned like `wayvnc`, not via portals
- VP9 encoding through `libvpx`
- optional VP9 VAAPI GPU encoding through an `ffmpeg` subprocess
- input protocol scaffold for VNC-like mouse/keyboard control
- future transport target: WebRTC video + DataChannel input

Current checked-in state: the VP9 encoder path is real and testable with generated frames. The wlroots screencopy backend is scaffolded separately so the project remains buildable on machines that only have `libvpx` installed.

## Install dependencies

On Arch-like systems:

```bash
sudo pacman -S meson ninja libvpx openssl
```

For the VP9 VAAPI GPU encoder and direct Wayland backend:

```bash
sudo pacman -S ffmpeg libva-utils wayland wayland-protocols
```

## Build

```bash
meson setup build
meson compile -C build
```

## Test VP9 output without Wayland

This writes an IVF VP9 stream to stdout:

```bash
./build/hyprtile-remoteviewer host --dummy-capture --fps 30 --bitrate 2500 > out.ivf
```

Play it with:

```bash
ffplay out.ivf
```

or inspect:

```bash
ffprobe out.ivf
```


## VP9 VAAPI GPU encoder

List detected encoders:

```sh
./build/hyprtile-remoteviewer list-encoders --json
```

When `vp9-vaapi` is shown as `usable=true`, start it explicitly with:

```sh
./build/hyprtile-remoteviewer host --output DP-1 --encoder vp9-vaapi --web-port 7780
```

The VAAPI path uses `ffmpeg` as a subprocess and still emits VP9/IVF, so it uses
the same browser WebCodecs path as `vp9-cpu`. You can override the render node with:

```sh
HTR_VAAPI_DEVICE=/dev/dri/renderD128 ./build/hyprtile-remoteviewer host --output DP-1 --encoder vp9-vaapi --web-port 7780
```

## Optional future encoder probing

`list-encoders --json` may report diagnostic HEVC/AV1 NVENC capability information.
Those entries are not selectable; the supported interactive stream encoders are
`vp9-cpu`, `vp9-vaapi`, and `auto`.

## Intended MVP architecture

```text
Hyprland / wlroots compositor
  -> wlr-screencopy / SHM initially
  -> BGRA/XRGB frames
  -> libvpx VP9 encoder
  -> WebRTC VP9 stream
  <- WebRTC DataChannel input events
  <- wlr-virtual-pointer / virtual-keyboard injection
```

## Near-term roadmap

1. Implement `wlr-screencopy-unstable-v1` SHM capture.
2. Add `list-outputs` using `wl_output` v4 names.
3. Replace IVF stdout sink with RTP/WebRTC sender.
4. Add DataChannel input parser.
5. Implement `wlr-virtual-pointer` for mouse.
6. Implement virtual keyboard or `uinput` fallback for keyboard.
7. Add DMA-BUF capture path for performance.

## Notes

The current `capture_wayland_wlr.c` is intentionally a scaffold. Compile the default target first and verify the VP9 encoder path. Then wire in the direct Wayland capture backend.


### Plain Makefile build

```sh
make
./build/hyprtile-remoteviewer host --dummy-capture --fps 30 --bitrate 2500 > out.ivf
ffplay out.ivf
```

Experimental wlroots/Wayland build:

```sh
make clean
make WAYLAND_WLR=1
```

## wlroots/Hyprland capture test

After fetching protocols:

```sh
./scripts/fetch-wlr-protocols.sh
make clean
make WAYLAND_WLR=1
./build/hyprtile-remoteviewer list-outputs
./build/hyprtile-remoteviewer host --output DP-1 --fps 30 --bitrate 2500 > out.ivf
ffplay out.ivf
```

This backend uses `zwlr_screencopy_manager_v1` + `wl_shm` and embeds the cursor in the frame for the MVP.


## Live TCP stream MVP

After the Wayland/wlroots build works, you can stream VP9/IVF directly to one TCP client:

```sh
make clean
make WAYLAND_WLR=1

./build/hyprtile-remoteviewer host --output eDP-1 --fps 30 --bitrate 2500 --listen --bind 0.0.0.0 --port 7777
```

On the client machine:

```sh
ffplay -fflags nobuffer -flags low_delay -framedrop -probesize 32 -analyzeduration 0 tcp://HOST_IP:7777
```

The old stdout mode still works:

```sh
./build/hyprtile-remoteviewer host --output eDP-1 --fps 30 --bitrate 2500 > out.ivf
ffplay out.ivf
```

This TCP mode is intentionally simple. It still sends IVF packets, so it is a transport milestone, not the final WebRTC/RTP layer. The next milestones are a separate input channel, then wlr-virtual-pointer and virtual-keyboard/uinput backends.

## TCP disconnect behavior

In `--listen` mode a client disconnect is not fatal. The host closes the client
socket, keeps capture/encoding alive, drops frames while no client is connected,
and accepts a new client on the same port. New clients receive the cached IVF
header before live VP9 frame packets.

## Input debug channel

The current MVP can open a second TCP port for JSON-line input events. Events are logged only; they are not injected into Wayland yet.

Host example:

```sh
./build/hyprtile-remoteviewer host \
  --output eDP-1 \
  --fps 30 \
  --bitrate 2500 \
  --listen \
  --bind 0.0.0.0 \
  --port 7777 \
  --input-port 7778
```

Video client:

```sh
ffplay -fflags nobuffer -flags low_delay -framedrop -probesize 32 -analyzeduration 0 tcp://HOST_IP:7777
```

Input tests:

```sh
printf '{"t":"move","x":0.5,"y":0.5}\n' | nc HOST_IP 7778
printf '{"t":"button","button":1,"down":true}\n' | nc HOST_IP 7778
printf '{"t":"button","button":1,"down":false}\n' | nc HOST_IP 7778
printf '{"t":"key","code":"KEY_A","down":true}\n' | nc HOST_IP 7778
printf '{"t":"key","code":"KEY_A","down":false}\n' | nc HOST_IP 7778
printf '{"t":"wheel","dx":0,"dy":-120}\n' | nc HOST_IP 7778
```

## Pointer injection MVP

Build with wlroots protocols enabled:

```sh
./scripts/fetch-wlr-protocols.sh
make clean
make WAYLAND_WLR=1
```

Run host with video + input + virtual pointer:

```sh
./build/hyprtile-remoteviewer host \
  --output eDP-1 \
  --fps 30 \
  --bitrate 2500 \
  --listen \
  --bind 0.0.0.0 \
  --port 7777 \
  --input-port 7778 \
  --enable-pointer
```

Video client:

```sh
ffplay -fflags nobuffer -flags low_delay -framedrop -probesize 32 -analyzeduration 0 tcp://127.0.0.1:7777
```

Pointer tests:

```sh
printf '{"t":"move","x":0.5,"y":0.5}\n' | nc 127.0.0.1 7778
printf '{"t":"button","button":1,"down":true}\n' | nc 127.0.0.1 7778
printf '{"t":"button","button":1,"down":false}\n' | nc 127.0.0.1 7778
printf '{"t":"wheel","dx":0,"dy":-120}\n' | nc 127.0.0.1 7778
```

`--enable-pointer` uses `wlr-virtual-pointer-unstable-v1`. Keyboard events are still debug-logged only; virtual keyboard support comes next.

## Dedicated-output remote control mode

Remote control is intentionally **read-only by default**. Incoming input events are logged unless you explicitly bind them to a dedicated output.

Use a separate monitor, dummy/headless output, or a remote-only workspace/output. Do **not** use your main laptop panel for remote control if you want to keep working locally.

Example:

```sh
./build/hyprtile-remoteviewer host \
  --output HDMI-A-1 \
  --listen --bind 0.0.0.0 --port 7777 \
  --input-port 7778 \
  --control-output HDMI-A-1
```

Safety behavior:

- `--control-output` must match the captured `--output`.
- If the output name cannot be resolved, the program refuses to create a global/unbound pointer.
- Without `--control-output`, input events are debug/log only.
- Pointer input uses `zwlr_virtual_pointer_v1`; keyboard input uses `zwp_virtual_keyboard_v1` when the compositor exposes it.
- Keyboard layout follows `XKB_DEFAULT_LAYOUT` when set; otherwise HyprTile currently defaults to `de` for the generated virtual-keyboard keymap.
- `--enable-pointer` is kept as a deprecated alias for older tests.

Suggested Hyprland workflow:

1. Use `hyprtile-remoteviewer list-outputs` to find the dedicated output.
2. Move a workspace or app to that output.
3. Stream and control only that output with `--control-output`.
4. Keep your local work on another monitor/workspace.


## Browser client MVP

A browser client is available with `--web-port`:

```sh
./build/hyprtile-remoteviewer host \
  --output HEADLESS-1 \
  --fps 30 \
  --bitrate 2500 \
  --web-bind 0.0.0.0 \
  --web-port 7780 \
  --control-output HEADLESS-1
```

Then open `http://HOST_IP:7780/` from another machine, or `http://127.0.0.1:7780/` locally.

`--web-bind 0.0.0.0` makes the browser client reachable from the LAN. If omitted, `--web-bind` defaults to `--bind`, and `--bind` defaults to `127.0.0.1`.

See `docs/BROWSER_CLIENT.md` for details and browser security-context notes.

### Browser fullscreen and pointer lock

The embedded browser client now has a remote-control mode for dedicated outputs:

- double-click the canvas, click **Fullscreen/Lock**, or press `Ctrl+Alt+F` to enter fullscreen + Pointer Lock
- press `Esc` to let the browser release Pointer Lock/fullscreen
- while locked, the local cursor is hidden and relative mouse movement is translated into absolute output coordinates
- canvas coordinate mapping accounts for `object-fit: contain` letterboxing, avoiding offset mouse positions when the browser aspect ratio differs from the streamed output

This is especially useful for headless Sway sessions streamed with `--control-output HEADLESS-1`.

### Headless Sway sessions

When streaming a wlroots headless output such as `HEADLESS-1`, the compositor may
announce 24-bit `BGR888/BG24` screencopy buffers. The remoteviewer converts those
frames to encoder-friendly `BGR0` internally.

For maximum stability in a separate headless user session, prefer the software
encoder first:

```sh
./build/hyprtile-remoteviewer host \
  --output HEADLESS-1 \
  --web-bind 127.0.0.1 \
  --web-port 7780 \
  --encoder vp9-cpu \
  --audio
```

If `--encoder auto` is used with an output whose name starts with `HEADLESS`, it
now resolves to `vp9-cpu` automatically. This avoids fragile VAAPI device access
from another user/headless compositor while keeping the normal GPU path available
for physical outputs such as `eDP-1` or `DP-1`.

### Android Chrome video disconnect fix

If Android Chrome connects and the host logs `browser video disconnected: Resource temporarily unavailable`, rebuild with this version. The embedded WebSocket server now handles `EAGAIN`/`EWOULDBLOCK` on nonblocking sockets by waiting for the socket to become writable instead of treating it as a disconnect.

## Android Chrome stream sync fix

This build makes the browser video WebSocket self-synchronizing for Android Chrome:

- every browser video WebSocket message is `IVF header + one complete IVF frame packet`
- the browser parser accepts repeated `DKIF` headers and skips them after decoder setup
- this avoids Android Chrome getting stuck on a stream that begins mid-IVF packet

Test with cache busting:

```bash
./build/hyprtile-remoteviewer host \
  --output eDP-1 \
  --fps 30 \
  --bitrate 2500 \
  --web-bind 0.0.0.0 \
  --web-port 7780
```

Open on Android Chrome:

```text
http://HOST_IP:7780/?v=5
```

## HTTPS / Android Chrome / WebCodecs

Android Chrome requires a secure context for WebCodecs when the viewer is opened via a LAN IP. You do **not** need Let's Encrypt for local/LAN development. Generate a self-signed certificate:

```bash
./scripts/gen-self-signed-cert.sh certs 192.168.1.50
```

Replace `192.168.1.50` with the IP or hostname you will open from Android.

Start the HTTPS browser client:

```bash
./build/hyprtile-remoteviewer host \
  --output eDP-1 \
  --fps 30 \
  --bitrate 2500 \
  --web-bind 0.0.0.0 \
  --web-port 7780 \
  --tls-cert certs/hyprtile-remoteviewer.crt \
  --tls-key certs/hyprtile-remoteviewer.key
```

Open:

```text
https://HOST_IP:7780/
```

The browser will warn about the self-signed certificate. For development, accept the warning or install/trust the generated certificate on the client device.

## Browser authentication

The browser transport can be protected with a shared password/key. This protects
both WebSocket endpoints used by the live video stream and remote input.

Password on the command line:

```sh
./build/hyprtile-remoteviewer host \
  --output HDMI-A-1 \
  --https-port 8443 \
  --tls-cert cert.pem \
  --tls-key key.pem \
  --auth-password 'choose-a-long-random-secret' \
  --control-output HDMI-A-1
```

Secret from a keyfile:

```sh
printf '%s\n' 'choose-a-long-random-secret' > viewer.key
chmod 600 viewer.key

./build/hyprtile-remoteviewer host \
  --output HDMI-A-1 \
  --https-port 8443 \
  --tls-cert cert.pem \
  --tls-key key.pem \
  --auth-keyfile viewer.key \
  --control-output HDMI-A-1
```

When authentication is enabled, the browser client asks for the password/key on
first load and stores it in browser local storage for that origin. You can also
open the client once with `?auth=SECRET`; the page removes the query parameter
from the visible URL after storing it.

Use authentication together with `--https-port` whenever the viewer is reachable
from another device. With plain HTTP, the shared secret can be observed on the
network.

## Encoder metadata and laptop-friendly auto policy

The `auto` encoder remains laptop-friendly and resolves to `vp9-vaapi` when a
working VAAPI VP9 path is available, otherwise `vp9-cpu`. The interactive browser
stream is VP9-only.

## Recommended remote-control mode: low-latency video + audio

For interactive desktop control, use `--audio`. This is the preferred HyprTile
remoteviewer path and the mode used by the F10 sharing overlay.

```sh
./build/hyprtile-remoteviewer host \
  --output eDP-1 \
  --web-bind 0.0.0.0 \
  --web-port 7780 \
  --encoder auto \
  --audio
```

This deliberately keeps audio separate from the proven realtime video pipeline:

- `/video`: existing low-latency video WebSocket
- `/audio`: separate low-latency PCM audio WebSocket
- `/input`: mouse/keyboard/control WebSocket

`--audio` starts an `ffmpeg` subprocess that captures 48 kHz stereo `s16le` PCM
from PulseAudio/PipeWire. By default it resolves the current default sink via
`pactl get-default-sink` and captures its `.monitor` source, so browser clients
hear the host output. A specific source can be selected explicitly:

```sh
./build/hyprtile-remoteviewer host --output eDP-1 --web-port 7780 \
  --audio \
  --audio-source alsa_output.pci-0000_00_1f.3-platform-skl_hda_dsp_generic.HiFi__Speaker__sink.monitor
```

The first click/tap on the canvas resumes WebAudio, which is required by modern
browsers before audio playback is allowed. If audio capture cannot be started,
the host logs a warning and continues video-only.

## Experimental AV/watch mode: muxed WebM + Opus

`--av` is intentionally **not** the recommended remote-control mode. It muxes
video and audio into a browser media stream so the browser can keep A/V sync, but
browsers add their own media buffering. Expect noticeably higher latency. Use it
only as an experimental watch/media mode, not for realtime control.

Start it manually with:

```sh
./build/hyprtile-remoteviewer host --output DP-1 --web-bind 0.0.0.0 --web-port 8099 --av
```

Or with an explicit Pulse/PipeWire source:

```sh
./build/hyprtile-remoteviewer host --output DP-1 --web-bind 0.0.0.0 --web-port 8099 --av \
  --audio-source alsa_output.pci-0000_00_1f.3-platform-skl_hda_dsp_generic.HiFi__Speaker__sink.monitor
```

In AV mode the browser page uses a native HTTP WebM stream at `/avstream.webm`.
The muxer uses ffmpeg externally:

- input video: raw BGRA frames from the existing HyprTile capture path
- input audio: Pulse/PipeWire monitor source
- output: WebM container with VP8 by default, VP9 optionally, and Opus audio
- browser transport: HTTP `video/webm` stream

The useful interactive modes are still:

```sh
./build/hyprtile-remoteviewer host --output DP-1 --web-port 8099
./build/hyprtile-remoteviewer host --output DP-1 --web-port 8099 --audio
```

### AV mobile profile

For Android Chrome or weaker clients in experimental AV/watch mode, use the
mobile-friendly preset. It starts a fresh per-client HTTP/WebM stream, downscales
before WebM encoding, and uses lower realtime settings:

```sh
./build/hyprtile-remoteviewer host --output eDP-1 --web-bind 192.168.2.36 --web-port 7780 --av --av-mobile
```

Equivalent manual tuning:

```sh
./build/hyprtile-remoteviewer host --output eDP-1 --web-bind 192.168.2.36 --web-port 7780 --av --av-size 1280x720 --fps 24 --bitrate 1800
```

`--av-size WxH` can be used independently, for example `--av-size 960x540` for an even lighter stream.
The browser client also reconnects the HTTP/WebM stream a few times if Android Chrome aborts the media pipeline.

## Low-latency vs AV mode

For interactive remote control, use:

```bash
./build/hyprtile-remoteviewer host --output eDP-1 --web-bind 192.168.2.36 --web-port 7780 --audio
```

`--audio` keeps the existing low-latency video WebSocket and adds audio on a
separate PCM WebSocket. This is the recommended mode for controlling the desktop.

`--av` is a buffered browser-media mode. It can be useful for experiments where
A/V sync matters more than latency, but it is not suitable as the default remote-control path.


### Pointer Lock behavior

In browser remote-control mode, use the Fullscreen/Lock button, double-click, or Ctrl+Alt+F to enter pointer-lock fullscreen. While pointer lock is active, click events do not send absolute browser coordinates anymore; only relative motion updates the remote pointer. This avoids the remote pointer jumping back to the position where fullscreen/pointer-lock was entered when clicking. Press Esc to leave pointer lock/fullscreen.

### Headless wlroots color note

Some wlroots headless outputs announce 24-bit `BG24`/`BGR888` frames. The remoteviewer normalizes these frames into `BGR0` before encoding. If red/blue channels appear swapped on `HEADLESS-1`, use this version or newer; the 24-bit headless conversion has been corrected for that case.


### v92 note: Gamepad button visibility

The browser page includes a visible **🎮 Gamepad** button in the top toolbar.
Click it before using the controller. The status text now reports how many pads
`navigator.getGamepads()` can currently see, which makes Android/Chrome tests
easier to diagnose.

## Browser Gamepad Support

The browser client now has first-class gamepad forwarding. Click **🎮 Gamepad** in
the browser toolbar, press any button on the local controller once, and the host
creates/feeds a virtual Linux gamepad named `HyprTile Remote Gamepad`.

Implementation details:

- Client side uses the browser Gamepad API and sends only changed buttons/axes over
  the existing `/input` WebSocket.
- Host side maps the standard browser layout to an Xbox-style Linux gamepad via
  `/dev/uinput`.
- Buttons 0-16 map to A/B/X/Y, shoulders, triggers, select/start, stick buttons,
  d-pad, and mode.
- Axes 0-3 map to left/right sticks; analog trigger buttons 6/7 are mirrored to
  `ABS_Z`/`ABS_RZ` so SDL, Wine/Proton, emulators, and native Linux games can see
  proper trigger values.
- On disconnect/page close all remembered gamepad buttons and axes are released.

Host permissions:

```sh
ls -l /dev/uinput
```

For development, load the module and allow your user to open it:

```sh
sudo modprobe uinput
sudo setfacl -m u:$USER:rw /dev/uinput
```

For a package, ship a udev rule instead, for example:

```text
KERNEL=="uinput", GROUP="input", MODE="0660", OPTIONS+="static_node=uinput"
```

Then make the HyprTile user a member of the `input` group and re-login.

Why not SDL mappings in the browser path? SDL's `gamecontrollerdb.txt` is great
for native SDL clients and can still be used by games on the remote host. The
web client itself runs in the browser sandbox, where the Gamepad API already
exposes the normalized `standard` layout for most pads. HyprTile forwards that
layout as a single predictable virtual Xbox-style device on the host, which is
what most Linux games and SDL-based apps expect.

### Capture debug for black-frame setups

If the browser connects and input/audio work, but the video stays black while `grim -o OUTPUT /tmp/test.png` works, start the host with capture diagnostics:

```bash
./hyprtile-remoteviewer host \
  --output DP-3 \
  --web-bind 0.0.0.0 \
  --web-port 7780 \
  --encoder vp9-cpu \
  --audio \
  --control-output DP-3 \
  --capture-debug \
  --dump-frame /tmp/htr-frame.ppm
```

This logs the screencopy format, stride and byte statistics for the first frames and writes the first pre-encoder frame as `/tmp/htr-frame.ppm`. Open it with e.g. `viewnior /tmp/htr-frame.ppm`.

Interpretation:

* `/tmp/htr-frame.ppm` looks correct: capture and conversion are fine; investigate encoder/browser transport.
* `/tmp/htr-frame.ppm` is black: capture/copy/format handling is still wrong for that output/GPU.
* `/tmp/htr-frame.ppm` has swapped colors: format conversion needs adjustment for the announced wl_shm format.

The wlr backend now also normalizes additional 32-bit wl_shm formats (`XBGR8888`, `ABGR8888`, `RGBX8888`, `RGBA8888`, `BGRX8888`, `BGRA8888`) into the encoder's canonical BGR0 input path.


### Hyprland window isolation

Stream the window under the cursor on the active workspace by moving it to a temporary 1920x1080 headless output. `--delay` is evaluated before the cursor position and active workspace are sampled:

```sh
hyprtile-remoteviewer host --window pick --delay 3 --web-port 7777 --audio
```

Other selectors:

```sh
hyprtile-remoteviewer host --window active --web-port 7777
hyprtile-remoteviewer host --window address:0x123abc --web-port 7777
```

Useful options: `--delay 3`, `--window-size 1920x1080`, `--window-output HEADLESS-HYPRTILE-RV`, `--window-workspace 9901`. The picker intentionally ignores Hyprland's `visible` field and requires `mapped: true`, `hidden: false`, matching `workspace.id`, and half-open window bounds so split-screen borders select the right tile.
