WebGPU notes   /   09

WebGPU HTML Mesh in Rust: Web Pages on a 3D Plane

Rasterize a small HTML document into 1024×576 RGBA pixels, upload those pixels to WebGPU, draw them on a tilted plane, and map pointer, touch, wheel, and keyboard input back into page coordinates.

By Pooya Eimandar·Rust / wgpu / WGSL

Page texture

1024×576

Shatter tiles

2,304

Refresh choices

15 / 30 / 60 fps

Put HTML on a WebGPU plane

The previous 3D text mesh example converts font outlines into vertices. HTML Mesh takes a different route: a browser or canvas rasterizer produces ordinary page pixels, and WebGPU treats the result as one sampled texture. The GPU never parses HTML, executes JavaScript, or lays out CSS.

The bundled page.html is a 2,295-byte document with a heading, a 280×76 button, a counter, CSS, and a short click handler. The original HTML Mesh commit introduced the page, renderer, shader, and screenshots. A later shatter-effect commit added the tiled geometry and shader motion. The current Rust example runs through sib::render.

The plane keeps the page’s 16:9 aspect ratio: its half-height is 1 and its half-width is 1024 / 576. A fixed camera at (0, 0.12, 4.25) looks near the origin through a 46° perspective projection. The model rotates −14° around Y and 4° around X, which makes both the 3D placement and the inverse hit-test visible without adding camera controls.

Rasterize the document into RGBA pixels

HtmlSurface owns a 2,359,296-byte CPU pixel vector: 1024 * 576 * 4. It also tracks the active source, a counter, hover/press/focus state, state and bitmap generations, pending work, and whether the GPU texture is dirty. Initialization draws a small software fallback before an asynchronous backend can replace it.

The word HTML covers several different implementations here. The current backends are deliberately explicit:

Scroll sideways to see all table columns.

For the WebAssembly memory path, the inline JavaScript parses the markup, reads the first heading, button, and status, summarizes selected body text, and paints a fixed card on a Canvas 2D context. Arbitrary CSS layout, images, transforms, video, and page scripts are therefore not reproduced by that path. The native macOS webview and the server-side browser URL path do execute real page content.

The WebAssembly URL endpoints live in the current local server source. Requests are accepted only when the HTTP Host and any Origin header identify loopback, because the API can navigate a browser and inject input. This is a local demo service, not a general public screenshot API.

Upload the page texture

The example creates one sampled 2D texture with linear minification and magnification. When texture_dirty is set, upload_html_if_dirty copies the complete CPU image with a 4,096-byte row pitch:

context.queue.write_texture(
    wgpu::TexelCopyTextureInfo {
        texture: &html_texture.texture,
        mip_level: 0,
        origin: wgpu::Origin3d::ZERO,
        aspect: wgpu::TextureAspect::All,
    },
    &html.rgba,
    wgpu::TexelCopyBufferLayout {
        offset: 0,
        bytes_per_row: Some(SURFACE_WIDTH * 4),
        rows_per_image: Some(SURFACE_HEIGHT),
    },
    html_texture.size,
);

Every accepted snapshot therefore uploads the full 2.25 MiB logical image; there is no dirty rectangle, compression on the GPU copy, mip chain, or partial update. Choosing a 15, 30, or 60 fps live refresh target can request roughly 33.75, 67.5, or 135 MiB/s of unpadded CPU-to-texture payload before transport, browser capture, and driver overhead. A pending capture prevents overlapping refresh work, so those buttons are targets rather than guaranteed rates.

Scroll sideways to see all table columns.

One bind group exposes the uniform buffer at binding 0, page texture at binding 1, and sampler at binding 2. The fragment shader samples a single texel color, adds directional diffuse and a view-dependent Fresnel accent, and outputs alpha blending. The pipeline has depth writes with LessEqual, no face culling, one sample, and no post-processing target.

Ray-map input into page coordinates

A tilted page needs more than scaling window coordinates. pointer_uv converts the cursor from framebuffer pixels to normalized device coordinates, unprojects near and far points with the inverse view-projection matrix, and forms a world-space ray. It then transforms that ray by the inverse plane model and intersects local Z = 0:

let model_inverse = plane_model().inverse();
let local_origin = model_inverse.transform_point3(ray_origin);
let local_direction = model_inverse.transform_vector3(ray_direction);
let t = -local_origin.z / local_direction.z;
let hit = local_origin + local_direction * t;

Some(glam::Vec2::new(
    (hit.x / PLANE_HALF_WIDTH + 1.0) * 0.5,
    1.0 - (hit.y / PLANE_HALF_HEIGHT + 1.0) * 0.5,
))

The UV is converted to a clamped 1024×576 page position. Mouse clicks, one active touch, wheel deltas, selected keyboard keys, and text can then be forwarded to the active document. The egui panel receives each window event first, so operating the panel does not also click the page behind it.

The bundled memory fallback exposes only one hit rectangle, from pixel (372, 250) to (652, 326), matching the button. Its unused state model still contains name, toggle, and slider variants, but elements() returns only the button. Tab always focuses that button; Enter or Space activates it; clicking increments a saturating u32 counter.

Hit testing always uses the original flat model. When the shatter animation displaces visible tiles, input does not follow each moved shard. That mismatch is useful to understand before treating the technique as a production 3D browser surface.

Switch from one quad to 2,304 shards

The shatter option switches buffers, not pipelines. Rust subdivides the same 16:9 rectangle into 64 columns and 36 rows. Every tile owns four vertices and six indices, so adjacent tiles deliberately do not share vertices. Each vertex carries position, UV, normal, and the tile-center shard coordinate in a 40-byte stride.

The vertex shader hashes that tile coordinate, weights tiles near the top of the page, and repeats a rising motion with a per-tile phase offset:

let top_weight = 1.0 - smoothstep(0.0, 0.62, input.shard.y);
let seed = hash2(input.shard * vec2<f32>(37.0, 91.0));
let rise = fract(uniforms.effect.x * 0.42 + seed * 0.31);
let lift = top_weight * rise;

local_position.x += enabled * x_dir * lift * 0.18;
local_position.y += enabled * lift * 0.72;
local_position.z += enabled * z_dir * lift * 0.16;

Animation time advances in seconds only while shatter is enabled. The fract(time * 0.42 + ...) term repeats about every 2.38 seconds, with deterministic offsets from each tile’s seed. Fragment shading adds green crack lines near tile boundaries in the upper region and fades those tiles. The current fade multiplier is 0.88, so alpha remains at least 0.12; the shader’s alpha < 0.02 discard guard is not reached with these values.

Compare native and WebAssembly backends

The rendering pipeline is shared, but page acquisition differs substantially. Native startup reads assets/htmlmesh/page.html synchronously and creates a wry child webview. On macOS, WKWebView snapshots are converted to 1024×576 RGBA, including channel conversion and resizing where necessary. Page loads schedule a capture after 120 ms; pointer and keyboard changes normally schedule one after 16 ms, while unpressed pointer moves use 33 ms.

WebAssembly fetches the same page asset asynchronously. Memory mode uses the local Canvas 2D painter. URL mode calls the local server, which starts or reuses one headless Chromium-family browser session for the requested URL and dimensions. Up to 16 pending inputs are retained; adjacent text events are joined and consecutive mouse moves are replaced by the newest one.

Scroll sideways to see all table columns.

Neither backend embeds a browser framebuffer directly in WebGPU memory. Captures pass through CPU-visible encoded or RGBA pixels, then a full queue.write_texture. Video, rapid animation, and high-frequency scrolling therefore amplify browser capture, transport, decode, and upload cost.

Inspect passes and controls

A rendered frame first updates a dirty HTML texture, rewrites the 160-byte uniform, and draws one selected mesh. It then asks egui to prepare and render its panel while preserving the scene color.

Scroll sideways to see all table columns.

The example records two render passes in its main encoder. Egui may also return auxiliary command buffers while updating its own textures and buffers. There is no fixed glyph-draw count, no MSAA resolve, and no intermediate scene color target.

Resizing rebuilds only the surface-size depth texture and projection uniform. The source texture remains 1024×576, so its browser page resolution and memory cost do not follow window size.

Run and extend the example

From a local checkout with Rust installed, run the native example:

cargo run --example htmlmesh

On macOS this path can capture the live wry page. Other native targets currently need their own webview-to-RGBA implementation for live texture updates.

For WebAssembly, install the wasm32-unknown-unknown target and the wasm-bindgen CLI version matching Cargo.lock, then build and use the repository server:

scripts/build-wasm.sh --release htmlmesh
cargo run --bin serve

Open http://127.0.0.1:8080/htmlmesh/ in a WebGPU-capable browser. The memory page works with the Canvas 2D summary rasterizer. URL capture additionally needs a supported Chromium-family executable; the server error suggests setting WEBGPU_HTMLMESH_BROWSER when automatic discovery fails. The loopback guard intentionally prevents that API from becoming a remote service.

Useful source experiments include:

• Upload dirty rectangles instead of 2.25 MiB on every small page change, then measure bandwidth at each refresh target.
• Replace the fixed Canvas 2D memory painter with a general raster path and compare the resulting CSS coverage.
• Make hit testing follow displaced shard geometry, or disable page interaction while the shatter effect is active.
• Add mipmaps and anisotropic filtering, then view the plane at steeper angles.
• Implement native capture outside macOS and keep the same HtmlSurface texture contract.

This example is a bridge between browser rendering and GPU composition, not an embeddable browser engine. Its fixed resolution, whole-image transfers, platform-specific capture, and local headless-browser service are the main constraints to solve before using the idea for production UI.