Paint by Numbers: High Performance Drawing in Wasm by Sean Isom @ Wasm I/O 2024

Wasm I/O 2024 / 14-15 March, Barcelona
Slides: github.com/renderlet/wasmio-2...
Have you ever tried to use WebAssembly to draw something? Complicated, isn’t it?
We all know that WebAssembly was originally created as a way to bring compiled code into the browser.
That compiled code typically needs a UI, but none of this is built into Wasm itself. Various parts of the web stack do polyfill and supplement Wasm by exposing WebGL through Emscripten bindings or wasm-bindgen; sometimes this leads to separate web UIs that communicate with a Wasm app core via linear memory. Although this is a complicated system, it can sometimes work well enough for apps built for a browser.
But what about applications intended to run outside of the browser? In pure WASI there’s no way to access the GPU, and much of the code written for the browser is not reusable here. We could use pure Wasm to render directly into a framebuffer stored in linear memory and copy that into VRAM, but this is not very performant or scalable. We could also bypass WASI and call host functions to wrap underlying hardware or graphics APIs, but this negates the security guarantees of Wasm for arbitrary code. None of these tradeoffs are particularly scalable or lead to a reasonable developer experience.
By building the wander toolkit, we’re on a mission to build universal rendering in Wasm - and learning a lot of lessons along the way. Come listen to some of these lessons about efficiently using linear memory, orchestrating effective task level parallelism, and get an early preview of how you could access GPUs in a standardized way through WASI, bridging the gap between native and web code.

[@lienmeat]

Wow, this is amazing. Please don't give up on this project! IMO this has massive potential.

[@martinfrances107]

A really clear explanation of the problem

[@zackgreinke2382]

IT'S awesome

[@tenthlegionstudios1343]

Super cool, I am very interested in rendering with web assembly. But I had a few questions. If anyone can answer, feel free :)
So if this was running in the browser would this need to be added to chromium? I am trying to see how it tackles the issue of the initial shim/bindgen example. I work with Bevy (or wgpu), which is a Rust game engine that can run in a web worker and use wasm bindgen to render with webgpu/webgl/vulkan/Direct etc... It can also run on mostly all other platforms you would run games on.
I am trying to figure out how this is different. My impression is that it would be similar to something like bevy (or WGPU which bevy uses under the hood), except designed for the explicit purpose of rendering/communicating with the GPU from web assembly (as a module). Basically, instead of talking to any of the webgpu/opengl/vulkan, you could interface with this instead on any platform. But it still seems to have the same problem as the shim code in the browser unless web workers can directly talk to the GPU, or it is added to chromium (or w/e browser).