Stable Diffusion Webgpu
The Stable Diffusion WebGPU demo is a web-based application that enables users to create images using the create-react-app framework. To access this application, users must have JavaScript enabled and use the latest version of Chrome, with the “Experimental WebAssembly” and “Experimental WebAssembly JavaScript Promise Integration (JSPI)” flags enabled.
The application executes a sequence of inference steps to generate an image, with each step taking approximately one minute, plus an additional 10 seconds for the VAE decoder to generate the image.
It’s important to note that having DevTools open can slow down the process by roughly a factor of two. The image generation process relies on the UNET model, which exclusively runs on the CPU due to its superior performance and accuracy compared to the GPU. For satisfactory results, it’s recommended to perform a minimum of 20 steps, although for demonstration purposes, three steps will suffice.
The model files are cached, eliminating the need for repeated downloads, and the application offers a user-friendly interface with options to load the model, initiate the image generation process, and view the results. Additionally, there’s an FAQ section available to address specific issues and provide troubleshooting guidance.
Despite its GPU utilization, the webgpu implementation in onnxruntime is still in its early stages, resulting in some incomplete operations. This continuous data transfer between the CPU and GPU affects performance, and multi-threading is currently unsupported. Limitations in WebAssembly prevent the creation of 64-bit memory with SharedArrayBuffer.
The developer intends to address these issues through proposed spec changes and engine patches. The source code for the Stable Diffusion WebGPU demo is accessible on GitHub, enabling users to run it locally.
Furthermore, a patched version of onnxruntime is supplied, facilitating the use of large language models with transformers.js, though its reliability in all scenarios is not guaranteed. The developer also plans to submit a pull request to the onnxruntime repository to contribute to its improvement.