Transformers.js runs in-browser image classification with Xenova/quickdraw-mobilevit-small for ML-powered web games

AI Impact Summary

This post demonstrates a full in-browser ML inference workflow using Transformers.js to run a finetuned MobileViT-small (Xenova/quickdraw-mobilevit-small) via ONNX Runtime for real-time image classification in a web game. It covers end-to-end steps from training on the Quick, Draw! dataset to converting to ONNX with Optimum and executing in a Web Worker for non-blocking, ~60 predictions per second performance. For engineering leadership, this enables server-free gameplay with lower backend load and latency, but requires careful management of model size (~20 MB), browser resource usage, and cross-browser compatibility.