A wearable music interface I built with three teammates at MakeUofT 2026. Bend a finger or move your hand and it turns into live sound in the browser, streamed off an ESP32 over WebSockets.
The idea
Digital music tools are powerful but they rarely feel physical. We built RythmWear at MakeUofT 2026 to close that gap: a glove that turns hand movement into sound. Flex sensors on the fingers trigger notes, and motion data shapes pitch and dynamics, so playing it feels closer to an instrument than an app.
How it works
An ESP32 reads the flex sensors and IMU, then streams gesture data over WebSockets to a Raspberry Pi and on to the browser. A React and TypeScript front end handles calibration and sequencing, with Node, Express and ws running the real-time link and the Web Audio API doing playback. Firmware is Arduino C++ on the metal.
The hard part
Sensor noise ate most of the weekend. Raw flex readings jitter, which threw phantom notes and double-fires, so the real fix turned out to be filtering, hysteresis, and careful state logic rather than fancier hardware. The whole thing was a systems engineering problem wearing a hardware costume.
Team and what's next
Built with Ansh Shah, Mohammed Zayed, and Kishore Ramesh. Next up is adaptive calibration with per-user sensitivity profiles, richer sound design, and eventually letting players upload their own sounds and mappings. The full build log lives on Devpost through the link above.