We Won't Forget

What it is

Award-winning live cross-reality circus show — three motion-captured circus artists driving virtual characters in real time, watched simultaneously by an in-room audience and remote viewers on VR, PCs, and phone browsers. I was Technical Artist at Lifelike & Believable: I built the VR client, the AWS-hosted multiplayer audience system, the cinematic lighting, character skinning polish, and VFX work on what the audience saw on screen. The show won the SIGGRAPH 2021 Real-Time Live Audience Choice Award and two Epic MegaGrants, and became the flagship piece of the ongoing LiViCi (Live Virtual Circus) series.

The piece was a partnership between The 7 Fingers (the Montreal contemporary circus collective behind multiple Broadway productions and Olympic ceremonies), Shocap Entertainment, and Lifelike & Believable, where I was based. The performers were motion-captured in Animatrik’s volume in Burnaby, BC; the virtual scene rendered live on AWS; viewers joined from anywhere. It premiered at the 2020 Performance & XR Symposium.

What I built

Cross-platform audience client. I built the VR experience for headset users, then wrote an input abstraction layer so the same Unreal Engine project could be driven from a VR headset, a PC with keyboard/mouse, or a phone — switchable mid-session — with different game modes tailored to each input type. Same live performance, three completely different ways to watch it.

The VR client and multi-platform input layer in detail

The audience client was a single Unreal Engine project that ran in three different input modes: VR headset (headset + motion controllers), PC (keyboard / mouse / gamepad), and phone (touch + tilt). The input abstraction layer routed every camera and interaction action through a single API so the game logic didn’t need to care which mode it was running in, and so the project could switch input modes mid-session without restarting.

Each mode had its own game-mode variant: the VR mode prioritised immersion and presence (head tracking, controller-mapped interactions); the PC mode prioritised the comfortable seated/desktop experience (mouse-look, keyboard nav); the phone mode prioritised one-handed accessibility (touch + tilt, lower visual fidelity for mobile GPU budgets).

AWS-hosted multiplayer + in-show chat. I implemented the multiplayer system that ran the show on AWS instances so multiple audience members could share a session and watch the live performance together in the same virtual space. The browser-based access path meant a viewer with no special hardware — no VR headset, no gaming PC — could join from any device with a web browser, with the AWS instance doing the rendering. At the time (Unreal Engine 4.26, late 2020), Pixel Streaming wasn’t bundled in the editor — I built a custom Unreal binary from source with the Pixel Streaming plugin patches applied, plus a custom signalling-server config with my own STUN-server gist, to make the browser path possible at all. I also wrote the in-show chat so audience members could talk to each other while watching, and I packaged a PC executable for users who wanted to install locally instead of streaming through AWS.

The AWS-hosted multiplayer architecture and in-show chat

The multiplayer system ran the game on AWS instances so multiple audience members could share a session and watch the live performance together inside the same virtual space. The “browser path” was important: a viewer with no special hardware could join from any device that could open a web browser, and the AWS instance handled the heavy rendering, streaming the rendered view back to the browser.

For users who preferred a local install, I packaged a PC executable that ran the same project natively without the AWS streaming path. Both routes connected to the same multiplayer session so a VR user, a streaming-browser user, and a local-PC user could be in the same audience together.

The custom Pixel Streaming build was involved enough that I wrote it up afterwards as a LinkedIn article and a video tutorial (December 2020) so other Unreal users could replicate the setup. That work became the foundation of the Pixel Streaming Pipeline I document separately.

In-show chat let those audience members talk to each other while the performance ran — a quiet social layer on top of the show, similar to a watch-party but inside the same virtual venue.

Performance polish. I did the cinematic lighting pass for the warehouse environment (windows, lamps, lit posters, ambient and accent lights) and the skinning polish on all the performer-driven characters so they deformed cleanly through the high-contact circus choreography. I also calibrated each digital character’s scale to its performer’s real-world height so contact positions — a hand on the staff, a foot on the spool, two performers meeting — registered accurately on the virtual side.

Performance polish — cinematic lighting, character skinning, performer-to-character scaling

The cinematic lighting pass on the warehouse environment was about making the space feel like a single coherent set: window light from outside (moonlight, distant planets in the Archipelago of Dreams), interior lamps on the warehouse beams, accent light on the vintage movie posters, and ambient fill that kept the dark passages legible without flattening the shadows.

The character skinning polish covered all three performer-driven characters. Live mocap from circus performers is high-contact, high-velocity, and full-body — limbs cross the body, hands grip props, characters lift each other, contort, and balance. Skinning weights that work for a standing T-pose break under that kind of movement, so I went through each character and tuned the weights for the joints that took the worst load (shoulders, hips, spine, hands).

Performer-to-character scaling was important for accuracy. Each digital character was scaled to the height (and limb proportions where possible) of the performer driving it. That way when the performer grabbed a prop, the character grabbed it at the same place; when two performers met for a balance or a lift, the characters met cleanly without one floating above the other’s hand. Without that scaling pass the contact positions drift and the audience reads the drift instantly — the illusion breaks.

VFX. I tuned the fire-trail effect on Wukong’s staff so it left a glowing arc of sparks as he spun it (visible in the hero image up top), the disintegration particle effect that dissolves the three characters into gold dust at the climax, ray-traced reflections on the warehouse floor, smoke effects in the aerial scenes, and the integration of real-performer footage into the scene as shadows, reflections, and atmospheric presence.

The VFX pass — fire trails, disintegration, ray-traced reflections, real-performer integration

The most visible VFX work was Wukong’s staff. I tuned the fire-trail particle system so it left a coherent glowing arc as the performer spun the staff — bright enough to read against the dark warehouse interior, but not so dense that it lost the underlying staff geometry. Sparks fell from the tips as a complementary secondary effect.

The disintegration at the climax was a particle dissolve that ran on each of the three characters simultaneously. The effect needed to land as a synchronized moment — all three characters vanishing into gold dust at the same musical cue — so the timing was set against the poem’s outro rather than per-character.

Ray-traced reflections gave the warehouse floor and the polished surfaces a real-time mirror layer — visible in the trio shot just above, where the three characters reflect onto the floor along with the live-performer footage projected in the scene.

The “real performer footage integrated into the scene” was a recurring atmospheric beat — silhouettes through windows, shadows on walls, brief inserts of the actors visible behind the characters they were driving. Polishing those passes meant tuning where they appeared, how strongly they read, and how they interacted with the lighting.

Docs and packaging. I owned the audience-onboarding pipeline end-to-end. I wrote and sent the text-instruction email package that went out to every invited viewer ahead of the show, walking them through downloading and installing the PC executable and embedding links to a series of video tutorials I produced for the rest of the setup: choosing graphics settings to match their hardware, moving around the scene, and interacting with the in-show controls. On the distribution side, I set up the packaging pipeline for the PC build itself.

The collaboration

Each studio brought a different layer of the production. The 7 Fingers brought decades of contemporary circus craft (acrobatics, contortion, juggling, aerial) and the narrative direction. Shocap Entertainment brought the cross-reality production framework that turned the performance into an XR experience. Lifelike & Believable handled the real-time character and animation systems in Unreal Engine. Animatrik provided the motion-capture volume in Burnaby, BC, where the show was performed using a hybrid marker setup — active markers in the mocap suits worn under the performers’ red plaid pyjama costumes, plus passive markers on the hats and gloves for head and hand tracking.

Three Vancouver-based circus artists — Sidonie Adamson, Cameron Fraser, and Kasha Konaka — performed the piece live in the volume. Samuel Tétreault (co-founder and Artistic Director of The 7 Fingers, in Montreal) directed remotely. Athomas Goldberg (co-founder of Shocap, on-site in Burnaby) directed the technical and performance side from the volume itself.

The narrative came from Shane Koyczan’s poem “Remember How We Forgot” (from his album Remembrance Year), which also serves as the soundtrack. The audience and the performers are placed inside a “Warehouse of Forgotten Childhood Memories in an Archipelago of Dreams” — vintage Disney posters on the walls, a wooden spool as a stage prop, that moment in everyone’s life where everything is still possible.

The piece

What it became