Validation record

Validated locally on 2026-09-12 with the installed Unreal Engine 5.8 Windows toolchain, .NET SDK 9.0.314, Livekit.Rtc.Dotnet 0.1.2, and isolated LiveKit server 1.13.3.

Automated checks

dotnet run --project Bridge/BPUnrealRTC.Bridge.csproj -c Release -- --self-test

  • Fill/mask split and compositing, missing/stale masks and new compositor generation.

  • LAN-first and Cloud-first RTC URL selection, URL validation/deduplication.

  • Bounded jittered backoff and latest-frame queue pressure.

  • Oversized/negative/truncated wire payload rejection before allocation.

  • 50 real loopback socket cycles with byte-by-byte fragmentation and mid-frame peer death.

  • Cloud code redeemed once, token retained, reverse-proxy prefix preserved, and 401 surfaced for replacement credentials.

  • Facial weight validation, OSC float decoding and truncation, ArtDmx decoding and truncation.

The backoff stress loop accounts for most of the 10,070 assertions; this is not 10,070 distinct test cases.

dotnet run --project Bridge/BPUnrealRTC.Bridge.csproj -c Release -- --integration-test <path-to-livekit-server.exe>

Starts a private loopback LiveKit server and a test HTTP server implementing the ShowServer join/preview contract. Uses isolated test credentials; never reads venue credentials. Sends real encoded video and mask, nonzero stereo PCM and topic data through the SDK. Intentionally advertises an unreachable direct URL before the reachable URL, kills/restarts the SFU, returns HTTP 503 from the test ShowServer, restores service, and checks media/data return and clean session deletion.

Observed extended run: 515 video frames, 1,724 nonzero audio frames, 953 data packets, 508 keyed frames; 4 HTTP joins and 1 session deletion. Frame counts vary with scheduling. The initial successful run caught and corrected an SDK distinction between track names and publication names for mask identification.

Scripts/Build.ps1 runs Unreal’s BuildPlugin for UnrealEditor Development, UnrealGame Development and UnrealGame Shipping, including UHT and companion runtime staging.

Unreal’s Python commandlet created and reopened the included demo map, verifying one native BPUnrealRTCCamera, its BPUnrealRTCComponent, the assigned DMX light, and five scene actors. This editor load test used NullRHI; it does not measure rendered-frame performance.

Boundaries

The HTTP portion of the isolated integration test is a contract fixture, not a running venue/cloud deployment. The following require the actual project and venue acceptance run: EC2 GPU performance and driver behavior, WAN/Cloud ICE-TCP routing, 1080p/4K soak, end-to-end NDI output and alpha quality, audio/video sync, proprietary facial tracker conversion, and actual rig/fixture bindings. See EC2 deployment for the acceptance checklist. No claim of audience-ready latency or completed AWS deployment is made from local checks.

Live bidirectional studio sample (2026-09-12)

The new BPUnrealRTCBidirectional map was exercised in Unreal Editor 5.8 with a real, isolated ShowServer built from this repository, its LiveKit 1.13.3 SFU, and the repository’s BPMediaNode engine. Initial runs configured the MediaNode engine through its local API. A subsequent real Desktop UI walkthrough configured IN 1 as an FFmpeg movie and OUT 1 as WebRTC with separate audio/video routes; four UI captures are included in MediaNodeSetup.md.

  • Big Buck Bunny was decoded by BPMediaNode, published to ShowServer, received onto an Unreal virtual monitor, rendered by the outgoing scene capture, and decoded again by a second BPMediaNode WebRTC input through ShowServer. Actual frames are in Docs/images.

  • Stopping the Bunny WebRTC output cleared the virtual screen. Restarting that output restored the picture without restarting Unreal. source-interrupted.png and source-recovered.png record that run.

  • Terminating the running native RTC companion caused it to restart automatically (new process observed one second later). Incoming and returned video resumed in the same Unreal process; companion-recovered.png records the returned picture. This is one local crash-recovery check, not a long-duration soak.

  • Incoming Unreal PCM had nonzero peaks. Unreal playback/submix return remained silent on this host, including a diagnostic imported tone routed to the same submix. The engine reported no default audio device and used its null callback. This was the initial run; the later RDP/audio investigation below supersedes its root-cause status.

  • The sample map verification script passed: nine editor actors, expected camera/screen bindings, material, submix, rotations, no automatic connection and no saved source ID. Diagnostic assets and credentials are not distributed.

  • Native automation BPUnrealRTC.Transport.IdleCompanion passed. It launches the real companion against a deliberately unavailable HTTP endpoint and verifies that idle local transport stays connected for five seconds. This caught an actual Unreal socket behavior: a successful nonblocking receive with zero bytes can mean would-block, not EOF. The worker now preserves that idle connection.

  • The updated plugin built for UnrealEditor Development, UnrealGame Development and UnrealGame Shipping. Companion self-tests passed again: 10,070 assertions and 50 socket chaos cycles.

These checks do not validate Cloud/EC2, NDI egress, alpha rendering quality, audio/video synchronization, or desktop UI documentation. See Validation for the missing real UI captures and Remote / Cloud studio for the remaining deployment acceptance work.

File playback and audio investigation (2026-09-12 follow-up)

  • Found accelerated playback in MediaNode’s FFmpeg file ingest: it decoded at disk speed while the latest-frame consumer skipped ahead. Local movie inputs now use -re; network/listener inputs retain their existing timing. All 11 FfmpegFilePlaybackTests passed, including a real FFmpeg test measuring 72 frame intervals from a 24 fps movie at approximately three seconds.

  • Added native diagnostics for actual procedural-wave consumption: Received Audio Mixer Peak, Received Audio Buffered Ms, and Received Audio Rendered Samples. All three Unreal build targets passed after adding the SoundWave subclass and AudioExtensions dependency.

  • Active RDP exposed a Remote Audio device. A diagnostic 440 Hz source produced incoming/mixer peaks around 0.25. With background audio disabled, switching away from Unreal produced a zero captured-submix peak. Unreal’s EditorEngine explicitly applies a zero volume multiplier when unfocused unless bAllowBackgroundAudio is enabled.

  • Restarted the isolated editor with Allow Background Audio enabled and background CPU throttling disabled. Ten samples over 18 seconds showed nonzero input (~0.25), consumed PCM (~0.25), captured submix (~0.177), and MediaNode’s decoded ShowServer return (~0.17) while the editor ran in the background. This confirms the audio route beyond just a network meter. The diagnostic tone was then removed from the route and the original movie audio restored.

  • One earlier subscriber restart was needed after repeated Unreal restarts; later editor restart recovered audio without that intervention. This is insufficient evidence to attribute a second transport defect. Long-running restart and A/V synchronization tests remain open.

  • The active RDP desktop allowed four real MediaNode UI screenshots. Later monitor capture failed with 0x80070057, blocking the remaining Unreal UI captures. A D3D12 editor launch also crashed in the graphics runtime; this isolated test ran with D3D11. Neither observation certifies operation after RDP disconnect or on EC2.

The sample now defaults to background audio enabled and background CPU throttling disabled. Existing projects must set these editor preferences themselves; the runtime plugin does not change global editor preferences. See the project-first guide and screenshot capture checklist for the remaining documentation work.

Audio format-change limitation found during restoration

Changing the live MediaNode route from the stereo diagnostic tone to the mono FFmpeg input stopped incoming audio until the WebRTC output was restarted. LiveKitContributePublisher.PumpAudioAsync creates its AudioSource using the first frame’s format, but submits subsequent frames with their original channel count. The route change therefore needs a publisher restart with the current implementation. A receiver meter also retained its old last-buffer peak until that monitoring input was restarted; a meter alone cannot prove current playback. These limitations remain open rather than being covered by the successful constant-format tone test.

Real desktop CineCamera and control validation (2026-09-12)

A standard CineCameraActor was added through the UE 5.8 editor. Cine Capture Component 2D was parented to its CameraComponent, and BPUnrealRTC was added to the same actor. Both components used RT_ShowServerCamera, an RGBA8 1280 × 720 target created in the editor. The supplied BPUnrealRTCCamera was not the video publisher in this run.

  • The CineCamera received Big Buck Bunny from the real local ShowServer onto a material surface, rendered the studio, and published it directly back to ShowServer. The WebRTC Tracks page decoded the rendered return. showserver-cinecamera-return.png is an actual browser UI capture.

  • Incoming audio, procedural-wave consumption and the program capture submix produced varying nonzero levels. A MediaNode return meter initially retained a constant value; that alone is not accepted as proof of continuing downstream audio. The fresh-engine format-switch run below records this separately.

  • A separate subscription used the included loopback control gateway and the actual local ShowServer. Unreal’s typed events received 5 facial JSON envelopes, 5 OSC position commands and 6 ArtDmx messages. The sample actor moved from X=-900 to X=-846 cm. Its light changed from intensity 1000 to approximately 501.96 for DMX value 128, then returned to 1000 for value 255. Facial payload reception was measured; no facial mesh or proprietary tracker was exercised.

  • The CineCamera RTC component sent reliable data back to ShowServer. The actual browser data panel displayed CineCamera real desktop E2E; showserver-cinecamera-data.png records the UI. The screen was black at that instant, so that image demonstrates outbound scene/data reception, not simultaneous movie-frame continuity.

  • Local test automation configured runtime credentials from the process environment and collected diagnostics. The end-user guide uses project components and runtime Blueprint input; it does not require this diagnostic script. No credentials, diagnostic project or movie file are included in the plugin package.

  • This session used an RTX 4090 with D3D11. The earlier editor left across the RDP transition had crashed, and a task-owned .NET helper later displayed an error. Fresh editor launches worked after helper cleanup. These observations do not establish D3D12 or EC2 reliability.

The remaining native screenshots were interrupted by Windows’ system-owned .NET error and camera-permission dialogs. Existing editor captures and browser captures are genuine; no prompts have been edited out and no missing screenshots have been fabricated. Cloud, AWS GPU deployment, NDI alpha quality, frame pacing under load, long-duration reconnection and A/V synchronization remain acceptance work.

Audio format switching and independent return decoder

MediaNode’s WebRTC publisher now normalizes PCM to a fixed 48 kHz stereo format before submitting frames to its AudioSource. Mono is duplicated to both channels; multichannel inputs use the first two channels. The existing streaming linear resampler handles sample-rate conversion and resets interpolation history when the incoming format changes. This fixes the previously recorded stereo/mono route-change failure; it does not restore stereo separation to a mono FFmpeg ingest.

  • 15 targeted tests passed: four audio-normalizer cases (including two rate variants) and eleven FFmpeg pacing checks. Rate tests cover 44.1 and 96 kHz and approximately one second of duration, channel tests cover stereo→mono→stereo, and invalid/NaN PCM is checked.

  • The rebuilt MediaNode engine switched the same live WebRTC output between a stereo 440 Hz tone and the mono movie twice, without restarting the output or Unreal. Tone receive/mixer peaks were approximately 0.25 and captured-program peaks approximately 0.177; movie PCM resumed between switches.

  • An independent companion-based RTC decoder, separate from the MediaNode monitor, received the Unreal return for 25 seconds. It recorded 1,184,640 stereo sample frames at 48 kHz (24.68 seconds after connection startup), with nonzero tone audio throughout the full observed seconds. Video reception was approximately 27–29 fps and RTC data continued arriving. A subsequent movie run also received continuously changing nonzero audio.

  • Both native companions were terminated while Unreal remained running. Its publisher and subscriber returned to Connected. Another 5 facial, 5 OSC and 6 DMX messages were applied afterward, bringing totals to 10/10/12. The independent recording overlapped the interruption near its end; it is not used to claim an exact recovery-time measurement.

The MediaNode monitoring input’s peak remained frozen even when the independent decoder received changing audio. That monitoring defect remains open. Intermittent movie-video gaps also triggered the plugin’s 500 ms stale-frame clearing; stable outgoing frame cadence does not prove continuous upstream movie frames. These are explicit remaining limitations, not a completed production soak.

Headless movie-video demand

The companion now explicitly enables video publications and requests a high, program-sized layer, using the same SDK-specific headless subscription mechanism already present in ShowServer. A 25-second comparison on the movie source received approximately 11–14 video frames per full second without the zero-frame full-second gap seen in the preceding run. This is an improvement, not proof of the configured 24 fps. The test feed’s remaining frame-rate limitation is open. Native stale-frame clearing remains enabled.

Companion self-tests passed again after this change: 10,070 assertions and 50 socket chaos cycles.

The final isolated SFU/API chaos run passed with the headless adjustment: 503 video frames, 1,706 nonzero audio frames, 935 data packets, 502 keyed frames, 5 HTTP joins and 1 clean session deletion. These are observed counts, not distinct test cases; the HTTP endpoint in this test remains a contract fixture.

Physical-desktop resumption

Native editor input and screenshots recovered after the user returned from RDP. Unreal logged a successful switch to the physical DELL display audio device at 48 kHz stereo. Real captures now cover direct connection settings, audio submixes, material nodes, surface bindings, the live viewport, MediaNode return-source selection and its rendered preview. The native CineCamera Level Blueprint with a runtime ConnectRequested event compiled successfully; a complete login-widget/operator pass is still outstanding.

A fresh independent return decoder initially received approximately 27–28 video frames per full second and RTC data, but only near-silent PCM. A known 0.25-peak test tone did not fix that silence. A separate subscriber to the upstream MediaNode source received exactly zero PCM while its engine routing meter reported 0.25. Restarting that WebRTC output restored Unreal input/mixer peaks near 0.25 and captured peaks near 0.177. This isolates the symptom upstream of Unreal playback, but does not identify the underlying sender failure.

The subsequent independent return recording received tone peaks near 0.1784 for its first 20 full seconds, with approximately 24–26 video fps. Its final seconds contained a gap overlapping material-editor activity, so this recording is not a clean uninterrupted 25-second acceptance pass. The movie audio route was restored afterward. Evidence is retained locally as physical-return-movie.json, physical-return-tone-before.json, physical-upstream-tone-before.json and physical-return-tone-recovered.json. These diagnostic files and credentials are not part of the distributed plugin.

The sender recovery gap is addressed below. The exact initiating exception in the earlier silent session was not retained at the prior debug log level. Input movie frame-rate limitation, frozen MediaNode monitoring meter, Cloud/EC2 deployment, venue NDI alpha output and long production soak remain open. RDP recovery and successful component/Blueprint screenshots do not close those acceptance items.

Publisher pump recovery

The previous MediaNode publisher swallowed media-pump exceptions and continued reporting a healthy room. The graph recovery loop consequently skipped a publisher whose audio pump had permanently exited. Audio capture also omitted shutdown cancellation, and teardown did not join pumps or dispose audio handles.

The corrected implementation latches pump failure independently of signaling state, reports the failure at warning level, bounds audio capture to eight seconds, propagates cancellation, joins pumps during teardown and releases audio/video source and track handles. The existing one-second graph recovery loop replaces the failed publisher while preserving source identity and routing. Legitimate silence or an unrouted audio source does not trigger recovery.

On September 12, 2026, 50 targeted MediaNode tests passed, including actual ShowServer + native LiveKit publication/subscription tests. Each fault was injected at the production audio capture boundary after an independent subscriber decoded nonzero tone:

| Injected failure | Fresh decoded tone returned | Publisher replacements |

| --- | --- | --- |

| Capture exception | 1.09 seconds | 1 |

| Capture stalled until cancellation | 9.05 seconds | 1 |

| Capture stalled and ignored cancellation | 9.71 seconds | 1 |

Each test then required five seconds of fresh decoded nonzero PCM without another replacement. The suite also covers audio format changes, paced file playback, graph reconnection and cancellation. These are local recovery measurements, not Cloud/EC2 latency guarantees or a production-duration soak. Test evidence is retained locally in publisher-recovery-suite.log and test-results/publisher-recovery.trx. The opt-in native tests require BPSHOW_RECOVERY_URL and BPSHOW_RECOVERY_TOKEN_FILE, with isolated source identities and NDI egress disabled; credentials are never included in test output.

Concurrent Unreal audio reset

The subsequent live check exposed an Unreal access violation in BPUnrealRTCComponent::TickComponent while managing received PCM. The plugin called USoundWaveProcedural::ResetAudio from the game thread on overflow; Unreal 5.8’s implementation empties the same single-consumer queue that the audio thread dequeues during GeneratePCMData. This also explains why the previous buffered-audio diagnostic could become negative during resets.

Received PCM now uses a synchronized, 100 ms bounded stereo buffer. Overflow discards the oldest complete stereo frames, underrun supplies silence, and reset/playback share the same lock. The native regression test BPUnrealRTC.Audio.ConcurrentResetAndPlayback exercises 20,000 writes/resets concurrently with 20,000 reads, checking frame integrity, silence and bounded nonnegative counts.

The final plugin package built successfully for Unreal 5.8 Editor Development, Game Development and Game Shipping. Both native automation tests passed against that final package (ConcurrentResetAndPlayback and IdleCompanion). The video surface uses the new buffer’s availability when starting playback.

A subsequent 90-second real-desktop run kept the same Unreal process alive through three intentional upstream publisher stops, each followed by a start three seconds later. A separate native companion decoded the Unreal camera/submix return throughout:

  • The return delivered 25–28 video frames per full second across all 89 complete measurement seconds, including upstream outages; this measures the rendered studio return, not the incoming movie’s frame rate.

  • Nonzero returned tone reappeared in the first complete one-second bucket following each restart (restart buckets 13, 33, 53; nonzero buckets 14, 34, 54).

  • The final 34 complete seconds delivered 26–28 video fps and 98–102 audio frames per second, with nonzero PCM in every bucket.

  • The decoder received 45 outbound data messages. Unreal also recorded five OSC, six DMX and five facial-control messages from the restored gateway.

  • Received-audio buffering remained between 0 and 100 ms; no crash occurred. The movie audio route was restored after the test.

Evidence remains local in audio-buffer-build3.log, audio-buffer-native-final.log, recovery-live-chaos.json/.wav, recovery-live-summary.json and Unreal/Saved/cine-controls.json. The earlier failed live run and crash log are retained separately. This closes the demonstrated pump-recovery gap and concurrent audio-queue crash path; the earlier unlogged capture trigger and long-duration production soak remain unverified.