F2F Followup: Render Loop Proposal

[toji]

A persistent issue that has come up with WebVR as it exists today is the details of how the render loop works. Things like when the frame data is allowed to update and what should happen if submitFrame is not called have remained somewhat stubbornly unresolved. Additionally, there's calls from the web platform side to tie the canvas presentation mechanism more closely to the commit() or transferToImageBitmap() patterns being established by OffscreenCanvas. This proposal attempts to give us a basic frame flow that we can work off of, and usefully extend in the future.

Issues being addressed:

• Define when VRFrameData is updated (When is getFrameData/getPose allowed to query new values when not presenting? #81, Clarify ambiguity regarding when VRFrameData is refreshed #112)
• Clarify behavior around submitFrame (Describe what happens when submitFrame is not called every frame. #69, Clarify what happens when submitFrame is called at the wrong time #138, Feedback: VRDisplay:submitFrame() should be dropped #170)
• Restructure how canvas rendering is presented to the display (bitmaprenderer instead of submitFrame? #141, Feedback: VRDisplay:submitFrame() should be dropped #170, Proposal: Remove vrDisplay.submitFrame(), vrDisplay.requestAnimationFrame and use OffscreenCanvas & commit() by default. #174)
• Resolve potential confusion with multiple requestAnimationFrame variants. (Feedback: Duplicate requestAnimationFrame API is confusing #171)

Overview:

The first big change proposed here is to get rid of VRDisplay.requestAnimationFrame and VRDisplay.submitFrame and combine them into a single, promise-returning call: VRSession.commit. commmit would function like submitFrame does today, taking any dirty canvases from the layer list and pushing their contents to the VR compositor. It's important that we can continue doing this explicitly so that the render loop can dispatch it's rendering work as soon as possible to reduce latency, then continue doing other work (rendering a mirrored view, simulation logic, etc.) in the same loop. The promise returned by commit resolves when the next frame is ready to be rendered, effectively allowing it to take the place of requestAnimationFrame as well.

(For the record this isn't too far off from what's being discussed for OffscreenCanvas)

To pre-emptively answer the question of why we wouldn't just want to use gl.commit() for this, as suggested by #174: We do plan on supporting multiple layers, each potentially with their own canvas eventually. In that scenario how would you update them all in sync? Which commit do you use to drive the render loop? When does the context commit switch from running at the monitor framerate to the VRDisplay framerate? Having the commit reside with the VR system that's using it rather than one of it's sources appears to be a more consistent solution.

As for frame lifetime the WebVR spec currently defines that the VRFrameData returned is the same between calls to submitFrame. This is so that apps composed of multiple libraries that all query their own frame data are in sync. This was problematic because submitFrame wasn't always called, such as in the case of magic window uses. This proposal changes that to say that the frame data is update with each commit resolve. So in essence the app must pump the render loop in order to continue receiving new frame data. This works better than it would have previously due to the proposal in #179 that makes even magic window mode into an explicit session that must be started and ended. So now any use of VRFrameData can be considered to occur while "presenting" (with different platforms able to make different choices about when that's allowed.)

I would expect that on many systems (like Chrome) the implementation will actually resolve the commit promise when new frame data has been made available, possibly over an IPC channel, but other platforms may worry that retrieving the frame data at the beginning of the callback like that may introduce unwanted latency. For those platforms they can simply defer fetching the frame data until the first call to getFrameData, and only reset the dirty bit on that data when the next requestVRFrame callback is ready.

Side note: While it's not part of this proposal future API revisions could also introduce an 'updateFramePrediciton' function as has been suggested by Microsoft in the past to force a mid-frame frame data refresh if we felt it was necessary.

Combined these changes create a more tightly controlled render loop with more predictable behavior that's harder to use "wrong". Hopefully that outweighs a couple of minor oddities listed below.

Rough IDL:

partial interface VRSession {
  Promise<DOMHighResTimeStamp> commit();
};

Example usage:

let vrSession = null;

function RenderLoop() {
  if (vrSession) {
    // VR path
    let frameData = vrSession.getFrameData(frameOfReference);
    DrawVRFrame(frameData);
    vrSession.commit().then(RenderLoop);
  } else {
    // Non-VR path
    DrawFrame();
    window.requestAnimationFrame(RenderLoop);
  }
  UpdatePhysicsOrSomething();
}

// Kick off non-VR rendering
window.requestAnimationFrame(RenderLoop);

function onEnterVRClick() {
  vrDisplay.requestSession().then(session => {
    vrSession = session;
    SetUpLayers();
    RenderLoop();
  });
}

Also, one really nice side effect of using promises is that with async/await support we can get something that looks an awful lot like a more traditional native game loop, complete with blocking swap!

async function RenderLoop() {
  do {
    // Render render render
  } while (await vrSession.commit())
}

Oustanding questions:

• If we use a promise for the rAF mechanism it doesn't leave us with an analog for cancelAnimationFrame. Do we care?
• What happens if you call commit multiple times? Reasonable options are to reject the promise or silently discard the rendering and resolve with the previously scheduled frame.
• Starting up a render loop that's been paused looks a little weird in this model. You'd call commit() without necessarily having rendered anything. Is that a problem?

[RafaelCintron]

One of the essential aspects about RAF is that it allows the user agent to throttle queuing of unnecessary work. For instance, the traditional RAF associated with the window is not raised when the document is on a hidden tab, minimized, or otherwise invisible. In Edge, if the GPU is bogged down with work, we throttle the RAF so that the GPU can keep up with the work the CPU is providing it. Failure to do this can lead to poor performance as frames are dropped on the floor and prevent the CPU/GPU from rendering other windows in the system in a timely manner. The nice thing about RAF is that it is a "pit of success" when it comes to expectations around frames. One callback == one frame; no more, no less. You can't get into a situation where you queue more than one frame of work because the API simply won't let you.

Given this, it is essential that we prevent commit from queuing more than one frame of work during JavaScript execution. Rejecting the promise is one avenue. I would prefer that we throw an exception to make it clear to the developer that they're doing it wrong. Dropping frames means wasting CPU/GPU cycles for frames the user never sees, a problem we do not have with today's approach.

Another disadvantage of the new approach is there is no way to signal the last frame or an "I am done now". Committing the last frame means you're forced to sign up for another one. I suppose you can call exitPresent right after you call Commit, but we still have to run the Commit's promise rejection code unnecessarily. The current approach also does not have this problem.

I agree having a similar look to the traditional game loop is attractive but the new approach seems to create more problems than it solves. What am I missing?

[toji]

Thanks for the speedy feedback! I figured this one would spur some good discussion. :)

First off, I'd like to point out that I consider there to be effectively two different but related proposals here: The frame data lifetime and the render loop shape. I totally understand having concerns about the commit proposal, but I'd also like to know if you have any concerns about tying the frame data's lifetime to the render loop, whether that be done via rAF or a promise?

Putting that aside, though, I'm not sure if I'm clear on the potential issues with a promise-based loop.

In Edge, if the GPU is bogged down with work, we throttle the RAF so that the GPU can keep up with the work the CPU is providing it.

Is there a reason I'm not seeing that this can't be done with a promise? We can delay resolving it indefinitely.

You can't get into a situation where you queue more than one frame of work because the API simply won't let you.

With rAF today you can't queue up multiple frames but you can queue up multiple callbacks per frame by calling rAF repeatedly. What I was suggesting was that repeated calls to commit() could act the same way. Just queue up multiple promises to be executed on the next display vsync. So:

vrSession.commit().then(funcA);
vrSession.commit().then(funcB);
vrSession.commit().then(funcC);

Is equivalent to:

requestAnimationFrame(funcA);
requestAnimationFrame(funcB);
requestAnimationFrame(funcC);

Actually we could probably have the commit variant even return the same promise object for each call (for the same frame, anyway). Not sure if that provides a concrete benefit, but it's architecturally nice.

I guess the bigger question, and what you may have been getting at, is what do if the developer does something like this:

DrawSomeStuffToTheLayer();
vrSession.commit();
DrawSomeMoreStuff();
vrSession.commit().then(renderLoop);

But we've already established that to be a problem with the current rAF/submitFrame pattern anyway. That's the scenario that I suggested silently dropping the rendering work done for (but still returning the existing outstanding promise so that nobody misses a frame). Console warnings are probably a more appropriate mechanism for communicating the undesired behavior at that point.

Another disadvantage of the new approach is there is no way to signal the last frame or an "I am done now". Committing the last frame means you're forced to sign up for another one. I suppose you can call exitPresent right after you call Commit, but we still have to run the Commit's promise rejection code unnecessarily. The current approach also does not have this problem.

Actually, now that you bring that up I'm suddenly concerned about a flaw in the current system. If we call vrDisplay.requestAnimationFrame and then vrDisplay.exitPresent immediately after we've effectively dropped that callback. That seems bad. I'd actually prefer and explicit reject to happen.

The concept of "last frame" is a funny one anyway, because I have a hard time imagining a scenario where commit(); exit(); makes sense. I guess maybe if you had a video and wanted to boot the user from VR mode on the last frame? (That sounds unpleasant.) I think as a developer I would consider this to be a benign race, with no guarantee that the frame would actually be shown before presentation ended. But most of the time the exit signal is going to be triggered by some event or other async input, which means you'll probably discover that you want to end the session while a frame is outstanding, and I don't expect anyone to catch that signal, wait for the next render loop cycle, draw a full frame, and THEN end the session.

Anyway, I'm not actually trying to argue that this is unquestionably better than a traditional rAF, I'd just like to understand your points a bit better. I think we could evaluate a few different options here as well. For example, we could keep the submit and rAF as a single call but do it callback-based if the use of promises prove to be a sticking point. We could also discuss having an implicit submit at the end of the animation loop callback, WebGL style, but that making mirroring trickier AND makes controlling latency harder, so I've been assuming the other UAs would not be in favor of it.

[AlbertoElias]

I like the idea of using Promises and I think it could emulate a the benefits offered by raf currently. Also, I believe cancellable promises have been worked on for some time, and we could also standardised on some boolean check that the developer could set to false when they want to stop the promise based render loop for the meantime

[toji]

FYI: It's my understanding that there were significant technical issues with cancellable promises and that they are no longer being considered for addition to the spec.

I don't object to a boolean indicating that you don't need the next frame, but I also don't see much harm in simply letting the developer ignore the returned promise when they're done?

[AlbertoElias]

Yup, I agree that the promise based implementation doesn't seem to have any major drawbacks and I wasn't a big fan of re implementing raf and calling it the same

[RafaelCintron]

@toji , The main thing I do not want to have happen is for the following case:

DrawSomeStuffToTheLayer();
vrSession.commit();
DrawSomeMoreStuff();
vrSession.commit().then(renderLoop);

to render DrawSomeMoreStuff on some hardware but force user agents to drop frames on other hardware if the GPU cannot keep up with the work. Instead, I think we should be clear to developers that following this pattern means DrawSomeMoreStuff gets completely dropped on the floor in all instances. Returning the previous promise on commit and warning in the console for this scenario seems reasonable to me.

Good point about the last frame being normally triggered as a result of some other async event. I agree this is a relatively minor drawback of the promise based approach.

I agree we should tie updates to the FrameData with resolution of the promise.

One thing I thought about after my initial reply. Suppose you have the case of the Three.js application with disparate pieces of code. Does each piece of code hook requestAnimationFrame on the VRDisplay itself and some primary code hooks it last and calls submitFrame? If so, the new approach will mean that the master code will need to do the moral equivalent of requestAnimationFrame to get everyone started. It will also have to pass around the subsequent promise so that the subordinate pieces of code can call then(). So, potentially more bookkeeping if that's how Three.js behaves, albeit relatively minor.

[toji]

Talking with @bfgeek about this particular issue, there's still a desire to merge this functionality with the CanvasRenderingContext.commit() concept which is looking more and more concrete. This would slightly complicate some things but make others more elegant, so I think it's worth discussing.

It's useful to consider where we want things to go in the future with other types of layers. Let's presume that we have a VRSkyboxLayer that takes in some images to present as a skybox and a VRVideoLayer that takes a video source and defines a quad in space where it should appear. A simple 3DoF video player could be built with zero WebGL by setting a theater environment to the skybox and positioning the video quad where the screen should be. In this scenario, having javascript pump the render loop is nonsensical. The video plays at it's own rate and the UA will be able to handle updating it with lower latency than JS+WebGL ever will, and the skybox doesn't update at all, it just needs to be re-rendered with the right pose. (And some APIs have special mechanisms to handle that case.) So this should clearly be a "set and forget" situation.

Coming back to the present, we can look at the VRCanvasLayer and see that there are occasions that should work the same way. For example: If the app just wants to present a splash screen it should be able to render it once and have the UA reproject it repeatedly while the page loads other resources. When we support multiple canvas layers we should be able to render the main scene to one canvas and some headlocked UI to another (Gaze reticle, etc.) The headlocked layer probably needs to be updated infrequently, so we shouldn't require it to produce a fresh image every frame in order to continue displaying, even if the main scene is updating each frame.

This implies two things about the design:

1. Layers should be able to commit imagery independently.
2. The UA will need to keep track of it's own VR render loop that is not dependent on the page activity. The page can observe the VR render loop, but does not drive it.

These two items probably require a formal concept of a VR compositor (no JS interface, just a spec description.)

With that in mind, I'll propose a slightly different flow: Use either CanvasRenderingContext.commit() when the canvas is a layer source or, if that doesn't seem workable, a new VRCanvasLayer.commit() to push the content of a Canvas to the VR compositor. The returned promise resolves when the current VR compositor frame has been presented. You can render frames and call commit() at any time, and the newly produced layer imagery will be associated with whatever frame the compositor is currently on. (This alleviates the need to call `commit() on something just to "get into" the render loop.) If multiple layers all call commit within the same frame they all get back the same promise. If a single layer calls commit multiple times in a single frame then only the most recently committed imagery is used, but both commits would get the same promise back. This should easily allow things like debug overlay layers to run independently of other content.

Everything committed in the space of a single frame is guaranteed to show up simultaneously on the display. Frame boundaries are defined by the VR compositor notifying the page when a frame has been completed. Since pages are single threaded they won't process that until javascript releases control back to the page, giving the app a simple mechanism for sync. (If you want multiple layers to display at the same time commit them all within a single callback. Note that committing and rendering don't necessarily have to happen at the same time.) Pose updates are available on the frame barriers.

Mirroring continues to work effectively the same way it does now. Anything you draw to a normal canvas without calling commit gets presented to the screen at the end of the callback. For offscreen canvases you can mirror with transferTo/FromImageBitmap.

It's a pretty big shift, but this feels to me like a pretty forward looking pattern. I'd love everyone's thoughts on it!

[toji]

One random follow up thought from my musing on the subject last night: One concrete benefit I could see to having commit() on the layer rather than the canvas context is that you could have multiple layers driven by a single context (enabling shared resources between layers). In other words, you could have multiple layers who's source is the same canvas, but you could draw separate content for each layer and commit individually. If the commit from the context is used then multiple layers pointing at the same canvas source would all get the same imagery on commit, which is pretty useless and we'd probably want to jump through some hoops to disallow it.

[mkeblx]

@toji If you commit from the session can you not do everything you want, or do really need explicit control per layer for commits?

// 3-DoF video player, no WebGL
var skyboxLayer = new VRSkyboxLayer(imgEl);
session.layers.push(skyboxLayer);
var videoLayer = new VRVideoLayer(videoEl, w, h, frameOfReference, position, /*etc*/); // 2D
session.layers.push(videoLayer);
session.commit(); // set and forget no?

// every frame of VR compositor uses latest state of videlLayer.videoEl, skyboxLayer.imgEl, etc.

Come to think of it scenarios like this (no WebGL canvas) what is (any) explicit commit doing? Do different layer types have different commit behavior, where some don't need commit per frame but others do?

--
Also, session commit of multiple layers with same canvas source would have same content but different end display due to layerBounds differences, or other layer properties.

var l1 = new VRLayer(canvas, mainBounds);
var l2 = new VRLayer(canvas, gazeCursorBounds);
session.layers.push(l1, l2);
renderLoop() {
  drawMainScene(mainBounds);
  drawCursor(gazeCursorBounds);
  canvas.commit().then(renderLoop);
}

[toji]

In your first code snippet I really don't think there should be a commit, unless you simply wanted to track the frame timing. Also, thanks for pointing out the possibility for using layer bounds to draw multiple layers with the same canvas! I hadn't considered that before!

There's pros and cons for both per-layer commits and session level commits. I think we could plausibly go either way and have solid reasoning for the choice. As far as I can see it's mostly a question of more explicit layer synchronization control vs. more flexible layer composibility.

(Of course none of this matters until we support multiple layers. Up to that point the two methods are identical, but it's worth considering the multi-layer case now so we don't have to change or complicate things down the road.)

I think it's probably a given that we want a way to guarantee that you can update N layers and have the new imagery for each layer be displayed on the HMD simultaneously. Relatedly, a way to indicate that all layers have been updated and can be submitted early is desirable, so we can have the imagery in flight while the page is doing physics simulations and such. A session-level commit handles this really cleanly:

var sceneLayer = new VRCanvasLayer(canvas1);
var cursorLayer = new VRCanvasLayer(canvas2, { headlocked: true });

function renderLoop() {
  drawScene();
  drawCursor();
  session.commit().then(renderLoop);
  updatePhysics();
}

It can also handle sparse layer updating by speccing that we only update imagery from canvases that are currently dirty (which is a well established concept already). Where it gets tricky is if we want to allow layers to be updated outside the main rendering loop. Picture a WebVR performance HUD that runs in an extension or as a drop in utility library.

// main.js
var sceneLayer = new VRCanvasLayer(canvas1);

function renderLoop() {
  drawScene();
  session.commit().then(renderLoop); // Hey, has the hud updated yet?
  updatePhysics();
}

// webvr-perf-hud.js
var hudLayer = new VRCanvasLayer(canvas2);

function renderHudLoop() {
  drawHud();
  // Whoops! Can't call session.commit() here.
  // Needs main.js to be aware of this file, which limits injectability.
  // Could monkeypatch session.commit() to do this drawing at the last second, I suppose?
}

A per layer commit allows this scenario to work out really nicely, though!

// main.js
var sceneLayer = new VRCanvasLayer(canvas1);

function renderLoop() {
  drawScene();
  sceneLayer.commit().then(renderLoop);
  updatePhysics();
}

// webvr-perf-hud.js
var hudLayer = new VRCanvasLayer(canvas2);

function renderHudLoop() {
  drawHud();
  hudLayer.commit().then(renderHudLoop);
}

Both commits would get the same promise back, so they'd stay in sync nicely even though they have effectively no knowledge of each other.

BUT! There's a less obvious issue here which is that we still want everything updated within a single frame to be presented to the HMD at the same time. This means I can't just shove a new image into the VR compositor immediately on calling commit on a layer because I may not know how many other layers are also going to commit that frame. So now we need to rely on an implicit sync point, which is probably the point at which the previous frame's commit promise has finished resolving. That's not terrible, WebGL currently works the same way with rAF, but it also means that the above code snippet is waiting for the physics simulation to complete before it sends the layer images off to the VR compositor. Not ideal!

You can work around that without too much trouble by doing something along the lines of

function renderLoop() {
  drawScene();
  sceneLayer.commit().then(renderLoop);
  setTimeout(updatePhysics, 0); // Force this to happen outside the commit promise resolve.
}

But that is a bit awkward.

Now is this style of loosely coupled, independently updating layers something that we must support? No. I personally like it, and I feel that the developer community will be able to use it to good effect, but I'm also certain that the API can survive and thrive without it. We've also faced questions in the past, however, about how to handle things like multiple calls to submitFrame that will come up again with a session-level commit. The per-layer model handles those types of edge cases more elegantly, IMO.

[toji]

Following up on the sync point issue from the previous comment if we went the per-layer commit route, I wanted to take a stab at trying to describe how it would work more formally to help clarify it. The idea being that this is the type of text I'd like to see in the spec:

The VR Compositor

Every UA which implements the WebVR spec MUST maintain a VR compositor which handles layer composition and frame timing. There are no direct interfaces to the compositor, but applications may submit imagery to it via the layer system and observe the frame timing via calls to VRCanvasLayer.commit.

The compositor consists of two different loops, assumed to be running is separate threads or processes. The Frame Loop, which drives the page script, and the Render Loop, which continuously presents imagery provided by the frame loop to the VR Display. The render loop maintains it's own copy of the session's layer list. Communication between the two loops is synchronized with a lock that limits access to the render loop's layer list.

Both loops are started when a session is successfully created. The compositor's render loop goes through the following steps:

1. The layer lock is acquired.
2. The render loop's layer list images are composited and presented to the display.
3. The layer lock is released.
4. Notify the frame loop that a frame has been completed.
5. Return to step 1.

The render loop MUST throttle it's output to the refresh rate of the VR Display. The exact point in the loop that is most effective to block at may differ between platforms, so no prescription is made for when that should happen.

Upon session creation, the following steps are taken to start the frame loop:

1. A new promise is created and set as the session's current frame promise. The current frame promise is returned any time VRCanvasLayer.commit is called.
2. The sessionchange event is fired.
3. The promise returned from requestSession is resolved.

Then, the frame loop performs the following steps while the session is active:

1. The render loop's layer lock is acquired.
2. Any dirty layers in the session's layer list are copied to the render loop's layer list.
3. The render loop's layer lock is released.
4. Wait for the render loop to signal that a frame has been completed.
5. The session's current frame promise is set as the the previous frame promise.
6. A new promise is created and set as the session's current frame promise.
7. The previous frame promise is resolved.
8. Once the promise has been resolved, return to step 1.

[RafaelCintron]

@toji , thank you very much for driving this issue and being so thorough.

I agree that everything committed in the space of a single frame should show up on the display at the same time. When the user agent resolves the (same) promise returned from all of the layers, all of the .then's that are part of the promise at the time of the resolve should be part of the frame. Calling getFrameData during this time will return the same set of matrices across all of the .thens'

However, we need to decide what happens if the web developer calls getFrameData/draws between the time the previous frame has retired but the new frame isn't ready yet. This corresponds to the time between step 3 and step 4 in the frameloop. In practice, I expect this to happen if the web developer draws during an onload handler or adds a .then to some, long ago, frame promise. I propose that we have getFrameData return null in these instances. This makes it clear to the web developer that they're doing it wrong and encourages them to only draw when the system is ready to consume new frames. WDYT?

Separately.

Drawing to multiple layers that point to the same canvas, with a commit call in between, seems like it will work. Since the user agent has no idea which layer a canvas draw is destined for, it will have to keep its own backing texture for each layer that is separate from the canvas. At VRLayer.commit time, user agents can copy the canvas to the per-layer texture or do a switch-a-roo between the two.

In an ideal world, we would provide web developers with a way to take a drawing context and set it on any canvas element via a new setContext or switchContext API. This lets the developer have their own Canvas element per-layer and each Canvas can be added to the DOM individually for debugging. However, it may or may not be worth making these kinds of changes to the Canvas API if we have alternatives we can employ.

[RafaelCintron]

Alternatively to my previous issue, we could have the frame data get passed as a parameter of the fulfillment. That makes it more clear to the web developer where and when they can get updated data. However, only .thens that were present on initial fulfillment of the promise show up on the display together. Ones that come in afterwards do not have that guarantee.

[toji]

Well crumb. I had a nice long response here and lost it in a browser crash. 😢 So here's my abbreviated version:

Rafael already told me personally that he realized an issue with the frame data getting passed as a parameter for the frame promise resolve, but for the sake of getting it down in text: That would prevent us from being able to use multiple coordinate systems within a single frame, which can be useful now (for things like tracking loss fallbacks) but will become much more important in the future with AR and tracking anchors. A possible alternative is to pass an object into the frame promise resolve that you query poses off of.

I think that's an interesting idea, but I'm not sure how well it would work with magic window mode, since there's not really a need to create a layer in that case. Actually, I just realized that I've got this wrong in the explainer as a result of the API shifting around and we should figure out what the pattern we want for that is. Should a non-exclusive session still set up a layer just for the purpose of calling commit()? I think you would probably just call window.requestAnimationFrame (Since you're presenting to the main window anyway) but if the pose inquiry was tied to the commit promise that wouldn't work.

If we assume that the pose still comes off the session to avoid magic window layer weirdness, the question of what happens to commits made outside the frame resolve is still valid. For one, we don't want to outright reject the commit: It should still get a valid promise, because that would be the only way to begin observing the main frame loop again if you had stopped for a while for whatever reason. The imagery is stuck in a weird spot, though. It wouldn't actually be accepted until, at earliest, the resolve of the next frame but it would be rendered with the pose from the last one. If it's a headlocked layer that wouldn't really matter. Outside of that, though, seems like the options are:

1. Discard committed imagery rendered outside a frame resolve.
2. Keep imagery and reproject it with the next frame's pose (inaccurately).
3. Keep imagery and reproject it with the previous frame's pose.

1 the most restrictive but easiest to support. 3 is probably best from a developers perspective, but I don't know if that could be supported on all platforms. 2 is an awkward middle ground that is probably supported anywhere, but I'm honestly not sure how objectionable the artifacts from that would be.

Gut says 1 is the best thing to do right now, and maybe we can loosen it up in the future if we see the need?

Semi-Unrelated: One thing I think I'd like to change from my proposal earlier in the thread is that only the first commit for a layer should be respected, instead of only the last. That way we can support a fast path that says "Once every committable layer has been committed the frame can be submitted early without waiting for the current callback to end."

(Parting thought: I apparently have a strange idea of an "abbreviated version".)

[toji]

Oh, and thanks for pointing out that you can do a .then on a promise that's already been resolved. Definitely needs to be accounted for in our spec text, and I agree that only rendering done on the initial promise fulfillment should be guaranteed to show up on screen synchronously.

[RafaelCintron]

Reading the explainer, I see that in both the magic window and exclusive mode branches of OnDrawFrame, you use the display pose. So passing an object used to retrieve the display pose as part of the promise would work for both rendering modes. Unless I am missing something, using (or not using) layers is an orthogonal manner.

One way we can make the in-between state more manageable is to have a BeginDraw function on VRLayer. If the user agent is not ready for a new frame, this can be point where we flag an error. Commit, or EndDraw, can still be the place where we return a promise.

Another advantage of BeginDraw is it provides a clear indication which layer is the destination of the draw commands when multiple layers point to the same Canvas element. This is particularly beneficial for APIs (such as HolographicSpace) which insist that clients draw into a provided color buffer that they manage on their own. With the current proposal, we only discover what layer the developer is interested in at commit time.

[toji]

Well, let me say up front that based on feedback we've gotten from web platform leads in the past I think that a begin/end call is probably something they'll protest. Just so we're all aware.

Putting that aside, I think the idea has merit. I especially like the idea of a concrete signal that we're about to start drawing VR content for the exact reason you stated. I'm not clear if you intend for BeginDraw to be the think that provides the aformentioned pose-getter, though? And if so, how does that play with magic window, which doesn't need any special mode switch for rendering?

I have a few more thoughts but I'm going to have to come back to them later, wanted to give @RafaelCintron a chance to answer my question in the meantime.

[toji]

Brought up the idea of BeginDraw in an internal API ergonomics review. The response was... unenthusiastic. 😅 How critical do you think that step is to enabling Edge to work correctly? Chrome, for the moment, is always doing a texture copy but we may move to a system where the intended render target is bound to any source canvases right before the commit promise is resolved, and then unbound after commit is called again. Worst case scenario is that commit is never called during the course of the callback, in which case we've wasted a tiny bit of time but that doesn't seem like a dealbreaker.

Stewed on the general topic of render loops some more over the weekend. I wanted to dive into magic window's relationship to the pose production a bit more.

The explainer currently uses a layer and commit() to drive the animation loop for both non-exclusive and exclusive sessions. I guess I don't mind this but it begs the question of what rate the frame loop runs at. For exclusive mode obviously it should be at the refresh rate of the headset, but in non-exclusive mode you'd want to render at the refresh rate of the monitor instead. Depending on the API implementation, however, the browser may only be able to provide updates at the headset refresh rate or the headset refresh rate may be inaccessible without an exclusive session. 🙄 I'm kinda leaning towards defining it as "when new poses are available" in that mode and disconnecting it from any specific display refresh. (You could also argue that devices that can't make a guarantee shouldn't expose non-exclusive mode at all. Not totally against that idea.)

I should note that as far as magic window mode goes I think that perfect, consistent, low latency pose delivery is far less important than when you've got a screen strapped to your eyeballs, so I wouldn't stress too much about using rAF to run the animation loop and having it be slightly out-of-sync with the pose update cycle. That way at least I know I'm not generating frames that will be discarded, and if I miss a pose or render the same one twice well then C'est la vie.

The important thing to note is that any reliance on rAF means we have to make the poses accessible outside the frame callback. And for magic window, again, that seems fine to me. They may end up a little stale, but that's much harder to notice when the device is in your hand. In exclusive mode I'd say that poses should still be available outside of the frame callback (useful for out-of-band logic) but that we intentionally discard any imagery committed outside of that callback (What I mentioned as "option 1" four comments back). Throw a console warning along with that and I doubt it would cause devs much trouble. That way the critical path stays pretty tight but the pose can still be used more flexibly.

[klausw]

Clarification request: what happens to window.requestAnimationFrame while VR presentation is in progress? Presumably, if there's an external display with a visible tab, window.rAF will continue firing at the appropriate rate independently from the VR display's refresh rate.

But what about the case if there's no external display such as on mobile? Since the native display is no longer visible while presenting, it would be natural to stop processing window.rAF while presenting (Chrome Canary currently does so), but this may be surprising to developers who are expecting it to keep running to handle background processing. This could happen indirectly if a non-VR-aware third party library uses its own window.rAF callbacks.

[spite]

That's basically my situation: i have a few libraries that rely on window.rAF for queuing and stepping processes, and don't really need to be aware if there's a VR display, or if the display has a refresh rate of 60 or 90 Hz.

It's been a pretty common pattern to have rAF as a replacement for setTimeout/setInterval, ever since rAF was first introduced. Some of this libraries work completely encapsulated and isolated, so it's messy to provide them with a changing rAF method.

[RafaelCintron]

Edge will work correctly without BeginDraw. However, lack of BeginDraw means we will be forced to do a buffer copy in the case where the same Canvas texture is used for multiple layers. I want to give as much performance back to the web developer as I can. Out of curiosity, when you say that you are going to move to a system where "the intended render target is bound to any source canvases right before the commit promise is resolves", how are you going to know what the intended render target is in the case where one Canvas is bound to multiple layers?

For pose data, I think we need to provide stronger spec guarantees than "whenever pose data is available". With this logic, a conformant implementation can provide different, updated poses with each call to getDevicePose, which is not desirable. I believe this will break existing Three.Js content that calls getFrameData multiple times within one Javascript callback.

For Edge, the plan for implementing VRCanvasLayer.commit is to create a new HolographicFrame right before promise fulfillment. Pose data is directly tied to a frame so we'll take this opportunity to snapshot that data locally in an efficient format for subsequent calls to VRSession.getDevicePose. When we Fulfill the promise, all of the .then callbacks are queued in our task system. Right after calling fulfill, we'll queue another task that accumulates all of the rendering and presents the completed frame to HolographicSpace. With this approach, we guarantee all getDevicePose calls will return the same data across all first level .then callbacks. Since each VRCanvasLayer.commit will return the same promise, there will most certainly be more than one .then in the multi-layer case. Three.Js content that calls getDevicePose multiple times in one callback will continue to work correctly as it does today.

The reason I suggested that VRDevicePose be passed into the promise or returned from BeginDraw (either way is fine) is to make it clearer to the web developer that device pose is tied to a frame and doesn't make sense to be called outside of a frame. With HolographicSpace, there is no way to get pose data outside of the frame. Indeed, we currently false for getFrameData outside of the VRDisplay.RAF in our v1.1 implementation.

Unfortunately, since VRCanvasLayers are optional for MagicWindow, this puts us in a bit of a bind. In your sample code, you write "in non-exclusive mode layers aren't needed, but it's convenient to create one anyway to observe the VR frame loop with." Should we consider making VRLayers mandatory for MagicWindow? This way, it seems we can rally around VRCanvasLayer.commit as the one mechanism to do render loops and simplify things.

[toji]

@klausw and @spite: window.requestAnimationFrame really should be unaffected by this spec, and it would be difficult for someone to convince me otherwise.

@RafaelCintron: Is there a reason I'm not seeing that even if the pose was only updated in the commit promise resolve you couldn't make the stale value available outside those callbacks?

I think I can get behind making commit() the blessed magic window driver if we settle on a few things. In non-exclusive mode, would you be OK with specifying that the frame loop run at the same rate as window.requestAnimationFrame? That'll probably be awkward for some devices to support, but I think it's a requirement that we give devs the ability to run content at the right rate for the intended display. Also, I think we should be able to create a VRCanvasLayer with a null source in that case, since the canvas contents aren't being used. That way we can specify that commit always clears contexts without preserveDrawingBuffer: false regardless of mode.

The single canvas for multiple layer case is one I admittedly didn't think of re: the need for BeginDraw. I'll have to ask around internally a bit more on that subject.

[RafaelCintron]

Is there a reason I'm not seeing that even if the pose was only updated in the commit promise resolve you couldn't make the stale value available outside those callbacks?

Yes, we can make the stale value available outside of the callbacks if we keep the getDevicePose function. My main concern is browser vendors updating the values returned from getDevicePose at inconsistent times. If one vendor keeps it steady on first level .then and returns stale values other times but other browser vendors update whenever you call getDevicePose or provide new values when called inside of DOMContentLoad or other HTML events, then content is not going to look the same across implementations.

In non-exclusive mode, would you be OK with specifying that the frame loop run at the same rate as window.requestAnimationFrame?

In any mode (exclusive or magic window), I think the frame loop should run at the rate of the VRDisplay that the layer is attached to. Depending on the display, that may or may not be the same rate as window.requestAnimationFrame.

Also, I think we should be able to create a VRCanvasLayer with a null source in that case, since the canvas contents aren't being used. That way we can specify that commit always clears contexts without preserveDrawingBuffer: false regardless of mode.

Specifying that commit always clears Canvases set on VRLayers regardless of the mode prevents web developers from setting true on preserveDrawingBuffer and adding it to the DOM for debugging purposes. Is that a scenario we want to keep working?

The single canvas for multiple layer case is one I admittedly didn't think of re: the need for BeginDraw. I'll have to ask around internally a bit more on that subject.

OK. Let me know what you hear back.

[toji]

My main concern is browser vendors updating the values returned from getDevicePose at inconsistent times.

I have zero qualms about requiring that fresh poses can only be exposed within the commit resolve (phrased that way to allow late querying within the callback if a UA wants to do it, though it should still return the same value after the first query.) Agreed that it would be bad to allow implementations to do a fresh query whenever.

I think the frame loop should run at the rate of the VRDisplay that the layer is attached to.

Okay, I don't object to that either. It does mean that Magic Window should be done with window.requestAnimationFrame, though, which makes the ability to get stale poses outside the frame loop all the more necessary.

Clear on commit yada yada yada

I think we're both saying the same thing here, I just explained it poorly. My point was that if commit() is called the canvas should behave as though it's being composited, regardless of whether or not the device is actually presenting. This means following the same logic as WebGL does once a callback ends. If preserveDrawingBuffer = false it clears/discards and if preserveDrawinBuffer = true it keeps the imagery around. The concession for allowing a null source is simply to avoid unnecessary clears or copies on a canvas that's not actually being used. Of course, if rAF is the magic window driver then that becomes less of an issue.

[RafaelCintron]

Okay, I don't object to that either. It does mean that Magic Window should be done with window.requestAnimationFrame, though, which makes the ability to get stale poses outside the frame loop all the more necessary.

The point I was trying to make was that VRCanvasLayer.commit will fulfill promises at the refresh rate of the underlying display. If the display is a Magic Window display, then the refresh rate will just so happen to be the same one as window.RAF.

My motivation for standardizing on VRCanvasLayer.commit is to try and find ways to simplify the API a bit. As it stands now, sometimes layers are optional, sometimes they're mandatory. Sometimes commit is what you use to know when you render again, sometimes its window.RAF. Sometimes, the left and right projection matrices are valid, sometimes you use the poseModelMatrix with a 'defaultProjectionMatrix', which is not defined anywhere. If it wasn't for your sample code, Brandon, I'd be lost about what to do by just looking at the WebIDL. Perhaps we need to go back to having different types of session objects to clarify things a bit. I'm torn.

Did you ever get an answer to BeginFrame when you asked around?

[toji]

The point I was trying to make was that VRCanvasLayer.commit will fulfill promises at the refresh rate of the underlying display. If the display is a Magic Window display, then the refresh rate will just so happen to be the same one as window.RAF.

Okay, I totally read your first comment wrong then. Sorry!

My motivation for standardizing on VRCanvasLayer.commit is to try and find ways to simplify the API a bit. [Descriptions of current confusingness]

Yeah, things are a bit of a mess in this regard right now. 😞 So your goal (and mine, at this point) is to unify the Magic Window and VR rendering paths as much as possible, right? How's this sound?

• VRCanvasLayer.commit() is the animation driver in all cases. It always runs at the rate of the intended output display, so in Magic Window scenarios it should run at the same rate as rAF.
• The projection matrices that the API provides need to take into account the aspect ratio of the output surface, so we can spec that if the baseLayer is a VRCanvasLayer the projection matrices provided are appropriate for use with that canvas.
• If the baseLayer doesn't have a source canvas we could fall back to using the full screen dimensions (or maybe window size?) as the basis for any projection matrices.
• We could further unify the rendering pipeline by changing the static left/right viewports provided in the pose to an N viewport system as discussed in #205

So now the setup and render loop looks identical for Magic Window and VR presentation:

function OnSessionCreated(session) {
  if (session.creationParameters.exclusive) {
    // This seems to be the only divergent point?
    let sourceProperties = vrSession.getSourceProperties();
    glCanvas.width = sourceProperties.width;
    glCanvas.height = sourceProperties.height;
  }

  vrSession.depthNear = 0.1;
  vrSession.depthFar = 100.0;

  vrCanvasLayer = new VRCanvasLayer(vrSession, glCanvas);
  vrSession.baseLayer = vrCanvasLayer;
 
  // Kick start the draw loop. If glCanvas is not dirty this wouldn't actually display anything.
  vrCanvasLayer.commit().then(OnDrawFrame);
}

function OnDrawFrame() {
  if (vrSession) {
    let pose = vrSession.getDevicePose(frameOfRef);

    // Only render if a new pose is available
    if (pose) {
      for (let i in pose.views) {
        let view = pose.views[i];
        let bounds = vrCanvasLayer.getViewBounds(i);
        // Remember that bounds are UVs and viewports are pixels
        gl.viewport(bounds[0] * gl.drawingBufferWidth,
                            bounds[1] * gl.drawingBufferHeight,
                            bounds[2] * gl.drawingBufferWidth,
                            bounds[3] * gl.drawingBufferHeight);
        DrawScene(view.viewMatrix, view.projectionMatrix);
      }
    }
    vrCanvasLayer.commit().then(OnDrawFrame);
  } else {
    // WebVR not supported or no session available. Render some fallback.
  }
}

That actually opens up a few interesting new possibilities: When browsing in-VR the matrices returned could take into account the canvas' position on the page and the position of the headset relative to the page to create an actual magic window through to the scene behind it. (ZSpace would probably also work well with that.) Additionally, the N viewport support allows even magic window to request stereo rendering. A nice feature here is that it actually leaves quite a bit of the decision making for how content is requested and presented in the hands of the UA/platform.

Thoughts?

[toji]

Updated the explainer (3e22c49) to take into account the discussion above, though I've left the N viewport stuff out of it for the moment. Let me know how that sits with everyone.

[RafaelCintron]

I like the unification you're proposing with regard to Magic Window and other session types! I also like the N-viewport proposal as well as the "interesting possibilities" you pointed out near the end.

Internally, several of us have been brainstorming ways we can improve the layer proposal with an eye towards leveraging multi-GPU scenarios and better solve the buffer copy problem with the current commit approach. @NellWaliczek has volunteered to write up a summary as a separate issue.

[toji]

Looking forward to seeing it, thanks!

[toji]

This has been incorporated into the explainer now, so closing.