Synced audio

[ronpadz]

The ability to support synced audio is required for rich experiences. Voice-over commentary/instructions/narrative, music or assistive ADA audio capability is very important. The implication is that an audio file would be bundled in with the experience and controls made available for both the creator and the user:

• The creator would require switches for auto-play, auto-loop, etc.
• The user would require start, stop, pause and volume controls.

[cdata]

Definitely agree. This isn't the first time we have heard this feedback. I think we might have a look into a first-party component that can synchronize a <model-viewer> and an <audio> tag.

[fabiotgolo]

Sorry! I didn't understand. We can already use audio or not?
If yes, could you please share a tutorial?
Best regards,

[hybridherbst]

I'd like to bump this! Now that both SceneViewer (via url parameters) and QuickLook (via reality files) support synced audio clients start requesting it.
Are you aware of any workaround of extracting current animation time from model-viewer to sync with an external (embedded in the website) audio source @cdata ?

[cdata]

I think we would need to do a little extra work to make this go. My understanding is that for syncing audio, the ideal way would be to key the currentTime of the model off of the currentTime of an <audio> element.

We are probably pretty close to enabling this, but I don't think it will work today via JS because updates to properties don't show up synchronously.

[hybridherbst]

Are you able to put an ETA on "pretty close" (pretty close to being able to hack something in from our end)? Goal is to have the same behaviour on model-viewer, SceneViewer and QuickLook - a single looping animation with a single looping sound.

(We have a project shipping end of February and I'd like to to figure out whether we can stay on model-viewer for that (which I'd like to do) or use something else as foundation.)

[cdata]

We are trying to get 1.0 shipped in the first quarter of this year. At this time, most new features that do not require breaking changes have been deprioritized and will be shipped post-1.0. Currently, this feature has not been prioritized at all. That's about as clear as I can be about the current state.

Can you comment at all on the scope of the project that is shipping at the end of February?

[cdata]

(by scope I mean: number of live URLs, number of models etc.)

[hybridherbst]

Understood. Scope is basically what I described - it's gonna be a single model on a single microsite, and it's gonna play a single looping animation and audio, as that's what we can get to work in SceneViewer/QuickLook as well.
So we want that microsite to do the same thing.
(if you have a hint on where in the code we could expose the current animation time that would already be useful, we can probably hack-expose it ourselves)

Basically, if the cute penguin's glb+ogg you can by now find through Google Search and open in SceneViewer would play synced sound in the web as well we'd be there :)

Link to cute penguin
(note: does not play audio on desktop but works on Android)
GLB file | Synced audio file

[cdata]

So, unlike most of our properties, you can currently read and write to currentTime synchronously, and it's just a light wrapper around the Three.js animation system:

https://github.com/GoogleWebComponents/model-viewer/blob/992faddcbb1c0fc9d665ff6a3d352c92cc7b047e/packages/model-viewer/src/features/animation.ts#L63-L69

I think the problem you might run into is that I'm not sure that we re-render when the model is paused and you update currentTime. If that's the only problem, I would be happy to review a PR that fixes it.

Once that works, you could theoretically glue an <audio> and a <model-viewer> together with something like (psuedocode):

<audio id="sound" src="..."></audio>
<model-viewer id="modelViewer" animation-name="talk" src="..."></model-viewer>
<script>
sound.addEventListener('timeupdate', () => {
  modelViewer.currentTime = audio.currentTime;
});
</script>

In the long run, I would like to support something along the lines of:

<model-viewer animation-name="talk" src="...">
  <audio src="..." slot="synced-audio"></audio>
</model-viewer>

So that no script is needed at all.

[hybridherbst]

Thanks a lot - that turned out to be super easy! https://pfc-modelviewer-audio-tests.glitch.me/ (controls are currently needed since Android does not allow autoplay of audio, works on desktop without the controls).

I'll see if we run into the issue you described.

[ryancrunchi]

Hello, sorry to bump this topic, but I have a question for @hybridherbst and did not know how to contact him.
I saw you managed to add audio to .reality file (penguin), via model-viewer as well as iOS quick look.
I did try to add audio behavior at scene start via Reality Composer. I tried with spatial audio or ambient audio with mp3, wav, m4a or aac.
The audio on exported .reality file plays on macOS quick look (10.15.6), but not on iOS quick look (13.7 nor 14). Is there something I missed ?

[dirrogate]

@cdata
Is there a definite way to either

• Embed audio within a .Glb file
• Or trigger an audio file in sync with model-viewer?

I've searched around and saw some references to a Microsoft solution and Babylon, but interested in a solution
Thanks all.

[hybridherbst]

You can't embed audio within glb.
What you can do though is have a regular audio tag play in sync with model-viewer by properly setting the time of both.

So, to have cross-platform audio, you need to have 3 paths:

• for web: model viewer + audio tag + some code to sync them
• for Android: either use SceneViewer and append the audio file to the URL, or use WebXR and keep the JS solution (we're using SceneViewer)
• for iOS: embed the audio file via RealityComposer or (better) directly in the USDZ via UsdPreviewAudio.

An example of how we solved that is here:
https://glitch.com/edit/#!/pfc-modelviewer-audio-tests

Hope that helps, at some point I should probably make a PR with that example and some cleanup :) what do you think @elalish?

[dirrogate]

@hybridherbst Thank you so much. Yes this helps a great deal. I'll follow the example you listed in the Glitch. (coding skills are nil to low, but can follow via example)
It's good to know USDZ can have embedded audio. I was hoping, given that Gltf/GLB is a wrapper for various streams of data there would be a robust way to embed an audio stream. (I've read that someplace and then there's this: https://forum.babylonjs.com/t/can-you-embed-sound-on-a-glb-3d-model-i-know-but-ufo-glb-seems-to-have-it/5030/2)

On a side note, barring the compression, i'm really curious how glb/gltf became a standard, given it does not yet support video textures on materials. But that's a topic for a different day.

[hybridherbst]

Yes, currently the MSFT extension is the only way to embed audio.
I didn't list it as an option here because it's not supported by either model-viewer nor SceneViewer currently - otherwise this would indeed be easier.

I would say gltf became a standard to grow :) starting simple and growing more complex over each version.
USDZ went the opposite direction: USD itself has millions of features, Apple supports a tiny subset of it and then adds custom extensions on top. Not sure any of these is strictly better - I spent way more time trying to understand what USDZ doesn't support than what GLTF supports.

[dirrogate]

Yes, currently the MSFT extension is the only way to embed audio.
I didn't list it as an option here because it's not supported by either model-viewer nor SceneViewer currently - otherwise this would indeed be easier....

Thank you! this clears my doubts well! I hope the Model-viewer team consider implementing/supporting the MSFT extension. It would make life easier for synced animations and indeed, for me - short narrative stories, via WebXR. (Apple's a whole different beast and I've made my thoughts clear on it over twitter to Tim and gang. Not that they'll care)