A question about using WebXRMode.VR in GaussianSplats3D.DropInViewer.

[pollyon]

I used your 'GaussianSplats3D.DropInViewer' method to load a splat file and added the viewer into 'Object3D' so that I can later control its position and size within the scene. Everything was working fine until I added the WebXR VR button. After entering VR mode, the splat model only partially renders or doesn’t render at all, and it seems like the camera is no longer calculating the distance to the splat correctly. Is there a way to resolve this issue?

Additionally, since I need to control the position and size of the splat within the scene, I am unable to use your 'GaussianSplats3D.Viewer' method to activate VR mode.

[mkkellogg]

There may actually be some problems with using DropInViewer with VR, I don't think I've ever tried doing that myself :) Can you post your source code? Maybe I can help you debug.

[mkkellogg]

Just circling back -- did you make any progress with this one?

[pollyon]

I sincerely apologize for the late response. I've been extremely busy with work over the past few weeks, but I finally have time to prepare the test files for the project.

Below is the link to the test project.
download link

We used the DropInViewer method to import the splat file, and after entering VR mode with the Meta Quest headset, the issue we previously discussed occurs. I've also attached a video recording of the VR view. I hope to receive your assistance. Thank you very much.

[pollyon]

We tried using your 'GaussianSplats3D.Viewer' method to start VR mode, but since your VR mode automatically generates a new WebXR configuration, it caused a conflict with our original WebXR settings, resulting in an error. We had to remove our own WebXR settings to run your VR mode properly.

On the other hand, when using the 'GaussianSplats3D.DropInViewer' method, since it does not trigger the WebXR generation mechanism in your JavaScript, no errors occur, and we can enter our own WebXR settings. However, for some reason, the Gaussian Splats appear broken and incomplete, as shown in the attached video, and the surface variations of the Gaussian Splats stop computing. These are the results of our testing over the past few weeks. We hope to receive your assistance. Thank you very much.

[mkkellogg]

One thing you can also try (if I understand the issue around the WebXR config) is to use the new memory-optimizations branch, which has a Viewer parameter for specifying WebXR initialization parameters: webXRSessionInit. Maybe that will work for you?

[pollyon]

We are using the version from the memory-optimizations branch and have tested webXRSessionInit. We made the settings as shown in the image,

but after clicking "ENTER VR," an error occurred, as shown in the image.


Aside from the issue of unsupported hand-tracking in the webXRSessionInit settings, the internally written layers also encountered the same unsupported issue. Even after we removed both settings, the issue persists with the error: "Cannot render to a XRWebGLLayer framebuffer outside of an XRSession animation frame callback."

[pollyon]

Currently, when testing with the viewer method, if we integrate our custom three.js scene, errors occur after clicking "ENTER VR." The only case where there is no error is when using the "DropInViewer" method, but this results in the splat model stopping surface computation and being incomplete, as shown in the image.

Is there a solution for the issues with the splat model stopping surface computation and its incompleteness?

Since "DropInViewer" does not execute the init() and setupWebXR() functions from gaussian-splats-3d.module.js, and we use our own VR button method in the three.js scene to enter VR, perhaps we don’t need to handle the WebXR settings in gaussian-splats-3d.module.js?

[mkkellogg]

FYI Using the DropInViewer should still result in init() and setupWebXR() being called.

For the error you're seeing using the GaussianSplats3D.Viewer mode, is it possible that the renderer that you're passing to GaussianSplats3D.Viewer when you initialize it is being used outside of the animation loop that it is supposed used in? When you tell the viewer to use WebXR, all rendering needs to occur in the callback function passed to renderer.setAnimationLoop(), and that is set up here:

GaussianSplats3D/src/Viewer.js

Line 1385 in 2d80511

this.renderer.setAnimationLoop(this.selfDrivenUpdateFunc);

Because you're passing in an external renderer, the viewer makes it a WebXR render here:

GaussianSplats3D/src/Viewer.js

Line 361 in 2d80511

this.renderer.xr.enabled = true;

After which it may not be able to be used anywhere else.