Does it support raytrace mouse clicks?

[abrahamezzeddine]

Hello,

I am wondering if this would support raytracing the splats themselves rather than the XYZ points? I am trying to create a mouse interactor but at the moment, it seems that mouse interaction with splats is not possible unless you click in the center of the splat (points).

Any suggestion? Or would this require further adjustments to the custom shaders?

[mkkellogg]

I'm not sure how you're currently doing raycasting, but it won't work with three.js' raycaster, because raycasting against a splat mesh works differently (it returns intersections with splats, not three.js meshes).

Here is an example of how to use the raycaster that I made for this library specifically for raycasting against splats:

const viewer = new GaussianSplats3D.Viewer();
viewer.addSplatScene(path)
.then(() => {
   viewer.start();
   viewer.renderer.domElement.addEventListener('pointerdown', onMouseClick, false);
});

const onMouseClick = function(mouse) {
  raycastSplatMesh(mouse.offsetX, mouse.offsetY);
}

const raycastSplatMesh = function() {

    const renderDimensions = new THREE.Vector2();
    const toNewFocalPoint = new THREE.Vector3();
    const mousePos = new THREE.Vector3();
    const outHits = [];

    return function(x, y) {
        viewer.getRenderDimensions(renderDimensions);
        mousePos.set(x, y);
        outHits.length = 0;
        viewer.raycaster.setFromCameraAndScreenPosition(viewer.camera, mousePos, renderDimensions);
        viewer.raycaster.intersectSplatMesh(viewer.splatMesh, outHits);
        if (outHits && outHits.length > 0) {
          console.log(outHits[0])
        }
    };

}();

[abrahamezzeddine]

Many thanks. I will try your code.

I was actually using raycasting against the XYZ points with world coordinates (like a point cloud). It was actually working, but it needed a bit of work to handle it. I just could not do it with the meshes themselves because three js saw the splats as invisible.

Do you also know if it’s possible to add objects that would be able to blend with the splats? For example, if I add an object with 50% opacity, in front of the splat, then the splats are still not visible through the box. Tried it with depth tests but nothing seems to work. I hope you understand what I mean. Thanks for the code!!

[abrahamezzeddine]

Do you also have a function where you can select a set of splats by dragging the mouse over a select splats and capture their data? Thanks!

[abrahamezzeddine]

Ok, I managed to be able to do this. Just by dragging I am able to select the hits within the selected area, even if several rays hits a splat, its only counted once.

If I want to change the color of the selected area, which function/data can I change to have them being rendered in another color? For example, I select an area and it becomes yellow to indicate selected gaussians. @mkkellogg

Array(98)
0
:
Hit {origin: Vector3, normal: Vector3, distance: 7.791569343969397, splatIndex: 3379030}
1
:
Hit {origin: Vector3, normal: Vector3, distance: 6.501858822219376, splatIndex: 238717}
2
:
Hit {origin: Vector3, normal: Vector3, distance: 8.145546947668452, splatIndex: 3866475}
3
:
Hit {origin: Vector3, normal: Vector3, distance: 7.704654750292739, splatIndex: 2975015}
4
:
Hit {origin: Vector3, normal: Vector3, distance: 7.007145544099755, splatIndex: 1806775}
5
:
Hit {origin: Vector3, normal: Vector3, distance: 6.441965736158495, splatIndex: 1401130}
6
:
Hit {origin: Vector3, normal: Vector3, distance: 6.386669952888312, splatIndex: 1815513}
7
:
Hit {origin: Vector3, normal: Vector3, distance: 6.572796986984113, splatIndex: 5277430}
8
:
Hit {origin: Vector3, normal: Vector3, distance: 6.649110114023007, splatIndex: 5809711}
9
:
Hit {origin: Vector3, normal: Vector3, distance: 7.966611424637027, splatIndex: 2464279}
10
:
Hit {origin: Vector3, normal: Vector3, distance: 7.57514852037996, splatIndex: 2496780}
11
:
Hit {origin: Vector3, normal: Vector3, distance: 7.450429896871575, splatIndex: 4121328}
12
:
Hit {origin: Vector3, normal: Vector3, distance: 7.071975992284323, splatIndex: 2963635}
13
:
Hit {origin: Vector3, normal: Vector3, distance: 6.409067107068281, splatIndex: 2145776}
14
:
Hit {origin: Vector3, normal: Vector3, distance: 6.569121598415119, splatIndex: 2899248}
15
:
Hit {origin: Vector3, normal: Vector3, distance: 6.748451482364997, splatIndex: 5131273}
16
:
Hit {origin: Vector3, normal: Vector3, distance: 7.708737542804063, splatIndex: 3963590}
17
:
Hit {origin: Vector3, normal: Vector3, distance: 7.651760758336675, splatIndex: 3486992}
18
:
Hit {origin: Vector3, normal: Vector3, distance: 7.676078690369003, splatIndex: 1421594}
19
:
Hit {origin: Vector3, normal: Vector3, distance: 7.464344290900274, splatIndex: 2523039}
20
:
Hit {origin: Vector3, normal: Vector3, distance: 7.373109692839261, splatIndex: 3138907}
21
:
Hit {origin: Vector3, normal: Vector3, distance: 7.1049169017198475, splatIndex: 2815412}
22
:
Hit {origin: Vector3, normal: Vector3, distance: 6.607358069917467, splatIndex: 4832920}
23
:
Hit {origin: Vector3, normal: Vector3, distance: 6.669811855401825, splatIndex: 5641386}
24
:
Hit {origin: Vector3, normal: Vector3, distance: 7.558577059771568, splatIndex: 1918458}
25
:
Hit {origin: Vector3, normal: Vector3, distance: 7.286929280991535, splatIndex: 4274145}

[mkkellogg]

There is no direct way to change a splat's color, so you'll have to implement it yourself. You can take a look at the SplatBuffer.getSplatColor() function to get some hints on how the data is structured:

GaussianSplats3D/src/loaders/SplatBuffer.js

Line 291 in ef524f4

getSplatColor(globalSplatIndex, outColor) {

Once you have changed the splat's color in the splat buffer, you'll need to update the GPU data textures with the appropriate values. That can be done using the SplatMesh.refreshDataTexturesFromSplatBuffers() function.
There is probably more complexity to doing this (and edge cases that I haven't thought of) than I have addressed here, so you'll probably have to spend some time digging through and learning the code. Also, if you change a splat's color, you'll need to remember it so you can store it later :)

[abrahamezzeddine]

Ok, thanks.

Can I assume globalsplatindex is the index of the splat itself, which we store in the sectionindex inside the function?

outColor[0-3] is RGBA, correct?

And outcolor[] is the one that we actually use in the refresh function?

So if I fetch the globalindex first, modify the colours, and then call splatbuffer and refreshtextures, it should be feasible? Does it refresh the whole Gaussian splats or just those we modify?

I am still new to programming so please have patience with me. 😆

[abrahamezzeddine]

So my plan here is to use AABB bounding box to get a box from two views. By fetching a screenshot and a specific cropped view, I will send these to SAM segmentation model and get the binary masks back.

I am wondering, I am trying to find the function to access the XYZ points within a set box, but cannot.

What function can I call to get the splatsindex as well as XYZ coordinates within bounding box? I am having a little trouble finding those functions. Thank you.

[abrahamezzeddine]

@mkkellogg

here is an example where I have managed to integrate SAM 2 with a backend with your interactive Gaussian Splatting Viewer GUI. Looks awesome. By selecting an area, I am able to segment certain areas.

However, would be cool to highlight to splats via some kind of selection hue color. Could you please share how to access the points values XYZ, RGBA, opacity etc... for each splat in the Viewer pipeline? Thank you!

[mkkellogg]

All splat data is stored in one or more SplatBuffer instances, and those are all stored in a single instance of SplatMesh. You can pull the data out of the splat buffers and store them in JavaScript arrays for manipulation by using SplatMesh.fillSplatDataArrays(). Look at this function to see how it's used:

GaussianSplats3D/src/splatmesh/SplatMesh.js

Line 886 in ef524f4

updateBaseDataFromSplatBuffers(fromSplat, toSplat) {

Currently when a scene is being loaded, the fillSplatDataArrays() function is used to pull the splat data (XYZ position, color, etc...) from the splat buffer and that data is pushed to a three.js texture, which is how it gets to the GPU. An example of that second step is here:

GaussianSplats3D/src/splatmesh/SplatMesh.js

Line 919 in ef524f4

this.updateDataTexture(paddedCenterColors, centerColorsTextureDescriptor.texture, centerColorsTextureDescriptor.size,

[abrahamezzeddine]

Thank you! I will get back to you in case of more questions. 👍

[abrahamezzeddine]

@mkkellogg I have tried to expose SplatMesh but it seems that I cannot still access those functions via the npm package. Any suggestions for a sample code on how you do it with the viewer, please? When I try to read the contents, it say that they are undefined for some reason.

[mkkellogg]

Sorry for the late response, you will have to access the instance of SplatMesh through the Viewer class. Something like:

const viewer = new GaussianSplats3D.Viewer();
viewer.splatMesh.updateDataTexture(...)
// or
viewer.splatMesh.updateBaseDataFromSplatBuffers(...)