Bidirectional communication between the tldraw and python

[kolibril13]

Bidirectional communication between python and tldraw would be amazing to have!
Here are three use cases that I can see:

• updating background Matplotlib figures without losing the overlay drawings
• image segmentation by getting the coordinates of the hand drawn strokes on the canvas
• programmatic shape generation, e.g. with functions and sliders from the python side.

Here is an example of how I imagine bidirectional communication could look like:

I've set up the JupyterLite notebook https://kolibril13.github.io/jupyter-tldraw/lab/?path=idea_bidirectional_communication.ipynb for investigation.
WARNING: All Code of this JupyterLite instance will be deleted on reload, so code should be copy+pasted to a save place before closing that page.

Brainstorming on how to get there:

• onChange, onUserChange or onCommand could be helpful in this context (related docs: https://github.com/tldraw/tldraw-v1/blob/a63319cf7b8e16a259f53a40028e7070300ab78f/guides/documentation.md?plain=1#L23-L28).

[steveruizok]

import ipyreact
from traitlets import Unicode
from IPython.display import display


class TldrawWidget(ipyreact.ReactWidget):
    my_text = Unicode("Hello World").tag(sync=True)
    my_text_out = Unicode("Hello World").tag(sync=True)

    _esm = """
    import { TDShapeType, Tldraw } from "@tldraw/tldraw";
    import * as React from "react";

    export default function App({ my_text, my_text_out }) {
    
        const [app, setApp] = React.useState()

         const handleMount = React.useCallback((app: Tldraw) => {
             setApp(app)
         }, []);

         React.useEffect(() => {
             if (app) {
             app.createShapes({
                 id: "text1",
                 type: TDShapeType.Text,
                 point: [100, 100],
                 text: my_text,
             });
             }
         }, [my_text, app])

        return (
            <div
            style={{
                position: "relative",
                width: "800px",
                height: "350px",
            }}
            >
            <Tldraw onMount={handleMount} onChange={e => my_text_out = "updated" }/>
            </div>
        );
    }
    """


tldraw = TldrawWidget()
tldraw.my_text = "This is Tldrawesome!!🎉"
tldraw.my_text_out = "This is Tldrawesome!!🎉"
display(tldraw)

[steveruizok]

^^ pasted during our call, but that should be enough to get you going! Basically you want to run a side effect whenever those props change, and in that side effect you'd want to interface with the tldraw imperative API (app), which has methods for creating shapes, updating shapes, etc.

[kolibril13]

Thanks so much for this gold nugget, that will be a great starting point!
The direction "python -> Tldraw" is now covered.
If you have a little bit of bandwidth, do you have an idea for "Tldraw -> python" side as well?

So that tldraw.my_text will give me 'My text that I wrote in tldraw' insead of 'Hey there!'

[kolibril13]

I experimented a bit more, and it seems that the onChange method in
<Tldraw onMount={handleMount} onChange={e => console.log("My log") }/>
works fine every time the canvas is updated, so I just have to find a way to update the my_text traitlet.
I think that I will be able to solve this on my own :)

[MarcSkovMadsen]

+1.

I was hoping to find bidirectional communication. I was playing around with Panel and AnyWidget examples. TlDraw with bidirectional communication would be amazing to have access to.

# pip install panel ipywidgets_bokeh tldraw
from tldraw import TldrawWidget

widget = TldrawWidget()

# HELP FUNCTIONALITY to convert Traitlets Classes/ Events to Param Classes/ Events
import anywidget
import param

_ipywidget_classes = {}
_any_widget_traits = set(anywidget.AnyWidget().traits())

def create_observer(obj, traits=None)->param.Parameterized:
    """Returns a Parameterized class with parameters corresponding to the traits of the obj
    
    Args:
        traits: A list of traits to observe. If None all traits not on the base AnyWidget will be
        observed.
    """
    if not traits:
        traits = list(set(obj.traits())-_any_widget_traits)
    print(traits)
    name = type(obj).__name__
    if name in _ipywidget_classes:
        observer_class = _ipywidget_classes[name]
    else:
        observer_class = param.parameterized_class(name, {trait: param.Parameter() for trait in traits})
        _ipywidget_classes[name] = observer_class
    
    values = {trait: getattr(obj, trait) for trait in traits}
    observer = observer_class(**values)
    obj.observe(lambda event: setattr(observer, event["name"], event["new"]), names=traits)
    return observer

# THE PANEL APP

import panel as pn
pn.extension("ipywidgets")

observer = create_observer(widget)

def some_output(value):
    print(value)
    return value

component = pn.Column(widget, pn.bind(some_output, observer.param.value))

pn.template.FastListTemplate(
    site="Panel",
    title="Works with Jupyter-TlDraw",
    main=[component],
).servable()