Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Present lower-level operator interfaces #326

Open
frankmcsherry opened this issue May 19, 2021 · 0 comments
Open

Present lower-level operator interfaces #326

frankmcsherry opened this issue May 19, 2021 · 0 comments

Comments

@frankmcsherry
Copy link
Member

Several operators are written with interfaces that are meant to be "foolproof", in that you should not be able to mis-use them. Specifically, the join class of methods allow you to provide closures that act on data, but are not provided time or diff, as it is possible to mis-use that responsibility. At the same time, there are useful operators that can benefit from that information: specifically, there are time and diff manipulating linear operators (e.g. temporal filters, negate, etc) that you might want to fuse on to the join output.

We could present "unsafe" versions of several operators that expose these details, for power users who are confident that they can write not-incorrect logic, or at least who are willing to absorb that responsibility. The upside for these users is the ability to avoid producing outputs in cases where filtering would apply (e.g. temporal filters) or actions that move information from data to diff (e.g. explode style operators).

Examples that seem like they are clear candidates are the join variants, as well as dogsdogsdogs's lookup_map and half_join operators. It wouldn't be unreasonable to do reduce as well, though I don't have an immediate use case for it that would be easy to validate ("also fusing operators").

One straw man example would be an appropriately named join_idk_internal_unsafe that exposes the most generality we can manage, without concern about shepherding the user to correct behavior, and then target the existing join_ variants onto it, mapping their safer closures to the framework of the unsafe operator.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant