RFC: Entity model + database implementation

[replygirl]

This RFC proposes an entity model compatible with our MVP wireframes, and an implementation for Fauna in which a client writes mostly-immutable Events to the database and reads back entities resolved from those Events.

Accessibility note: This draft contains a class diagram that will be made redundant with upcoming changes to the proposal text. The body text of this proposal will still make sense with a screen reader, but most of the entities in the model are stuck in the images until that next update.

1. Introduction
   1. Prior work
   2. Lucidchart
2. Entity model
   1. Entities
   2. Changes since initial information architecture
   3. Out-of-scope entities and relations
3. Database implementation
   1. Fauna
      1. Why Fauna?
      2. Identity-based access
         1. Authentication
         2. Access control
      3. Migrations
         1. Versioning for breaking changes
      4. User data controls
      5. Concerns with Fauna
   2. Events
      1. Semi-immutability
   3. Entity resolution
      1. GraphQL schema impacts
      2. Alternatives to entity resolution in the database
      3. Concerns with entity resolution
4. Next steps
   1. Upcoming changes
   2. Votes

Introduction

OTD (BSA Open Tech Development) is currently building out the MVP of Dual Power App based on information architecture and wireframes developed with @Manhattan-Hydraulics, and the biggest upcoming blocker to a working client is our data environment: the shape of that data that realizes the information architecture (entities), and the data store in which that information lives and with which clients & services interact (implementation).

This proposal addresses both entities and implementation, with the latter designed around the former. Through the architecture defined for the implementation, it also aims to consolidate work on Dual Power App's back end into the database by limiting the scopes of needs for infrastructure and for an API.

Sophisticated needs for our data model & data store include:

• Versioned content: The Proposal flow requires support for a rich text body with discrete drafts, with each draft having distinct related entities (e.g. Suggestions, Polls).
  • Addressed with the Draft entity
• Status on changes: In the long term, the Proposal flow will be applied to other entities (e.g. documents, crowdfunding campaigns). The ambition for democracy on the platform is both universal, so there's value in setting up the model such that we can later opt to subject any change to the proposal flow.
  • Addressed with Events
• History: User and Circle dashboards provide a feed of both entities (e.g. Posts) and changes to entities (e.g. voting period for a poll ended), so a client will need to be able to query across a set of changes to many items.
  • Addressed with Events

Long-term concerns include:

• Feature compatibility: While out of scope, we are confident about some upcoming features that we'd like to know our MVP entity model does not preclude.
• Extensibility: BSA has other active and planned projects that would ideally integrate directly with Dual Power App, and such integration would require ability to save data not represented by the core entity model. If we set reasonable constraints, it may be possible to support this kind of "third-party" development without changes to the structure of the data store.
• Protocol-friendly: Based on prior discussions and our goals to interoperate with a broader ecosystem of platforms, we are likely to move to a decentralized architecture in the distant future. This RFC hopes to present a solution where the core patterns of interacting with the data store don't change (i.e. query language changes are ok, i.e. query structure changes are discouraged), and that clients and nodes communicate with one another using the same pattern.
• OTD labor expenditure: We are a resource-constrained organization, and an elegant solution for our data can reduce the overall work we need to do to build & grow the platform.

Editor's note: Hi, I'm Imogen, I work as an engineer + platform architect and write about transportation. I've been an OTD member since early 2020. I'm stewarding this proposal as an individual, and first presented it as a work-in-progress at the September 1, 2021 Open Design + Build. There are still some elements of this write-up that are missing, but the substance of the proposal will not change—I'm fleshing out the details then it's ready for a vote.

Prior work

• Data modeling with @LogicalDash (June 2020)
• Org modeling w/ Devin from @Manhattan-Hydraulics (May 2021)
• Dual Power App Information Architecture by @Manhattan-Hydraulics (July 2021)
• Entity model draft with @LogicalDash (August 2021)
• Thread: Choosing a Data Store (July 2021–present) (Basecamp members may not see comments)

Lucidchart

Part of this proposal is represented in a Lucidchart document, containing a class diagram of the entity model, a domain diagram of the data store implementation, and investigation into the considered approaches. The relevant elements are also embedded and linked throughout this proposal.

Entity model

This model builds on the Information Architecture by @Manhattan-Hydraulics, and aims to represent all of the functionality for our MVP, except where we need to determine the model as we build a feature (in which case, relations haev been determined but fields have not).

Entities

View in Lucidchart

All entities & descriptions:

This list will be populated in the lead-up to the second vote, on the database implementation. In the meantime, see everything in the class diagram above.

• Draft: One version of a passage of rich text. Accepts suggestions, and may be considered "codified" when a Poll passes. Belongs to a Proposal. Could support other parents in the future, for Circle resources to be democratically updated on an ongoing basis.

  interface Draft {
    parent: Proposal | Document | Desire | Engagement | Fund | Post
    suggestions?: Suggestion[]
    poll?: Poll<Draft>
    thread?: Thread
    body: RichText | string
  }

Changes since initial Information Architecture

• Consolidated Circle: Org, Circle, and Area are now just Circle.
• Draft: Proposal and other entities have their content edited with a Draft, which provides revision history and version-specific approvals.
• Backwards Desire: Desires used to be created with Proposals, now they are addressed by Proposals and can be amended with Drafts
• Actions: Actions are addressed in the database implementation section, where Events can be associated with PollOptions to give them an approval status. This means an Action object is not necessary.

Out of scope entities and relations

The class diagram for this proposal includes likely upcoming features that are out of scope for MVP, to ensure forward compatiibility. These are drawn into the diagram at a reduced opacity, and won't be fully fleshed out until after this proposal.

• Community forums
• Advanced Location management for Circles.
• Proposals resulting in new or updated Documents, Desires, Engagements, Funds, and VoteModelTypes.
• Liquid democracy: Delegate your Vote to another User.
• CircleRole: A means of delegating authority to a User. A CircleRole and grants/revocations thereof would likely be bound to Proposals and authority delegated by a different CircleRole.
• Federation: A formation of many Circles. A Federation differs from a parent Circle in that the Circle is not "part of" the Federation, but a participant, an can be active in many Federations just as a User may be active in many Circles.
• Social connections:
  • Watch: Represents a User receiving updates about a User | Circle. This is similar to following on another social platform, but the language is explicitly selected to avoid a connotation of endorsement.
  • Boost: Represents a User | Circle supporting, elevating, or endorsing a User | Circle. This was conceived alongside Watch as an alternative to following, but could be generalized to a more broad idiom.

Database implementation

This implementation part of this proposal implements the entity model in Fauna with a collection of Events, using user-defined functions in the database to resolve entities at runtime (i.e. construct them from the event history, rather than saving their state to the database):

• Events simplify implementation of core features of the platform. This proposal focuses on feeds and the Proposal flow.
• Resolving entities at runtime from Events eliminates concerns of state drift between entities and their history, and allows users granular, retroactively-effective privacy controls.
• Fauna vastly reduces devops overhead, provides a GraphQL service, connects to external identity providers, and lets us write callable, complex business logic for our transactions (= less reliance on an API).

View in Lucidchart

Fauna

Fauna is a flexible, developer-friendly, transactional database delivered as a secure and scalable cloud API with native GraphQL. Never again worry about database provisioning, scaling, sharding, replication, or correctness.

There are a few different ways to use Fauna, but this proposal adopt the most common pattern:

• Maintain schema in GraphQL
• Interact with the database's GraphQL API directly from the client
• Read & write data in FQL with user-defined-functions
  • Why use UDFs? Moving business logic into the database reduces the complexity of services & clients and the impact of changes to business logic, and guarantees that data changes exactly when and how we expect it to.
  • For some data models, it's not necessary to write any UDFs—just upload a GraphQL schema with types, queries, and mutations, and Fauna will generate inputs and handle resolution for you. This proposal's migration and entity resolution strategies rely on UDFs for all resolution.

Why Fauna?

• Managed: OTD won't have to worry about infratructure deployments, availability, or performance. Updating the schema and user-defined functions become the only ops concern.
• Powerful: User-defined functions and the FQL standard library enable us to handle authorization and most (or maybe all) business logic without an external service.
• Predictable pricing: Despite an annoying $150 up-front cost for a multi-user dashboard, Fauna is affordable overall, and costs can be easily forecast thanks to a simple pricing model based on total storage and read, write, & compute ops.

Identity-based access

Authentication

Fauna can accept JSON Web Tokens from external identity providers, and grant one or more roles to those tokens. By creating an AccessProvider for Auth0 or another provider, we can safely consume user-specific tokens, associate them with internal User documents, and validate their permissions with predicate functions and UDFs.

Access control

With identity-based access, there are two ways to enforce permissions in Fauna:

• Attribute-based access control provides for predicate functions on role grants, which can deny access to documents in a given collection.
• UDFs resolving GraphQL queries and mutations allow us to deny access to documents in a given transaction.

A provisional list of instances where access may be selectively granted is available to OTD members in Nimbus's 9/20 comment on the "Choosing a Data Store" thread.

Migrations

A "migration" consists of a replacement of the GraphQL schema and/or addition/replacement of UDFs, both of which have been validated in a test database. Literal migration of the data from an old format to the current format happens at runtime as documents are accessed, after the "migration" process

#83 provides two documented examples of the basic migration pattern, with step-by-step instructions for each.

Phased migrations without downtime are feasible, but this proposal advocates for a simple approach that accepts occasional late-night downtime;

• When schema changes are non-destructive and there is no need to change multiple UDFs in one step, the database can be updated without impacts.
• When schema changes include modifications to existing fields or query/mutation signatures, and/or multiple UDFs need to change at the same time, a few minutes should be scheduled for downtime. "Downtime" can mean any of the following:
  • Clients may receive errors when interacting with the GraphQL API, as the return signatures of UDFs do not match the GraphQL schema.
  • Clients may be made aware by an external service to avoid interacting with the GraphQL API.
  • Preferred: A universally-applied UDF that returns a boolean value is updated before & after the migration, rejecting all transactions while the migration takes place.

Versioning for breaking changes

To accommodate versioned requests, a version argument may be passed to query or mutation and handled by its UDF. This should be adopted in the first breaking change after the MVP release, and a discussion will be needed on whether to version globally or per-resolver.

User data controls

This proposal does not cover endorsement of user-defined visibility controls, retention policies, or bulk deletes functionality. If this proposal passes, the following approaches should be considered:

• Similar to the implementation of just-in-time migrations, UDFs can be called during any transaction to enforce retention policies and visibility controls.
• ABAC provides for predicate function on role grants, which can subject all client-originating transaction to visibility controls.
• UDFs on mutations can safely bulk delete documents while making whatever change are necessary to preserve the integrity of impacted documents.

Concerns with Fauna

• FQL introduces a learning curve: Syntax always takes a minute to get comfortable with, and this is especially the case with query languages, as common patterns aren't as established as with programming and markup langauges. That said, while niche, FQL itself is easy to learn, with minimal syntax and a small standard library. After writing your first UDF or two, you'll be self-sufficient with the cheat sheet. SQL users can read this quick comparison between SQL and FQL.
• We won't control the environment the data lives in: This is true. The assumption this proposal makes is that the additional labor cost involved with of data sovereignty (i.e. a self-hosted environment) is unreasonable when OTD has limited capacity for ongoing work commitments, before the platform is out in the world, and when we may eventually transition to a decentralized architecture anyway.

Events

An Event is an additional entity representing a change to the internal state of Dual Power App. By making the Event a persistent entity and source of truth for other entities, we can accommodate our needs to subject state changes to the Proposal process and to resolve a User or Circle's activity feed from a mixed set of entities and changes to entities.

All Event types & descriptions (coming in next update):

Come back soon :)

Semi-immutability

With a collection of Events capturing all changes to entities, there's no need to ever update an Event, with the exceptions of migrations and user-requested data removal. Given an uncorrupted collection of Events, Entity state is replayable from the beginning of time.

Entity resolution

• TODO: Add entity resolution logic to RFC: Event-based data model, Fauna demos #83:
• TODO: Share live demo of entity resolution:

Since an Event represents a change, it's not necessary to persist entities to the database. Instead, we can use each query's UDF to resolve (i.e. construct) entities on demand, based on the log of changes to that entity in the Events collection, referencing the entity with the _id of the Event that created the entity and reducing across the following related Events until the current time. This grants a few benefits:

• Since Events are the source of truth, entity state will never drift.
• Since the only state persisted is that of Events, the scope of possible problems that can break our data is confined to (a) improperly written Event mutation resolver UDFs and (b) unavoidable concerns about security controls.
• History-related features like feeds and tracked changes can be applied retroactively.
• OTD can provide Dual Power App moderators with tools for investigating & resolving community disputes.
• Since this approach uses events, does not rely on persisted state, allows business logic to live in the same place as the data, and talks to the rest of the world in a common language (GraphQL), Dual Power App can more easily transition to a decentralized architecture in phases later on.

GraphQL schema

• TODO: Add enough schema to demonstrate patterns

Our schema for this approach would differ from an entities-first schema in the following ways:

• Most types apart from Event would become embedded types
• There would be more and stricter mutations

Alternatives to entity resolution in the database

• Don't use Events, and instead create & mutate entities directly: With this approach basic CRUD operations would be easiest to scaffold. However, due to the openness of entities to mutations, more collection-level permissions would be required to enforce safe mutations (as opposed to index-level permissions and input validation). Additionally, our data model is sufficiently complex that additional schema or logic would be required to resolve things like an activity feed or document history anyway.
• Use Events, but persist entity state along with the Events: This was initially the preferred option in this proposal. Basic entity queries would be free with GraphQL schema upload, and only mutations would require custom logic in UDFs. However, it was raised in the 9/1 Open Design + Build meeting that a buggy mutation UDF slipping through the crack would cause destructive changes to our data, the approach of resolving entities would mean no UDFs need to update data unless the Events collection is migrated. There's also a possibility of state drift within the database itself, between the event log and the entities.
• Use Events, and resolve entity state on the client instead of in the database: This option was proposed as a means of reducing the number and size of client-db transactions and enabling message passing between clients. This option seems infeasible, considering the size of initial load times would increase over time unless we maintained a second GraphQL service in front of Fauna with a cache of entity state.

See the end of the "Data store implementation" section of the Lucidchart for a breakdown of pros & cons with each approach.

Concerns with entity resolution

• Performance: While Fauna is very fast, combing through a log for just-in-time resolution of entities is in most cases slower than directly serving an entity from its own and related table. We should keep an eye on this but there isn't hard data on this impact yet—and, with Fauna, we will not need to hit an API in many places we otherwise would have.
• Deriving entity state from history sounds complex: This is true, but the concerns noted in the introduction demand complexity in any approach. With a traditional database this means functions and views or an external API, and with Fauna this means UDF. The priority, then, is about consolidating that complexity and putting it in places where it minimizes risk (see the note above on the second alternative to entity resoltuion). The proposed approach requires UDFs for resolution in all reads, but reduces the problem space in writes to validation. Additionally, core resolution functionality can be encapsulated in reusable UDFs called within resoslver UDFs.

Next steps

Upcoming changes

There are a bunch of "TODO" items throughout this RFC, mostly amounting to:

• Listing all entities, with fields & descriptions, in this write-up
• Listing all event types, with parameters & descriptions, in this write-up

These are required for the second vote on database implementation, but not for the first on entity model.

Votes

Votes on this proposal can be cast asynchronously or at an Open Design + Build meeting, which happens every other Wednesday. Dates and instructions for the votes will be posted about a week before voting starts, and voting will be open for a week.

• Entity model: Voting begins or ends during OTD meeting Wednesday, October 13, 2021
• Database implementation: TBD

[nimbusgo]

When we're talking about Events, are we referencing the fauna Events function? Or are we describing a definition within the data model, where we define create-only objects (Events) which can be resolved into Entities on the fly?

If we're talking about the events fauna function, is there an example of an interesting kind of udf based entity resolution that showcases the need here? Resolving the entity to return the latest state of the entity would be the default behavior in any case (no need for any interesting resolution).

[replygirl]

When we're talking about Events, are we referencing the fauna Events function? Or are we describing a definition within the data model, where we define create-only objects (Events) which can be resolved into Entities on the fly?

Here, an Event is an additional entity in our model, potentially the only one saved to a collection.

Fauna events don't give us flexibility with types and payloads, you're just going back in the snapshot history for the newest version of a document. They're static snapshots from built-in transactions. You could query across all events for a set of documents, as the Events function accepts a Ref or a Set of Refs, and save event metadata on every update so it's available in snapshots, but there is no way to index built-in events that I've seen, which I think we'd need since dashboard feeds will be a big part of this.

I avoided mentioning Fauna temporality as much as possible in the write-up, assuming I would avoid confusion, but maybe I should at least put a note and link to this thread. Idk where to draw the line on specificity here.

If we're talking about the events fauna function, is there an example of an interesting kind of udf based entity resolution that showcases the need here? Resolving the entity to return the latest state of the entity would be the default behavior in any case (no need for any interesting resolution).

Examples are hard to come by. Either way, I do need to write a resolution demo. Maybe two:

1. Show that resolution is possible with findFooById queries: One UDF taking a ref is called by two different findFooById query resolvers. The UDF is helped by a few indexes over the collection. I know this is possible, so success = the UDF makes sense to other people
2. Show that resolution scales with foosByBar queries: One query resolver lists a set of entities filtered by a field. The assumption I'm making here is our entity construction will not be considerably slower than Fauna's native construction of entities from event history.

I intend to write those last as the entity model outside of Events does not depend on them

[replygirl]

updated with vote-ready class diagram :)

[replygirl]

Looks like I forgot to post a link here—part 1 of the RFC is open to votes! You can vote through this form, or Wednesday during the beginning of our weekly meeting at dualpower.app.

[replygirl]

We'll be taking final votes and checking the results tonight at 7pm. If you need clarification on something before you're ready to vote, I'll make sure there's time for that in the meeting.