-
Notifications
You must be signed in to change notification settings - Fork 34
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: tracking #268
base: main
Are you sure you want to change the base?
feat: tracking #268
Changes from 3 commits
5efd0ed
66349b6
50d7b99
12862f2
c70a6ab
ee902e3
691d625
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -10,35 +10,88 @@ toc_max_heading_level: 4 | |
|
||
## Overview | ||
|
||
Experimentation is a primary use case for feature flags. | ||
In practice, this often means flag variants are assigned to users at random or in accordance with a business rule, while the impact of the assigned variant on some business objective is measured. | ||
Vendors and custom solutions often support a _tracking_ or _goal measuring_ API to facilitate the measurement of these business objectives. | ||
|
||
### Goals | ||
|
||
- Develop official terminology to support consistent implementation | ||
- Specify a flexible API widely compatible with basic vendor functionality | ||
- Define tracking event payload | ||
- Define tracking event identifier | ||
- Support A/B testing and experimentation use-cases | ||
- Support client and server paradigms | ||
- Provide recommendations around: | ||
- Async vs sync | ||
- Flushing mechanisms | ||
- Event batching | ||
|
||
### Non-goals | ||
|
||
- Creating an experimentation platform | ||
- Covering every user-tracking use case | ||
- We will not define any data aggregation mechanisms | ||
- We will not focus on "metrics", but instead, "facts" | ||
|
||
### Design Principles | ||
|
||
We value the following: | ||
|
||
- Adherence to, and compatibility with OpenFeature semantics | ||
- Maximum compatibility and ease-of-adoption for existing solutions | ||
- Minimum traffic and payload size | ||
- Ease-of-use for application authors, integrators, and provider authors (in that order) | ||
The `tracking API` enables the association of feature flag evaluations with subsequent actions or application states, in order to facilitate experimentation, and analysis of the impact of feature flags on business objectives. | ||
|
||
Combined with hooks which report feature flag evaluations to the analytics platform in question, tracking can allow for robust experimentation even for flag management systems that don't support tracking directly. | ||
|
||
```mermaid | ||
sequenceDiagram | ||
Evaluation API->>+Tracking Hook: evaluate | ||
Tracking Hook->>Analytics Platform: before | ||
Tracking Hook->>Analytics Platform: after | ||
Tracking Hook->>-Evaluation API: evaluate | ||
Evaluation API->>Analytics Platform: track | ||
``` | ||
|
||
### 6.1. Tracking API | ||
|
||
#### Condition 6.1.1 | ||
|
||
> The implementation uses the dynamic-context paradigm. | ||
|
||
see: [dynamic-context paradigm](../glossary.md#dynamic-context-paradigm) | ||
|
||
##### Conditional Requirement 6.1.1.1 | ||
|
||
> The `client` **MUST** define a function for tracking the occurrence of a particular action or application state, with parameters `occurrence key` (string, required), `evaluation context` (optional) and `occurrence details` (optional) which returns nothing. | ||
|
||
```java | ||
// example tracking occurrence recording that a subject reached a page associated with a business goal | ||
client.track("visited-promo-page", evaluationContext); | ||
beeme1mr marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
// example tracking occurrence recording that a subject performed an action associated with a business goal, with the occurrence details having a particular numeric value | ||
client.track("clicked-checkout", evaluationContext, new OccurrenceDetails(99.77)): void; | ||
toddbaert marked this conversation as resolved.
Show resolved
Hide resolved
|
||
``` | ||
|
||
See [evaluation context](../types.md#evaluation-context), [occurrence details](#62-occurrence-details). | ||
|
||
#### Condition 6.1.2 | ||
|
||
[![experimental](https://img.shields.io/static/v1?label=Status&message=experimental&color=orange)](https://github.com/open-feature/spec/tree/main/specification#experimental) | ||
|
||
> The implementation uses the static-context paradigm. | ||
|
||
see: [static-context paradigm](../glossary.md#static-context-paradigm) | ||
|
||
##### Conditional Requirement 6.1.2.1 | ||
|
||
> The `client` **MUST** define a function for tracking the occurrence of a particular action or application state, with parameters `occurrence key` (string, required) and `occurrence details` (optional) which returns nothing. | ||
|
||
The track function is a void function (function returning nothing). | ||
Though it may be associated with network activity or other I/O, it need not be awaited by application authors. | ||
|
||
```java | ||
// example tracking occurrence recording that a subject reached a page associated with a business goal | ||
client.track("visited-promo-page"); | ||
|
||
// example tracking occurrence recording that a subject performed an action associated with a business goal, with the occurrence details having a particular numeric value | ||
client.track("clicked-checkout", new OccurrenceDetails(99.77)): void; | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I see the spec below requires the ability to set both a There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Good point, I've added this: new OccurrenceDetails(99.77).add("currencyCode", "USD") Which is probably how it will end up looking if implemented in Java, using our existing |
||
``` | ||
|
||
#### Requirement 6.1.3 | ||
|
||
> The evaluation context passed to the provider's track function **MUST** be merged in the order: API (global; lowest precedence) -> transaction -> client -> invocation (highest precedence), with duplicate values being overwritten. | ||
|
||
The SDK passes a merged evaluation context to the provider's track function similarly to the manner it does in resolvers. | ||
|
||
See: [context levels and merging](./03-evaluation-context.md#32-context-levels-and-merging). | ||
|
||
#### Requirement 6.1.4 | ||
|
||
> If the client's `track` function is called and the associated provider does not implement tracking, the client's `track` function **MUST** no-op. | ||
|
||
## 6.2. Occurrence Details | ||
|
||
The `occurrence details` structure defines optional data pertinent to a particular `occurrence`. | ||
|
||
### Requirement 6.2.1 | ||
|
||
> The `occurrence details` structure **MUST** define an optional numeric `value`, associating a scalar quality with an `occurrence`. | ||
|
||
`Value` is a well-defined field which some providers may map to equivalent numeric values in their API. | ||
|
||
See [provider tracking support](./02-providers.md#27-tracking-support). | ||
|
||
### Requirement 6.2.2 | ||
|
||
> The `occurrence details` **MUST** support the inclusion of custom fields, having keys of type `string`, and values of type `boolean | string | number`. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think that this details should be equivalent to the structure value available for contexts. Allowing not just a single-level map of raw types, but effectively any JSON representation. If it is just a flag map, then it places what seems to be an unnecessary constraint on less restrictive systems. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm in agreement with this proposal, with the caveat that we're OK with some additional complexity in provider implementation. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think big use case for occurrences is joining occurrences with variants to see what "business impact" flag changes have.
The way I understand it, most people will be just "joining" this with what's proposed in Feature Flag Impression SemConv
People just need something to join both entries on. SemConv proposal only suggest logging "context id" without full context. I have been using targetingId for this use case so far. I am thinking - wouldn't it make sense to have track method schema match schema of SemConv?
I am not advocating for any particular solution, but for more alignment between those two proposals and enabling use case of joining variant impressions with occurrences.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
100%. This is how I envision things would work, and how most people will use it, I think.
I struggled a lot with this same question.
I agree that generally most people would probably join on the
targetingKey
but I'm not sure that would be the case 100% of the time. If we include the entire evaluation context in the track API, users have the ability to join on whatever attributes they want which are common to the telemetry data from a flag evaluation and from thetrack
call (perhaps an attribute calledemail
, for example). It's for this maximum flexibility I proposed the API take the evaluation context instead of thetargetingKey
only.Additionally, taking the entire
evaluationContext
is more consistent with existing functions, keeping the API a bit more isomorphic.All that said, I understand why taking the
targetingKey
directly in the track method is appealing. Like I said, I struggled with this question for a while, and I'm open to other perspectives.cc @beeme1mr @dyladan @nicklasl
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
💯 percent agree with what @toddbaert wrote on the flexibility aspect and I would like to add something:
Passing the whole evaluation context in the track API allows the tracking functionality to stand on its own legs as we (the OpenFeature API) provide enough data to the tracking providers to be enable slicing and dicing the produced tracking data by imaginative dimensions that could be deduced in the evaluation context.
Additionally:
We are talking about joining data between the moment of evaluation (I think of this as the moment of exposure) and the moment of event tracking. The evaluation context may have changed between these two moments in time, it can also be that these two moments are captured in services with different evaluation context.
The question is; what context is important to track?
If I could only choose one context to store, I would keep the context of the tracking event but of course, having both of these stored would for sure be the best. So I'm arguing that from a business perspective the context data from the event is more important than the context data from the evaluation.
Now there may be some nuances in the above reasoning between static- and dynamic context paradigms, especially when it comes to the moment of exposure in the static paradigm. I would argue that for the static paradigm it is not the flag evaluation (when the client bulk fetches the flags from the backend) but rather the flag access that should be counted as the moment of exposure.