From 80243a4a3d2262097b54cbef9b0eb10fa00aee19 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Thu, 12 May 2022 16:09:00 -0400 Subject: [PATCH 01/16] Initial proposal for dehydrated devices with SSSS --- .../xxxx-dehydrated-devices-with-ssss.md | 224 ++++++++++++++++++ 1 file changed, 224 insertions(+) create mode 100644 proposals/xxxx-dehydrated-devices-with-ssss.md diff --git a/proposals/xxxx-dehydrated-devices-with-ssss.md b/proposals/xxxx-dehydrated-devices-with-ssss.md new file mode 100644 index 0000000000..a1b2a175c9 --- /dev/null +++ b/proposals/xxxx-dehydrated-devices-with-ssss.md @@ -0,0 +1,224 @@ +# MSCxxxx: Dehydrated Devices with SSSS + +[MSC2697](https://github.com/matrix-org/matrix-doc/pull/2697) introduces device +dehydration -- a method for creating a device that can be stored in a user's +account and receive megolm sessions. In this way, if a user has no other +devices logged in, they can rehydrate the device on the next login and retrieve +the megolm sessions. + +However, the approach presented in that MSC has some downsides, making it +tricky to implement in some clients, and presenting some UX difficulties. For +example, it requires that the device rehydration be done before any other API +calls are made (in particular `/sync`), which may conflict with clients that +currently assume that `/sync` can be called immediately after logging in. + +In addition, the user is required to enter a key or passphrase to create a +dehydrated device. In practice, this is usually the same as the SSSS +key/passphrase, which means that the user loses the advantage of verifying +their other devices via emoji or QR code: either they will still be required to +enter their SSSS key/passphrase (or a separate one for device dehydration), or +else that client will not be able to dehydrate a device. + +This proposal introduces another way to use the dehydrated device that solves +these problems by storing the dehydration key in SSSS, and by not changing the +client's device ID. Rather than changing its device ID when it rehydrates the +device, it will keep its device ID and upload its own device keys. The client +will separately rehydrate the device, fetch its to-device messages, and decrypt +them to retrieve the megolm sessions. + +## Proposal + +### Dehydrating a device + +The dehydration process is the same as in MSC2697. For completeness, it is +repeated here: + +To upload a new dehydrated device, a client will use `PUT /dehydrated_device`. +Each user has at most one dehydrated device; uploading a new dehydrated device +will remove any previously-set dehydrated device. + +`PUT /dehydrated_device` + +```jsonc +{ + "device_data": { + "algorithm": "m.dehydration.v1.olm" + "other_fields": "other_values" + }, + "initial_device_display_name": "foo bar" // optional +} +``` + +Result: + +```json +{ + "device_id": "dehydrated device's ID" +} +``` + +After the dehydrated device is uploaded, the client will upload the encryption +keys using `POST /keys/upload/{device_id}`, where the `device_id` parameter is +the device ID given in the response to `PUT /dehydrated_device`. The request +and response formats for `POST /keys/upload/{device_id}` are the same as those +for `POST /keys/upload` with the exception of the addition of the `device_id` +path parameter. + +Note: Synapse already supports `POST /keys/upload/{device_id}` as this was used +in some old clients. However, synapse requires that the given device ID +matches the device ID of the client that made the call. So this will be +changed to allow uploading keys for the dehydrated device. + +### Rehydrating a device + +To rehydrate a device, a client first calls `GET /dehydrated_device` to see if +a dehydrated device is available. If a device is available, the server will +respond with the dehydrated device's device ID and the dehydrated device data. + +`GET /dehydrated_device` + +Response: + +```json +{ + "device_id": "dehydrated device's ID", + "device_data": { + "algorithm": "m.dehydration.v1.olm", + "other_fields": "other_values" + } +} +``` + +If no dehydrated device is available, the server responds with an error code of +`M_NOT_FOUND`, http code 404. + +If the client is able to decrypt the data and wants to use the dehydrated +device, the client retrieves the to-device messages sent to the dehydrated +device by calling `POST /dehydrated_device/{device_id}/events`, where +`{device_id}` is the ID of the dehydrated device. Since there may be many +messages, the response can be sent in batches: the response can include a +`next_batch` parameter, which can be used in a subsequent call to `POST +/dehydrated_device/{device_id}/events` to obtain the next batch. + +``` +POST /dehydrated_device/{device_id}/events +{ + "next_batch": "token from previous call" // (optional) +} +``` + +Response: + +```jsonc +{ + "events": [ + // array of to-device messages, in the same format as in + // https://spec.matrix.org/unstable/client-server-api/#extensions-to-sync + ], + "next_batch": "token to obtain next events" // optional +} +``` + +Once a client calls `POST /dehydrated_device/{device_id}/events`, the server +can delete the device (though not necessarily its to-device messages). Once a +client calls `POST /dehydrated_device/{device_id}/events` with a `next_batch` +token, the server can delete any to-device messages delivered in previous +batches. It is recommended that, for the last batch of messages, the server +still send a `next_batch` token, and return an empty `events` array when called +with that token, so that it knows that the client has successfully received all +the messages. + +### Device Dehydration Format + +TODO: define a format. Unlike MSC2679, we don't need to worry about the +dehydrated device being used as a normal device, so we can omit some +information. So we should be able to get by with defining a fairly simple +standard format, probably just the concatenation of the private device keys and +the private one-time keys. This will come at the expense of implementations +such as libolm needing to implement extra functions to support dehydration, but +will have the advantage that we don't need to figure out a format that will fit +into every possible implementation's idiosyncrasies. The format will be +encrypted, which leads to ... + +#### Encryption key + +The encryption key used for the dehydrated device will be randomly generated +and stored/shared via SSSS using the name `m.dehydrated_device`. + +## Potential issues + +The same issues as in +[MSC2697](https://github.com/matrix-org/matrix-doc/pull/2697) are present for +this proposal. For completeness, they are repeated here: + +### One-time key exhaustion + +The dehydrated device may run out of one-time keys, since it is not backed by +an active client that can replenish them. Once a device has run out of +one-time keys, no new olm sessions can be established with it, which means that +devices that have not already shared megolm keys with the dehydrated device +will not be able to share megolm keys. This issue is not unique to dehydrated +devices; this also occurs when devices are offline for an extended period of +time. + +This may be addressed by using fallback keys as described in +[MSC2732](https://github.com/matrix-org/matrix-doc/pull/2732). + +To reduce the chances of one-time key exhaustion, if the user has an active +client, it can periodically replace the dehydrated device with a new dehydrated +device with new one-time keys. If a client does this, then it runs the risk of +losing any megolm keys that were sent to the dehydrated device, but the client +would likely have received those megolm keys itself. + +Alternatively, the client could perform a `/sync` for the dehydrated device, +dehydrate the olm sessions, and upload new one-time keys. By doing this +instead of overwriting the dehydrated device, the device can receive megolm +keys from more devices. However, this would require additional server-side +changes above what this proposal provides, so this approach is not possible for +the moment. + +### Accumulated to-device messages + +If a dehydrated device is not rehydrated for a long time, then it may +accumulate many to-device messages from other clients sending it megolm +sessions. This may result in a slower initial sync when the device eventually +does get rehydrated, due to the number of messages that it will retrieve. +Again, this can be addressed by periodically replacing the dehydrated device, +or by performing a `/sync` for the dehydrated device and updating it. + +## Alternatives + +As mentioned above, +[MSC2697](https://github.com/matrix-org/matrix-doc/pull/2697) tries to solve +the same problem in a similar manner, but has several disadvantages that are +fixed in this proposal. + +Rather than keep the name "dehydrated device", we could change the name to +something like "shrivelled sessions", so that the full expansion of this MSC +title would be "Shrivelled Sessions with Secure Secret Storage and Sharing", or +SSSSSS. However, despite the alliterative property, the term "shrivelled +sessions" is less pleasant, and "dehydrated device" is already commonly used to +refer to this feature. + +The alternatives discussed in MSC2697 are also alternatives here. + + +## Security considerations + +The security consideration in MSC2697 also applies to this proposal: If the +dehydrated device is encrypted using a weak password or key, an attacker could +access it and read the user's encrypted messages. + +## Unstable prefix + +While this MSC is in development, the `/dehydrated_device` endpoints will be +reached at `/unstable/org.matrix.mscxxxx.v1/dehydrated_device`, and the +`/dehydrated_device/{device_id}/events` endpoint will be reached at +`/unstable/org.matrix.mscxxxx.v1/dehydrated_device/{device_id}/events`. The +dehydration algorithm `m.dehydration.v1.olm` will be called +`org.matrix.mscxxxx.v1.olm`. The SSSS name for the dehydration key will be +`org.matrix.mscxxxx` instead of `m.dehydrated_device`. + +## Dependencies + +None From ed2c5eb2cc6f66a8276183bf949ad8dd0a952799 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Thu, 12 May 2022 16:12:33 -0400 Subject: [PATCH 02/16] use MSC number --- ...th-ssss.md => 3814-dehydrated-devices-with-ssss.md} | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) rename proposals/{xxxx-dehydrated-devices-with-ssss.md => 3814-dehydrated-devices-with-ssss.md} (96%) diff --git a/proposals/xxxx-dehydrated-devices-with-ssss.md b/proposals/3814-dehydrated-devices-with-ssss.md similarity index 96% rename from proposals/xxxx-dehydrated-devices-with-ssss.md rename to proposals/3814-dehydrated-devices-with-ssss.md index a1b2a175c9..35d4af8c19 100644 --- a/proposals/xxxx-dehydrated-devices-with-ssss.md +++ b/proposals/3814-dehydrated-devices-with-ssss.md @@ -1,4 +1,4 @@ -# MSCxxxx: Dehydrated Devices with SSSS +# MSC3814: Dehydrated Devices with SSSS [MSC2697](https://github.com/matrix-org/matrix-doc/pull/2697) introduces device dehydration -- a method for creating a device that can be stored in a user's @@ -212,12 +212,12 @@ access it and read the user's encrypted messages. ## Unstable prefix While this MSC is in development, the `/dehydrated_device` endpoints will be -reached at `/unstable/org.matrix.mscxxxx.v1/dehydrated_device`, and the +reached at `/unstable/org.matrix.msc3814.v1/dehydrated_device`, and the `/dehydrated_device/{device_id}/events` endpoint will be reached at -`/unstable/org.matrix.mscxxxx.v1/dehydrated_device/{device_id}/events`. The +`/unstable/org.matrix.msc3814.v1/dehydrated_device/{device_id}/events`. The dehydration algorithm `m.dehydration.v1.olm` will be called -`org.matrix.mscxxxx.v1.olm`. The SSSS name for the dehydration key will be -`org.matrix.mscxxxx` instead of `m.dehydrated_device`. +`org.matrix.msc3814.v1.olm`. The SSSS name for the dehydration key will be +`org.matrix.msc3814` instead of `m.dehydrated_device`. ## Dependencies From 703281efb9ea501d244d6b62edfb11965ab880b8 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Mon, 5 Sep 2022 11:44:46 -0400 Subject: [PATCH 03/16] wording improvements and clarifications --- .../3814-dehydrated-devices-with-ssss.md | 22 +++++++++++-------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/proposals/3814-dehydrated-devices-with-ssss.md b/proposals/3814-dehydrated-devices-with-ssss.md index 35d4af8c19..5df866c09d 100644 --- a/proposals/3814-dehydrated-devices-with-ssss.md +++ b/proposals/3814-dehydrated-devices-with-ssss.md @@ -90,7 +90,7 @@ Response: ``` If no dehydrated device is available, the server responds with an error code of -`M_NOT_FOUND`, http code 404. +`M_NOT_FOUND`, HTTP code 404. If the client is able to decrypt the data and wants to use the dehydrated device, the client retrieves the to-device messages sent to the dehydrated @@ -122,11 +122,14 @@ Response: Once a client calls `POST /dehydrated_device/{device_id}/events`, the server can delete the device (though not necessarily its to-device messages). Once a client calls `POST /dehydrated_device/{device_id}/events` with a `next_batch` -token, the server can delete any to-device messages delivered in previous -batches. It is recommended that, for the last batch of messages, the server -still send a `next_batch` token, and return an empty `events` array when called -with that token, so that it knows that the client has successfully received all -the messages. +token, the server will delete any to-device messages delivered in previous +batches. For the last batch of messages, the server will still send a +`next_batch` token, and return an empty `events` array when called with that +token, so that it knows that the client has successfully received all the +messages and can clean up all the to-device messages for that device. + +If the given `device_id` is not the dehydrated device ID, the server responds +with an error code of `M_FORBIDDEN`, HTTP code 403. ### Device Dehydration Format @@ -205,9 +208,10 @@ The alternatives discussed in MSC2697 are also alternatives here. ## Security considerations -The security consideration in MSC2697 also applies to this proposal: If the -dehydrated device is encrypted using a weak password or key, an attacker could -access it and read the user's encrypted messages. +A similar security consideration to the one in MSC2697 also applies to this +proposal: if SSSS is encrypted using a weak passphrase or key, an attacker +could access it and rehydrate the device to read the user's encrypted +messages. ## Unstable prefix From 0a149c540ebe8b1d581a4e2f0e0b1d1c83a1f131 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Damir=20Jeli=C4=87?= Date: Wed, 9 Aug 2023 14:49:24 +0200 Subject: [PATCH 04/16] Uploading a dehydrated device now uploads the public keys as well --- .../3814-dehydrated-devices-with-ssss.md | 66 +++++++++++++------ 1 file changed, 47 insertions(+), 19 deletions(-) diff --git a/proposals/3814-dehydrated-devices-with-ssss.md b/proposals/3814-dehydrated-devices-with-ssss.md index 5df866c09d..ac5f09837d 100644 --- a/proposals/3814-dehydrated-devices-with-ssss.md +++ b/proposals/3814-dehydrated-devices-with-ssss.md @@ -2,9 +2,9 @@ [MSC2697](https://github.com/matrix-org/matrix-doc/pull/2697) introduces device dehydration -- a method for creating a device that can be stored in a user's -account and receive megolm sessions. In this way, if a user has no other +account and receive Megolm sessions. In this way, if a user has no other devices logged in, they can rehydrate the device on the next login and retrieve -the megolm sessions. +the Megolm sessions. However, the approach presented in that MSC has some downsides, making it tricky to implement in some clients, and presenting some UX difficulties. For @@ -24,28 +24,68 @@ these problems by storing the dehydration key in SSSS, and by not changing the client's device ID. Rather than changing its device ID when it rehydrates the device, it will keep its device ID and upload its own device keys. The client will separately rehydrate the device, fetch its to-device messages, and decrypt -them to retrieve the megolm sessions. +them to retrieve the Megolm sessions. ## Proposal ### Dehydrating a device -The dehydration process is the same as in MSC2697. For completeness, it is -repeated here: +The dehydration process is similar as in MSC2697. One important change is that +the dehydrated device, the public device keys, and one-time keys are all +uploaded in the same request. This change should prevent the creation of +dehydrated devices which do not support end-to-end encryption. To upload a new dehydrated device, a client will use `PUT /dehydrated_device`. Each user has at most one dehydrated device; uploading a new dehydrated device will remove any previously-set dehydrated device. +The client *should* use the public Curve25519 identity key of the device, +encoded as unpadded base64, as the device ID. + +The `device_keys`, `one_time_keys`, and `fallback_keys` fields use the same +structure as for the `/keys/upload` response. + `PUT /dehydrated_device` ```jsonc { + "device_id": "dehydrated_device_id", "device_data": { "algorithm": "m.dehydration.v1.olm" "other_fields": "other_values" }, - "initial_device_display_name": "foo bar" // optional + "initial_device_display_name": "foo bar", // optional + "device_keys": { + "user_id": "", + "device_id": "", + "valid_until_ts": , + "algorithms": [ + "m.olm.curve25519-aes-sha2", + ] + "keys": { + ":": "", + }, + "signatures:" { + "" { + ":": "" + } + } + }, + "fallback_keys": { + ":": "", + "signed_:": { + "fallback": true, + "key": "", + "signatures": { + "": { + ":": "" + } + } + } + }, + "one_time_keys": { + ":": "" + } } ``` @@ -57,18 +97,6 @@ Result: } ``` -After the dehydrated device is uploaded, the client will upload the encryption -keys using `POST /keys/upload/{device_id}`, where the `device_id` parameter is -the device ID given in the response to `PUT /dehydrated_device`. The request -and response formats for `POST /keys/upload/{device_id}` are the same as those -for `POST /keys/upload` with the exception of the addition of the `device_id` -path parameter. - -Note: Synapse already supports `POST /keys/upload/{device_id}` as this was used -in some old clients. However, synapse requires that the given device ID -matches the device ID of the client that made the call. So this will be -changed to allow uploading keys for the dehydrated device. - ### Rehydrating a device To rehydrate a device, a client first calls `GET /dehydrated_device` to see if @@ -183,7 +211,7 @@ the moment. ### Accumulated to-device messages If a dehydrated device is not rehydrated for a long time, then it may -accumulate many to-device messages from other clients sending it megolm +accumulate many to-device messages from other clients sending it Megolm sessions. This may result in a slower initial sync when the device eventually does get rehydrated, due to the number of messages that it will retrieve. Again, this can be addressed by periodically replacing the dehydrated device, From a4e87a624f5f65c1d68f9c8b00d4b59a494bc296 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Damir=20Jeli=C4=87?= Date: Wed, 9 Aug 2023 16:34:01 +0200 Subject: [PATCH 05/16] Make the next_batch token non-optional in the response --- proposals/3814-dehydrated-devices-with-ssss.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/3814-dehydrated-devices-with-ssss.md b/proposals/3814-dehydrated-devices-with-ssss.md index ac5f09837d..1a6df9955e 100644 --- a/proposals/3814-dehydrated-devices-with-ssss.md +++ b/proposals/3814-dehydrated-devices-with-ssss.md @@ -143,7 +143,7 @@ Response: // array of to-device messages, in the same format as in // https://spec.matrix.org/unstable/client-server-api/#extensions-to-sync ], - "next_batch": "token to obtain next events" // optional + "next_batch": "token to obtain next events" } ``` From 3827bc01c96f8392283c7cb2254292f6fb34970f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Damir=20Jeli=C4=87?= Date: Wed, 9 Aug 2023 16:34:37 +0200 Subject: [PATCH 06/16] Let's not delete to-device events when a client receives them --- proposals/3814-dehydrated-devices-with-ssss.md | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/proposals/3814-dehydrated-devices-with-ssss.md b/proposals/3814-dehydrated-devices-with-ssss.md index 1a6df9955e..32913ca484 100644 --- a/proposals/3814-dehydrated-devices-with-ssss.md +++ b/proposals/3814-dehydrated-devices-with-ssss.md @@ -147,14 +147,17 @@ Response: } ``` -Once a client calls `POST /dehydrated_device/{device_id}/events`, the server -can delete the device (though not necessarily its to-device messages). Once a -client calls `POST /dehydrated_device/{device_id}/events` with a `next_batch` -token, the server will delete any to-device messages delivered in previous -batches. For the last batch of messages, the server will still send a +Once a client calls `POST /dehydrated_device/{device_id}/events` with a `next_batch` +token, unlike the `/sync` endpoint, the server should *not* delete any to-device +messages delivered in previous batches. This should prevent the loss of messages +in case the device performing the rehydration gets deleted. In the case the +rehydration process gets aborted, another device will be able to restart the +process. + +For the last batch of messages, the server will still send a `next_batch` token, and return an empty `events` array when called with that -token, so that it knows that the client has successfully received all the -messages and can clean up all the to-device messages for that device. +token, this signals to the client that it has received all to-device events and +it can delete the dehydrated device and create a new one. If the given `device_id` is not the dehydrated device ID, the server responds with an error code of `M_FORBIDDEN`, HTTP code 403. From 12acd435007165243b12ce8b8b1951fb711cf18d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Damir=20Jeli=C4=87?= Date: Wed, 9 Aug 2023 16:35:05 +0200 Subject: [PATCH 07/16] Introduce the DELETE endpoint --- proposals/3814-dehydrated-devices-with-ssss.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/proposals/3814-dehydrated-devices-with-ssss.md b/proposals/3814-dehydrated-devices-with-ssss.md index 32913ca484..53222a1486 100644 --- a/proposals/3814-dehydrated-devices-with-ssss.md +++ b/proposals/3814-dehydrated-devices-with-ssss.md @@ -162,6 +162,24 @@ it can delete the dehydrated device and create a new one. If the given `device_id` is not the dehydrated device ID, the server responds with an error code of `M_FORBIDDEN`, HTTP code 403. +### Deleting a dehydrated device + +A dehydrated device will get replaced whenever a new device gets uploaded using +the `PUT /dehydrated_device`, this makes a `DELETE /dehydrated_device` +unnecessary, though for completeness sake and to give client authors to get back +to a state where no dehydrated device exists for a given user we will introduce +one. + +`DELETE /dehydrated_device` + +Response: + +```json +{ + "device_id": "dehydrated device's ID" +} +``` + ### Device Dehydration Format TODO: define a format. Unlike MSC2679, we don't need to worry about the From f756db3bcb6d6416fca575223a24d886a450f96d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Damir=20Jeli=C4=87?= Date: Thu, 10 Aug 2023 16:21:52 +0200 Subject: [PATCH 08/16] Attempt to define the dehydration format --- .../3814-dehydrated-devices-with-ssss.md | 172 ++++++++++++++++-- 1 file changed, 154 insertions(+), 18 deletions(-) diff --git a/proposals/3814-dehydrated-devices-with-ssss.md b/proposals/3814-dehydrated-devices-with-ssss.md index 53222a1486..771229dd17 100644 --- a/proposals/3814-dehydrated-devices-with-ssss.md +++ b/proposals/3814-dehydrated-devices-with-ssss.md @@ -1,10 +1,10 @@ -# MSC3814: Dehydrated Devices with SSSS +# MSC3814: Dehydrated Devices with [SSSS] -[MSC2697](https://github.com/matrix-org/matrix-doc/pull/2697) introduces device +[MSC2697] introduces device dehydration -- a method for creating a device that can be stored in a user's -account and receive Megolm sessions. In this way, if a user has no other +account and receive [Megolm] sessions. In this way, if a user has no other devices logged in, they can rehydrate the device on the next login and retrieve -the Megolm sessions. +the [Megolm] sessions. However, the approach presented in that MSC has some downsides, making it tricky to implement in some clients, and presenting some UX difficulties. For @@ -13,14 +13,14 @@ calls are made (in particular `/sync`), which may conflict with clients that currently assume that `/sync` can be called immediately after logging in. In addition, the user is required to enter a key or passphrase to create a -dehydrated device. In practice, this is usually the same as the SSSS +dehydrated device. In practice, this is usually the same as the [SSSS] key/passphrase, which means that the user loses the advantage of verifying their other devices via emoji or QR code: either they will still be required to -enter their SSSS key/passphrase (or a separate one for device dehydration), or +enter their [SSSS] key/passphrase (or a separate one for device dehydration), or else that client will not be able to dehydrate a device. This proposal introduces another way to use the dehydrated device that solves -these problems by storing the dehydration key in SSSS, and by not changing the +these problems by storing the dehydration key in [SSSS], and by not changing the client's device ID. Rather than changing its device ID when it rehydrates the device, it will keep its device ID and upload its own device keys. The client will separately rehydrate the device, fetch its to-device messages, and decrypt @@ -30,7 +30,7 @@ them to retrieve the Megolm sessions. ### Dehydrating a device -The dehydration process is similar as in MSC2697. One important change is that +The dehydration process is similar as in [MSC2697]. One important change is that the dehydrated device, the public device keys, and one-time keys are all uploaded in the same request. This change should prevent the creation of dehydrated devices which do not support end-to-end encryption. @@ -39,11 +39,11 @@ To upload a new dehydrated device, a client will use `PUT /dehydrated_device`. Each user has at most one dehydrated device; uploading a new dehydrated device will remove any previously-set dehydrated device. -The client *should* use the public Curve25519 identity key of the device, -encoded as unpadded base64, as the device ID. +The client *must* use the public [Curve25519] [identity key] of the device, +encoded as unpadded Base64, as the device ID. The `device_keys`, `one_time_keys`, and `fallback_keys` fields use the same -structure as for the `/keys/upload` response. +structure as for the [`/keys/upload`] request. `PUT /dehydrated_device` @@ -124,7 +124,7 @@ If the client is able to decrypt the data and wants to use the dehydrated device, the client retrieves the to-device messages sent to the dehydrated device by calling `POST /dehydrated_device/{device_id}/events`, where `{device_id}` is the ID of the dehydrated device. Since there may be many -messages, the response can be sent in batches: the response can include a +messages, the response can be sent in batches: the response must include a `next_batch` parameter, which can be used in a subsequent call to `POST /dehydrated_device/{device_id}/events` to obtain the next batch. @@ -147,12 +147,12 @@ Response: } ``` -Once a client calls `POST /dehydrated_device/{device_id}/events` with a `next_batch` -token, unlike the `/sync` endpoint, the server should *not* delete any to-device -messages delivered in previous batches. This should prevent the loss of messages -in case the device performing the rehydration gets deleted. In the case the -rehydration process gets aborted, another device will be able to restart the -process. +Once a client calls `POST /dehydrated_device/{device_id}/events` with a +`next_batch` token, unlike the `/sync` endpoint, the server should *not* delete +any to-device messages delivered in previous batches. This should prevent the +loss of messages in case the device performing the rehydration gets deleted. In +the case the rehydration process gets aborted, another device will be able to +restart the process. For the last batch of messages, the server will still send a `next_batch` token, and return an empty `events` array when called with that @@ -192,11 +192,135 @@ will have the advantage that we don't need to figure out a format that will fit into every possible implementation's idiosyncrasies. The format will be encrypted, which leads to ... +```text + ┌───────────────────────────────────────────────────────────┐ + │ Pickle │ + ├───────────────────────────────────────────────────────────┤ + │Name │ Type │ Size (bytes) │ + ├────────────────────────┼───────────────┼──────────────────┤ + │Version │ u32 │ 4 │ + │Ed25519 key pair │ KeyPair │ 64 │ + │Curve25519 key pair │ KeyPair │ 64 │ + │Number of one-time keys │ u32 │ 4 │ + │One-time keys │ [OneTimeKey] │ N * 69 │ + │Fallback keys │ FallbackKeys │ 2 * 69 │ + │Next key ID │ u32 │ 4 │ + └────────────────────────┴───────────────┴──────────────────┘ + ┌───────────────────────────────────────────────────────────┐ + │ KeyPair │ + ├────────────────────────┬───────────────┬──────────────────┤ + │Name │ Type │ Size (bytes) │ + ├────────────────────────┼───────────────┼──────────────────┤ + │Public key │ [u8; 32] │ 32 │ + │Private key │ [u8; 32] │ 32 │ + └────────────────────────┴───────────────┴──────────────────┘ + + ┌───────────────────────────────────────────────────────────┐ + │ OneTimeKey │ + ├────────────────────────┬───────────────┬──────────────────┤ + │Name │ Type │ Size │ + ├────────────────────────┼───────────────┼──────────────────┤ + │Key ID │ u32 │ 4 bytes │ + │Is published │ u8 │ 1 byte │ + │Curve 25519 key pair │ KeyPair │ 69 bytes │ + └────────────────────────┴───────────────┴──────────────────┘ + + ┌───────────────────────────────────────────────────────────┐ + │ FallbackKeys │ + ├────────────────────────┬───────────────┬──────────────────┤ + │Name │ Type │ Size │ + ├────────────────────────┼───────────────┼──────────────────┤ + │Number of fallback keys │ u8 │ 1 byte │ + │Fallback-key │ OneTimeKey │ 69 bytes │ + │Previous fallback-key │ OneTImeKey │ 69 bytes │ + └────────────────────────┴───────────────┴──────────────────┘ +``` + +TODO: Explain why we must ignore public keys when decoding them and why they are +included in the first place. + +When decoding, clients *must* ignore the public keys and instead derive the +public key from the private one. + #### Encryption key +TODO: Decide if the Latex format or the pseudocode format is preferred, or maybe +both. + +TODO: Explain why the double derivation is necessary. + The encryption key used for the dehydrated device will be randomly generated and stored/shared via SSSS using the name `m.dehydrated_device`. +The randomly generated encryption key *must* be expanded using the HMAC-based +Key Derivation function defined in [RFC5869]. The notation in this document are +to be interpreted as described in [RFC5869]. + +$$ +\begin{aligned} + DEVICE\_KEY + &= \operatorname{HKDF} \left( + \text{``Device ID``}, + RANDOM\_KEY, + \text{``dehydrated-device-pickle-key"}, + 32\right) +\end{aligned} +$$ + +```text +PRK = HKDF-Extract("Device ID", random_encryption_key) +device_key = HKDF-Expand(PRK, "dehydrated-device-pickle-key", 32) +``` + +The `device_key` is then further expanded into a AES256 key, HMAC key and +initialization vector. + + +$$ +\begin{aligned} + AES\_KEY\;\parallel\;HMAC\_KEY\;\parallel\;AES\_IV + &= \operatorname{HKDF}\left(0,DEVICE\_KEY,\text{``Pickle"},80\right) +\end{aligned} +$$ + +```text +PRK = HKDF-Extract("", DeviceKey) +output = HKDF-Expand(PRK, "Pickle", 80) + +aes_key = output[0..32] +mac_key = output[32..64] +initialization_vector = output[64..80] +``` + +The plain-text is encrypted with [AES-256] in [CBC] mode with [PKCS7] padding, +using the key $`AES\_KEY`$ and the IV $`AES\_IV`$ to give the cipher-text. + +Then the cipher-text are passed through [HMAC-SHA-256]. The first 8 bytes of the +MAC are appended to the cipher-text. + +The cipher-text, including the appended MAC tag, are encoded using unpadded +Base64 to give the device pickle. + +The device pickle can be inserted into the `device_pickle` field of the +`device_data` JSON message. + + ```json +{ + "device_data": { + "algorithm": "m.dehydration.v1.olm", + "device_pickle": "encrypted dehydrated device" + } +} +``` + +#### Test vectors + +Device pickle: +``` +Gc0elC7k7NISzWW/C2UIuzRMDSHzzRLfM3lMnJHMLMcuyLtZHljhV/YvIctIlepxevznEcwBc40Q0CtS3k5SI9gGyN7G+95hnQan0rKe64a1Vx1Vx4Ky8i+m1y9JVT++WcQ54CGhMuCGoN2O1xEQb+4fM+UVS/bLNJ4Pzzqa1ilzCrs4SCTz70eriShvzt7y1cn2A6ABNhK4aXnLB8gK9HuMLyctyX5ikvIjkAIAdVr1EI1azetZDQ +``` + + ## Potential issues The same issues as in @@ -275,3 +399,15 @@ dehydration algorithm `m.dehydration.v1.olm` will be called ## Dependencies None + +[RFC5869]: https://datatracker.ietf.org/doc/html/rfc5869 +[AES-256]: http://csrc.nist.gov/publications/fips/fips197/fips-197.pdf +[CBC]: http://csrc.nist.gov/publications/nistpubs/800-38a/sp800-38a.pdf +[PKCS#7]: https://tools.ietf.org/html/rfc2315 +[Curve25519]: http://cr.yp.to/ecdh.html +[identity key]: https://gitlab.matrix.org/matrix-org/olm/-/blob/master/docs/olm.md#initial-setup +[Megolm]: https://gitlab.matrix.org/matrix-org/olm/blob/master/docs/megolm.md +[SSSS]: https://spec.matrix.org/v1.7/client-server-api/#storage +[MSC2697]: https://github.com/matrix-org/matrix-doc/pull/2697 +[`/keys/upload`]: https://spec.matrix.org/v1.7/client-server-api/#post_matrixclientv3keysupload +[device keys]: https://spec.matrix.org/v1.7/client-server-api/#device-keys From 6223db448a2a36a6c2fcb48cac13a95506694cb2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Damir=20Jeli=C4=87?= Date: Thu, 10 Aug 2023 16:49:31 +0200 Subject: [PATCH 09/16] Don't use operatorname, try to unwedge the Latex --- proposals/3814-dehydrated-devices-with-ssss.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/proposals/3814-dehydrated-devices-with-ssss.md b/proposals/3814-dehydrated-devices-with-ssss.md index 771229dd17..840454b8e0 100644 --- a/proposals/3814-dehydrated-devices-with-ssss.md +++ b/proposals/3814-dehydrated-devices-with-ssss.md @@ -259,7 +259,7 @@ to be interpreted as described in [RFC5869]. $$ \begin{aligned} DEVICE\_KEY - &= \operatorname{HKDF} \left( + &= \text{HKDF} \left( \text{``Device ID``}, RANDOM\_KEY, \text{``dehydrated-device-pickle-key"}, @@ -279,7 +279,7 @@ initialization vector. $$ \begin{aligned} AES\_KEY\;\parallel\;HMAC\_KEY\;\parallel\;AES\_IV - &= \operatorname{HKDF}\left(0,DEVICE\_KEY,\text{``Pickle"},80\right) + &= \text{HKDF}\left(0,DEVICE\_KEY,\text{``Pickle"},80\right) \end{aligned} $$ From e3c9ac8be25c51c1b6783b846520bbed4baa8e34 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Damir=20Jeli=C4=87?= Date: Thu, 10 Aug 2023 16:53:00 +0200 Subject: [PATCH 10/16] More Latex tweaks --- proposals/3814-dehydrated-devices-with-ssss.md | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-) diff --git a/proposals/3814-dehydrated-devices-with-ssss.md b/proposals/3814-dehydrated-devices-with-ssss.md index 840454b8e0..f343fc4d67 100644 --- a/proposals/3814-dehydrated-devices-with-ssss.md +++ b/proposals/3814-dehydrated-devices-with-ssss.md @@ -259,11 +259,7 @@ to be interpreted as described in [RFC5869]. $$ \begin{aligned} DEVICE\_KEY - &= \text{HKDF} \left( - \text{``Device ID``}, - RANDOM\_KEY, - \text{``dehydrated-device-pickle-key"}, - 32\right) + &= \text{HKDF} \left(\text{``Device ID``}, RANDOM\_KEY, \text{``dehydrated-device-pickle-key"}, 32\right) \end{aligned} $$ @@ -278,7 +274,7 @@ initialization vector. $$ \begin{aligned} - AES\_KEY\;\parallel\;HMAC\_KEY\;\parallel\;AES\_IV + AES\_KEY \parallel HMAC\_KEY \parallel AES\_IV &= \text{HKDF}\left(0,DEVICE\_KEY,\text{``Pickle"},80\right) \end{aligned} $$ @@ -292,7 +288,7 @@ mac_key = output[32..64] initialization_vector = output[64..80] ``` -The plain-text is encrypted with [AES-256] in [CBC] mode with [PKCS7] padding, +The plain-text is encrypted with [AES-256] in [CBC] mode with [PKCS#7] padding, using the key $`AES\_KEY`$ and the IV $`AES\_IV`$ to give the cipher-text. Then the cipher-text are passed through [HMAC-SHA-256]. The first 8 bytes of the From 7f24f0d05f07997c2f5f61f31891674e1569d94d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Damir=20Jeli=C4=87?= Date: Tue, 5 Sep 2023 10:46:26 +0200 Subject: [PATCH 11/16] Remove the bytes unit from every single row, put it in the header --- proposals/3814-dehydrated-devices-with-ssss.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/proposals/3814-dehydrated-devices-with-ssss.md b/proposals/3814-dehydrated-devices-with-ssss.md index f343fc4d67..651f1ca52a 100644 --- a/proposals/3814-dehydrated-devices-with-ssss.md +++ b/proposals/3814-dehydrated-devices-with-ssss.md @@ -218,21 +218,21 @@ encrypted, which leads to ... ┌───────────────────────────────────────────────────────────┐ │ OneTimeKey │ ├────────────────────────┬───────────────┬──────────────────┤ - │Name │ Type │ Size │ + │Name │ Type │ Size (bytes) │ ├────────────────────────┼───────────────┼──────────────────┤ - │Key ID │ u32 │ 4 bytes │ - │Is published │ u8 │ 1 byte │ - │Curve 25519 key pair │ KeyPair │ 69 bytes │ + │Key ID │ u32 │ 4 │ + │Is published │ u8 │ 1 │ + │Curve 25519 key pair │ KeyPair │ 69 │ └────────────────────────┴───────────────┴──────────────────┘ ┌───────────────────────────────────────────────────────────┐ │ FallbackKeys │ ├────────────────────────┬───────────────┬──────────────────┤ - │Name │ Type │ Size │ + │Name │ Type │ Size (bytes) │ ├────────────────────────┼───────────────┼──────────────────┤ - │Number of fallback keys │ u8 │ 1 byte │ - │Fallback-key │ OneTimeKey │ 69 bytes │ - │Previous fallback-key │ OneTImeKey │ 69 bytes │ + │Number of fallback keys │ u8 │ 1 │ + │Fallback-key │ OneTimeKey │ 69 │ + │Previous fallback-key │ OneTImeKey │ 69 │ └────────────────────────┴───────────────┴──────────────────┘ ``` From f85c18db89e454d539958e152c6202f36d5736fd Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Damir=20Jeli=C4=87?= Date: Tue, 5 Sep 2023 10:46:59 +0200 Subject: [PATCH 12/16] Attempt to fix the math rendering --- .../3814-dehydrated-devices-with-ssss.md | 29 ++++--------------- 1 file changed, 6 insertions(+), 23 deletions(-) diff --git a/proposals/3814-dehydrated-devices-with-ssss.md b/proposals/3814-dehydrated-devices-with-ssss.md index 651f1ca52a..67611bf44c 100644 --- a/proposals/3814-dehydrated-devices-with-ssss.md +++ b/proposals/3814-dehydrated-devices-with-ssss.md @@ -244,48 +244,30 @@ public key from the private one. #### Encryption key -TODO: Decide if the Latex format or the pseudocode format is preferred, or maybe -both. - TODO: Explain why the double derivation is necessary. The encryption key used for the dehydrated device will be randomly generated and stored/shared via SSSS using the name `m.dehydrated_device`. The randomly generated encryption key *must* be expanded using the HMAC-based -Key Derivation function defined in [RFC5869]. The notation in this document are -to be interpreted as described in [RFC5869]. +Key Derivation function defined in [RFC5869]. -$$ +```math \begin{aligned} DEVICE\_KEY &= \text{HKDF} \left(\text{``Device ID``}, RANDOM\_KEY, \text{``dehydrated-device-pickle-key"}, 32\right) \end{aligned} -$$ - -```text -PRK = HKDF-Extract("Device ID", random_encryption_key) -device_key = HKDF-Expand(PRK, "dehydrated-device-pickle-key", 32) ``` The `device_key` is then further expanded into a AES256 key, HMAC key and initialization vector. -$$ +```math \begin{aligned} AES\_KEY \parallel HMAC\_KEY \parallel AES\_IV &= \text{HKDF}\left(0,DEVICE\_KEY,\text{``Pickle"},80\right) \end{aligned} -$$ - -```text -PRK = HKDF-Extract("", DeviceKey) -output = HKDF-Expand(PRK, "Pickle", 80) - -aes_key = output[0..32] -mac_key = output[32..64] -initialization_vector = output[64..80] ``` The plain-text is encrypted with [AES-256] in [CBC] mode with [PKCS#7] padding, @@ -297,12 +279,12 @@ MAC are appended to the cipher-text. The cipher-text, including the appended MAC tag, are encoded using unpadded Base64 to give the device pickle. -The device pickle can be inserted into the `device_pickle` field of the +The device pickle is then inserted into the `device_pickle` field of the `device_data` JSON message. ```json { - "device_data": { + "device_data": { "algorithm": "m.dehydration.v1.olm", "device_pickle": "encrypted dehydrated device" } @@ -407,3 +389,4 @@ None [MSC2697]: https://github.com/matrix-org/matrix-doc/pull/2697 [`/keys/upload`]: https://spec.matrix.org/v1.7/client-server-api/#post_matrixclientv3keysupload [device keys]: https://spec.matrix.org/v1.7/client-server-api/#device-keys +[HMAC-SHA-256]: https://datatracker.ietf.org/doc/html/rfc2104 From 4954c274c69700b2c00a4b0a071434142364a36f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Damir=20Jeli=C4=87?= Date: Tue, 5 Sep 2023 11:11:10 +0200 Subject: [PATCH 13/16] Align the table headers for the pickle format --- proposals/3814-dehydrated-devices-with-ssss.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/proposals/3814-dehydrated-devices-with-ssss.md b/proposals/3814-dehydrated-devices-with-ssss.md index 67611bf44c..4953719cd3 100644 --- a/proposals/3814-dehydrated-devices-with-ssss.md +++ b/proposals/3814-dehydrated-devices-with-ssss.md @@ -194,7 +194,7 @@ encrypted, which leads to ... ```text ┌───────────────────────────────────────────────────────────┐ - │ Pickle │ + │ Pickle │ ├───────────────────────────────────────────────────────────┤ │Name │ Type │ Size (bytes) │ ├────────────────────────┼───────────────┼──────────────────┤ @@ -207,7 +207,7 @@ encrypted, which leads to ... │Next key ID │ u32 │ 4 │ └────────────────────────┴───────────────┴──────────────────┘ ┌───────────────────────────────────────────────────────────┐ - │ KeyPair │ + │ KeyPair │ ├────────────────────────┬───────────────┬──────────────────┤ │Name │ Type │ Size (bytes) │ ├────────────────────────┼───────────────┼──────────────────┤ @@ -216,7 +216,7 @@ encrypted, which leads to ... └────────────────────────┴───────────────┴──────────────────┘ ┌───────────────────────────────────────────────────────────┐ - │ OneTimeKey │ + │ OneTimeKey │ ├────────────────────────┬───────────────┬──────────────────┤ │Name │ Type │ Size (bytes) │ ├────────────────────────┼───────────────┼──────────────────┤ @@ -226,7 +226,7 @@ encrypted, which leads to ... └────────────────────────┴───────────────┴──────────────────┘ ┌───────────────────────────────────────────────────────────┐ - │ FallbackKeys │ + │ FallbackKeys │ ├────────────────────────┬───────────────┬──────────────────┤ │Name │ Type │ Size (bytes) │ ├────────────────────────┼───────────────┼──────────────────┤ From 087154af6a7a42f3684c3ee9dd52a0914f8bb27d Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Wed, 7 Feb 2024 20:39:39 -0500 Subject: [PATCH 14/16] Fix JSON example --- proposals/3814-dehydrated-devices-with-ssss.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/proposals/3814-dehydrated-devices-with-ssss.md b/proposals/3814-dehydrated-devices-with-ssss.md index 4953719cd3..29564bd17c 100644 --- a/proposals/3814-dehydrated-devices-with-ssss.md +++ b/proposals/3814-dehydrated-devices-with-ssss.md @@ -65,8 +65,8 @@ structure as for the [`/keys/upload`] request. "keys": { ":": "", }, - "signatures:" { - "" { + "signatures": { + "": { ":": "" } } From e7c8266c52ca665ce623052cff403a506064a8f7 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Mon, 12 Feb 2024 11:43:03 -0500 Subject: [PATCH 15/16] link to fallback key spec --- proposals/3814-dehydrated-devices-with-ssss.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/proposals/3814-dehydrated-devices-with-ssss.md b/proposals/3814-dehydrated-devices-with-ssss.md index 29564bd17c..e9857e7560 100644 --- a/proposals/3814-dehydrated-devices-with-ssss.md +++ b/proposals/3814-dehydrated-devices-with-ssss.md @@ -315,8 +315,7 @@ will not be able to share megolm keys. This issue is not unique to dehydrated devices; this also occurs when devices are offline for an extended period of time. -This may be addressed by using fallback keys as described in -[MSC2732](https://github.com/matrix-org/matrix-doc/pull/2732). +This may be addressed by using [fallback keys](https://spec.matrix.org/v1.9/client-server-api/#one-time-and-fallback-keys). To reduce the chances of one-time key exhaustion, if the user has an active client, it can periodically replace the dehydrated device with a new dehydrated From cf5ae9939a9502917cea946d0495d75e70b8f583 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Fri, 23 Feb 2024 17:04:14 -0500 Subject: [PATCH 16/16] add dehydrated flag --- proposals/3814-dehydrated-devices-with-ssss.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/proposals/3814-dehydrated-devices-with-ssss.md b/proposals/3814-dehydrated-devices-with-ssss.md index e9857e7560..12d5a33081 100644 --- a/proposals/3814-dehydrated-devices-with-ssss.md +++ b/proposals/3814-dehydrated-devices-with-ssss.md @@ -45,6 +45,21 @@ encoded as unpadded Base64, as the device ID. The `device_keys`, `one_time_keys`, and `fallback_keys` fields use the same structure as for the [`/keys/upload`] request. +We add a new optional property to the device keys: `dehydrated`, which is set +to `true` for dehydrated devices. Defaults to `false`. Clients can use this +flag to handle the dehydrated device specially. For example: + +- display dehydrated devices differently from normal devices, to avoid + confusing from users who do not expect to see another device +- don't send key forwarding requests to the dehydrated device, since it will + not respond to them +- don't send room keys to the dehydrated device if the user has a sufficient + number of other devices, with the assumption that if the user logs in to a + new device, they can get the room keys from one of their other devices and/or + from key backup. This will reduce the chances that the dehydrated device + will run out of one-time keys, and reduce the number of events that the + dehydrated device will need to decrypt. + `PUT /dehydrated_device` ```jsonc @@ -59,6 +74,7 @@ structure as for the [`/keys/upload`] request. "user_id": "", "device_id": "", "valid_until_ts": , + "dehydrated": true, "algorithms": [ "m.olm.curve25519-aes-sha2", ]