- TAP: 7
- Title: Conformance testing
- Version: 2
- Last-Modified: 14-August-2017
- Author: Vladimir Diaz, Sebastien Awwad
- Status: Rejected
- Content-Type: text/markdown
- Created: 20-Jan-2017
- Post-History: v1: 12-June-2017, v2: None
NOTE: This TAP will be informally implemented but will not be part of the specification for TUF.
Conformance testing can determine whether an implementation meets the requirements set by a given specification. At this point, no tool or set of data exists to help developers and users affirm that an implementation of an update system behaves according to the TUF specification. Although the reference implementation contains unit tests that verify correct behavior (such as updating metadata in the expected order and blocking known updater attacks) these unit tests only work within the parameters of the reference implementation. This is problematic due to the diversity of TUF implementations.
This proposal describes the design of a data configuration and format for defining test cases that determine TUF compliance - most importantly, for testing resilience to attacks against updaters.
The goals are to enable determination of whether or not an updater implementation conforms to the TUF specification, to interoperate with implementations in diverse languages and environments, and to minimize the burden of such testing on implementers.
Developers need a convenient way of verifying whether an implementation conforms to the TUF specification, affirming that the tested implementation meets a recognized standard of secure operation.
Up to this point, adopters of TUF who had written an implementation could only test for conformance by (1) verifying that metadata generated in some language X matches that of the reference implementation, and/or (2) reproducing the unit tests of the reference implementation in language X. In the first case, only the metadata generated by X can be said to conform to the specifications. The client would still need to test for the expected behavior when the generated metadata is updated. In the second case, the implementation is said to conform depending on how thoroughly the unit tests are reproduced in X. There are bound to be inconsistencies between the two sets of unit tests. Any improvements in TUF testing or changes to the program would require implementers to add test code in parallel.
The ability to distribute test data accompanied by prescriptions for the behavior expected from the updater processing each test case can ensure update behavior as intended by the designers of TUF, and, most importantly, ensure that an updater is secure against the types of attacks and weaknesses listed in Section 1.5.2 of the TUF Specification.
The strategy for testing TUF conformance proposed in this TAP is to generate sets of metadata and targets with the expectation that all implementations should react to certain sets by successfully validating and updating, and react to other sets by rejecting the invalid metadata or targets. Determining which data sets an implementation accepts and which it rejects helps to determine the implementation's TUF conformance, including its resilience against the attacks listed in the TUF Specification (section 1.5.2). If an updater implementation performs as prescribed in every test case, it is likely that it complies with the TUF Specification.
For instance, a set of test data may provide metadata signed by an an untrusted key to test whether or not the implementation will reject an untrusted signature.
While test cases may note the expected reason for an expected failure to update, it is not expected that specific errors be raised; the only prescription is that an updater correctly update or fail to update. TUF conformance shouldn't depend on specific kinds of errors being communicated.
The validation behavior during testing should not vary significantly from that in production so that test results can represent real updater performance.
The official test cases should be publicly available so that anyone can use them to test their updater implementation.
The client updater implementation to be tested should operate as described in the Client Workflow in Section 5 of the TUF Specification. Tests will verify this behavior, determining whether or not the updater defends against attacks described in Section 1.5.2 of the TUF Specification.
The updater will be expected to reject untrustworthy metadata and targets and accept trustworthy metadata and targets. Positive test cases will be used to verify correct behavior in the absence of an attack.
Tests will attempt endless data attacks, indefinite freeze attacks, replay attacks, and a variety of others discussed in the TUF Specification. The full listing of conformance tests and expected results will be provided in documentation alongside the TUF Specification.
Each test case will be specified with the following pieces of information:
- Description
- Expected Result
- Initial Trusted Metadata
- Repository Data
- Target to Install
The Description field will indicate the purpose of the test -- what attack it expects to test the updater's defense against, or what problematic updater behavior it checks for.
For example:
Test defense against metadata replay attacks: provide a snapshot role that
has a lower version than that already trusted by the client.
This will be 'Success' or 'Failure'. If the value is 'Success', that will mean that the client is expected to indicate that it was able to validate the provided metadata and the target the updater was instructed to "install" ('Install Instruction') was deemed valid and OK to install. If the value is 'Failure', this means that the client is expected to indicate that the target could not be installed - whether because an attack should be detected in the metadata, or the target doesn't match values provided in the metadata, or so on.
This is the path to a directory containing initial metadata the updater client will have and trust before an update is attempted. TUF always requires some established root of trust to be present in the updater client. The common case is generally a trustworthy root.json file that shipped with the updater. For testing purposes, this may also be a full set of metadata that would have been validated in a hypothetical previous update.
Metadata in the TUF specification's metadata format will be provided in the
directory, with the directory structure below. Data here should be converted to
whatever format the updater requires and delivered in the manner the updater
requires. Also provided in the indicated directory is a keys.json file, in
case metadata has to be converted to a different format and re-signed, as
explained below.
Additional files may appear in test data directories to support new TUF features. See Adapting TAP 7 for Future Changes below.
In most cases, the contents of the directory indicated as Initial Trusted Metadata will simply be:
- keys.json
- metadata
|- root.json
But more may be provided:
- keys.json
- metadata
|- root.json
|- timestamp.json
|- snapshot.json
|- targets.json
|- <a delegated role>.json
|- <another delegated role>.json
| ...
The keys.json file will specify the keys used to generate the given metadata.
This information does not need to be used if the updater implementation uses
the same structure and canonicalized JSON metadata format described in the TUF
Specification and used by the TUF Reference Implementation. If metadata has to
be converted and signed over a different format for the updater, these keys can
be used to re-sign the metadata and generate equivalent metadata in the new
format.
The format of this dictionary of keys represented in keys.json is as follows.
(Note that the individual keys resemble ANYKEY_SCHEMA in the
TUF format definitions)
{
<rolename_1>: [ // This role should be signed by these two keys:
{'keytype': <type, e.g. 'ed25519'>,
'keyid': <id string>,
'keyval': {'public': <key string>, 'private': <key string>},
},
{'keytype': <type, e.g. 'ed25519'>,
'keyid': <id string>,
'keyval': {'public': <key string>, 'private': <key string>},
}],
<rolename_2>: [...],
...
}This listing indicates which key(s) should be used to sign each role in the test metadata. Sometimes (in the case of some attacks), these will not be the correct keys for the role.
Here's an excerpt from a particular example:
{
'root': [{
'keytype': 'ed25519',
'keyid': '94c836f0c45168f0a437eef0e487b910f58db4d462ae457b5730a4487130f290',
'keyval': {
'public': 'f4ac8d95cfdf65a4ccaee072ba5a48e8ad6a0c30be6ffd525aec6bc078211033',
'private': '879d244c6720361cf1f038a84082b08ac9cd586c32c1c9c6153f6db61b474957'}}],
'timestamp': [{
'keytype': 'ed25519',
'keyid': '6fcd9a928358ad8ca7e946325f57ec71d50cb5977a8d02c5ab0de6765fef040a',
'keyval': {
'public': '97c1112bbd9047b1fdb50dd638bfed6d0639e0dff2c1443f5593fea40e30f654',
'private': 'ef373ea36a633a0044bbca19a298a4100e7f353461d7fe546e0ec299ac1b659e'}}],
...
'delegated_role1': [{
'keytype': 'ed25519',
'keyid': '8650aed05799a74f5febc9070c5d3e58d62797662d48062614b1ce0a643ee368',
'keyval': {
'public': 'c5a78db3f3ba96462525664e502f2e7893b81e7e270d75ffb9a6bb95b56857ca',
'private': '134dc07435cd0d5a371d51ee938899c594c578dd0a3ab048aa70de5dd71f99f2'}}]
}This component of the test set is a directory of data - metadata and targets, along with the keys used to sign the metadata - that should be made available to the updater when it tries to update. This data should be treated normally by the updater (i.e. not as initially-shipped, trusted data).
It is similar to Initial Trusted Metadata in its form, but will have a
targets directory alongside the metadata directory. For example:
- keys.json
- metadata
|- root.json
|- timestamp.json
|- snapshot.json
|- targets.json
|- <a delegated role>.json
|- <another delegated role>.json
| ...
- targets
|- <some_target.img>
|- ...
keys.json is as specified above.
Filepaths in the targets directory map directly to the filepaths used to identify targets in the repository. For example, a target identified in metadata with the filepath 'package1/tarball.tar' would be found in 'targets/package1/tarball.tar'.
The path of a target file that the updater should try to update/install, as listed in the metadata provided. Filepaths are relative to the root of targets directories. It is not necessary for the updater to have a notion of a filesystem; this piece of information is intended to specify the target to update to.
The test cases, containing the elements above (see for details on each element) will be provided as a directory accompanied by a JSON file in the following JSON format.
[
// Each test case is an element in this list.
{
'description': 'A description of the test case',
'expected_result': <either 'success' or 'failure'>,
'initial_data': <path to directory containing the metadata to use>,
'repository_data': <path to directory containing data to use>,
'target': <repository filepath of target to try obtaining and updating to>
},
{
...
}
]Example:
[
{
'description': '0: Basic positive case, valid test data',
'expected_result': 'success',
'initial_data': 'test_sets/test0/initial/',
'repository_data': 'test_sets/test0/repository/',
'target': 'target.txt'
},
{
'description':
'1: replay: snapshot w/ lower version than that already trusted by client',
'expected_result': 'failure',
'initial_data': 'test_sets/test1/initial/',
'repository_data': 'test_sets/test1/repository/',
'target': 'target.txt'
},
{
'description':
'2: target file provided does not match hash in trustworthy metadata',
'expected_result': 'failure',
'initial_data': 'test_sets/test2/initial/',
'repository_data': 'test_sets/test2/repository/',
'target': 'target.txt'
},
{
'description':
'3: delegated role is signed by key no longer trusted by delegating role'
'expected_result': 'failure',
'initial_data': 'test_sets/test3/initial/',
'repository_data': 'test_sets/test3/repository/',
'target': 'target_123.txt'
},
{
'description': '4: Positive w/ delegations (also controls for Test 3)',
'expected_result': 'success',
'initial_data': 'test_sets/test3c/initial/',
'repository_data': 'test_sets/test3c/repository/',
'target': 'target_123.txt'
}
]Because different updaters may operate very differently, feeding the test data to the client updater may require additional work. The Dealing with Implementation Restrictions section below addresses a variety of such scenarios in detail. Here are some examples of what may be necessary:
- move metadata or target files into the directory structure an updater implementation expects.
- if, e.g., the updater doesn't have a notion of a filesystem, read the files the provided and distribute data to the updater in the manner the updater expects.
- if the updater uses a different metadata format (for example, ASN.1/DER instead of canonical JSON -- or just a different arrangement in canonical JSON), translate metadata (Initial and Repository) from the format the Tester provides into the format the updater expects.
- if the updater requires signatures to be over a different format, re-sign
metadata (using
keys.json) after translating it.
If an updater implementation uses a different metadata format than that of the test data, it is necessary to convert test metadata for the updater's use. Further, two situations arise depending on the behavior of the updater:
- When the updater receives the converted metadata, it converts it back to canonical JSON and checks the signatures in this (original) format.
- When the updater receives the converted metadata, it will not convert it back into canonical JSON, and will expect the signatures to be over the same new format (foreign to TUF).
In the second case, the converted metadata must also be re-signed, so that the updater will be able to correctly validate the metadata.
For this reason, a keys.json file is provided with any test case metadata.
This file,
described above, indicates which keys should be used to
sign each piece of metadata, if that metadata should need to be converted and
re-signed.
For an example of how such re-signing code might look, consider the JSON-to-DER
converter convert_signed_metadata_to_der employed by the Uptane Reference
Implementation's current TUF fork
here.
Accommodating future TUF features or TAPs may require a slightly different arrangements of test data. TAP 4 support is provided below as an example. Future TAPs that affect test data should specify how test data must be changed.
Operating with multiple repositories, an ability slated for addition to TUF in TAP 4, requires that test data provide for multiple repositories. TAP 4 support expands the formats in this TAP somewhat, as indicated below.
For TAP 4 support in TAP 7, these changes are required:
- Include map.json file with Initial Trusted Metadata
- Separate metadata and targets in data directories by repository (in Initial Trusted Metadata and Repository Data)
- Organize keys.json by repository
TAP 4 requires that clients have a map.json to identify known
repositories and the client's trust relationships with them / delegations to
them (which repositories or combinations of repositories a client trusts to
provide target files in different namespaces. For TAP 4 Support in TAP 7, a
map.json file is included in
Initial Trusted Metadata.
map.json normally maps:
- target file namespaces to (groups of) repository names to trust for those target files
- repository names to the locations where clients can find data for each repository
As normal, the repository names will tell you which directories will contain relevant test data for a given repository, and which repository to trust for which targets. See the next two sections on TAP 4 Support below.
Since test setups vary, however, we cannot in pre-generated test data map the names of repositories to the locations of mirrors or assume what transport mechanism will be employed. The locations listed should be corrected to reflect the real location of the repositories where test data will be made available to the clients. (In more exotic setups -- for example, if http(s):// or file:// are not to be used and repository data is distributed through some other protocol -- the map file should be adjusted to ensure that the client is able to find the appropriate test data for each repository.)
For example, here is a map file that requires Repository1 and Repository2 to
agree on target file info for targets in project123/, trusts Repository3
alone for all other projects, and specifies the (irrelevant for test data)
location of the repositories.
{
"repositories": {
"Repository1": [<test_case_0/repo1>],
"Repository2": [<test_case_0/repo2>],
"Repository3": ...,
},
"mapping": [
{
"paths": ["project123/*"],
"repositories": ["Repository1", "Repository2"],
"terminating": true,
},
{
"paths": ["*"],
"repositories": ["Repository3"]
}
]
}When TAP 4 is supported, the structure of the directories of metadata and
targets in the test cases' Initial Trusted Metadata
and Repository Data changes slightly to organize contents
by repository, using the repository names listed in map.json. An extra level of
directories is added, one per repository named in map.json, each containing
'metadata' (and in the case of Repository Data, 'targets') directories.
Initial Trusted Metadata will now look like this:
- map.json // See section above on map.json
- keys.json
- <repository_1_name>
|- metadata
|- root.json
...
- <repository_2_name>
|- metadata
|- root.json
...
...
And Repository Data will now look like this:
- keys.json
- <repository_1_name>
|- metadata
|- root.json
|- timestamp.json
|- snapshot.json
|- targets.json
|- <a delegated role>.json
|- <another delegated role>.json
| ...
|- targets
|- <some_target.img>
|- ...
- <repository_2_name>
|- metadata
|- root.json
|- ...
|- targets
|- ...
...
When TAP 4 is supported, the keys.json file provided in Initial Trusted
Metadata and Repository Data gains an additional level of strucutre at the
root of the dictionary, separating keys by repository, using repository names
from map.json, like this:
{
<repository_1_name>: {
<rolename_1>: [ // This role should be signed by these two keys:
{'keytype': <type, e.g. 'ed25519'>,
'keyid': <id string>,
'keyval': {'public': <key string>, 'private': <key string>},
},
{'keytype': <type, e.g. 'ed25519'>,
'keyid': <id string>,
'keyval': {'public': <key string>, 'private': <key string>},
}],
<rolename_2>: [...]},
<repository_2_name>: {...}
}A particular example of keys.json with TAP 4 support:
{
'imagerepo': {
{'root': [{
'keytype': 'ed25519',
'keyid': '94c836f0c45168f0a437eef0e487b910f58db4d462ae457b5730a4487130f290',
'keyval': {
'public': 'f4ac8d95cfdf65a4ccaee072ba5a48e8ad6a0c30be6ffd525aec6bc078211033',
'private': '879d244c6720361cf1f038a84082b08ac9cd586c32c1c9c6153f6db61b474957'}}]},
{'timestamp': [{
'keytype': 'ed25519',
'keyid': '6fcd9a928358ad8ca7e946325f57ec71d50cb5977a8d02c5ab0de6765fef040a',
'keyval': {
'public': '97c1112bbd9047b1fdb50dd638bfed6d0639e0dff2c1443f5593fea40e30f654',
'private': 'ef373ea36a633a0044bbca19a298a4100e7f353461d7fe546e0ec299ac1b659e'}}]},
...
{'delegated_role1': [{
'keytype': 'ed25519',
'keyid': '8650aed05799a74f5febc9070c5d3e58d62797662d48062614b1ce0a643ee368',
'keyval': {
'public': 'c5a78db3f3ba96462525664e502f2e7893b81e7e270d75ffb9a6bb95b56857ca',
'private': '134dc07435cd0d5a371d51ee938899c594c578dd0a3ab048aa70de5dd71f99f2'}}]}
},
'director': {
{'root': [{
...This TAP does not detract from existing security guarantees because it does not propose architectural changes to the specification.
This TAP does not introduce any backward incompatibilities.
Full test sets will be written after this TAP is accepted, and that will be linked to from here. Code employing them to test the TUF Reference Implementation will also be provided, as a further guide for their use.
This document has been placed in the public domain.