diff --git a/docs/core/testing/media/icommandlineoptionsprovider-sequence-diagram.png b/docs/core/testing/media/icommandlineoptionsprovider-sequence-diagram.png new file mode 100644 index 0000000000000..6f69457b7aff6 Binary files /dev/null and b/docs/core/testing/media/icommandlineoptionsprovider-sequence-diagram.png differ diff --git a/docs/core/testing/media/message-bus.png b/docs/core/testing/media/message-bus.png new file mode 100644 index 0000000000000..d26a890eff3b6 Binary files /dev/null and b/docs/core/testing/media/message-bus.png differ diff --git a/docs/core/testing/media/platform-testhostcontroller-testhost.png b/docs/core/testing/media/platform-testhostcontroller-testhost.png new file mode 100644 index 0000000000000..9a5f2302d91f5 Binary files /dev/null and b/docs/core/testing/media/platform-testhostcontroller-testhost.png differ diff --git a/docs/core/testing/media/platorm-testhost.png b/docs/core/testing/media/platorm-testhost.png new file mode 100644 index 0000000000000..3689178a84866 Binary files /dev/null and b/docs/core/testing/media/platorm-testhost.png differ diff --git a/docs/core/testing/media/test-framework-sequence-diagram.png b/docs/core/testing/media/test-framework-sequence-diagram.png new file mode 100644 index 0000000000000..41c5faa1411ed Binary files /dev/null and b/docs/core/testing/media/test-framework-sequence-diagram.png differ diff --git a/docs/core/testing/unit-testing-platform-architecture-capabilities.md b/docs/core/testing/unit-testing-platform-architecture-capabilities.md new file mode 100644 index 0000000000000..5fdc8e51a14ac --- /dev/null +++ b/docs/core/testing/unit-testing-platform-architecture-capabilities.md @@ -0,0 +1,126 @@ +--- +title: Microsoft.Testing.Platform capabilities overview +description: Learn about Microsoft.Testing.Platform capabilities concept. +author: MarcoRossignoli +ms.author: mrossignoli +ms.date: 07/11/2024 +--- + +# Microsoft.Testing.Platform capabilities + +In the context of the testing platform, a *capability* refers to the *potential to perform a specific action or provide specific information*. It's a means for the testing framework and extensions to *declare* their *ability* to *operate* in a certain manner or provide specific information to the *requesters*. + +The *requesters* can be any component involved in a test session, such as the platform, an extension, or the testing framework itself. + +The primary objective of the capability system is to facilitate effective communication among the components involved in a test session, enabling them to exchange information and meet their respective needs accurately. + +## Guided example + +Let's consider a hypothetical example to demonstrate the necessity of a capability system. + +> [!NOTE] +> This example is purely for illustrative purposes and isn't currently implemented within the testing platform or any testing framework. + +Imagine a situation where you have an extension that requires the testing framework to execute no more than one test at a time. Furthermore, after each test, the extension needs to know the CPU usage for that specific test. + +To accommodate the preceding scenario, you need to inquire from the testing framework if: + +1. It has the capability to execute only one test at a time. +2. It can provide information regarding the amount of CPU consumed by each test. + +How can the extension determine if the testing framework has the ability to operate in this mode and provide CPU usage information for a test session? In the testing platform, this capability is represented by an implementation the `Microsoft.Testing.Platform.Capabilities.ICapability` interface: + +```csharp +// Base capabilities contracts + +public interface ICapability +{ +} + +public interface ICapabilities + where TCapability : ICapability +{ + IReadOnlyCollection Capabilities { get; } +} + +// Specific testing framework capabilities + +public interface ITestFrameworkCapabilities : ICapabilities +{ +} + +public interface ITestFrameworkCapability : ICapability +{ +} +``` + +As you can see, the `ICapability` interface is a *marker* interface because it can represent *any capability*, and the actual implementation will be context dependent. You'll also observe the `ITestFrameworkCapability`, which inherits from `ICapability` to classify the capability. The capability system's generic nature allows for convenient grouping by context. The `ITestFrameworkCapability` groups all the capabilities implemented by the [testing framework](./unit-testing-platform-architecture-extensions.md#creating-a-testing-framework). The `ICapabilities` interface reveals the *set* of all capabilities implemented by an extension. Similarly, for the base one, there's a context-specific testing framework called `ITestFrameworkCapabilities`. The `ITestFrameworkCapabilities` is provided to the platform during the [testing framework registration](./unit-testing-platform-architecture-extensions.md#registering-a-testing-framework) process. + +To create a capability that addresses the aforementioned scenario, you define it as follows: + +```csharp +public interface IDisableParallelismCapability : ITestFrameworkCapability +{ + bool CanDisableParallelism { get; } + bool CanProvidePerTestCpuConsumption { get; } + void Enable(); +} +``` + +If the testing framework implements this interface, at runtime, the following can be queried: + +* Verify if the testing framework has the ability to turn off parallelism `CanDisableParallelism = true`. +* Determine if the testing framework can supply CPU usage data `CanProvidePerTestCPUConsumption = true`. +* Request the testing adapter to activate this mode by invoking the `Enable()` method before the test session commences. + +The hypothetical code fragment inside the extension could be something like: + +```csharp +IServiceProvider provider = null; // TODO: Get IServiceProvider. +var capabilities = serviceProvider.GetRequiredService(); + +// Utilize the `GetCapability` API to search for the specific capability to query. +var capability = capabilities.GetCapability(); +if (capability is null) +{ + // Capability not supported... +} +else +{ + capability.Enable(); + if (capability.CanDisableParallelism) + { + // Do something... + } + + if (capability.CanProvidePerTestCpuConsumption) + { + // Do something... + } +} +``` + +The preceding example illustrates how the capability infrastructure enables a powerful mechanism for communicating abilities between the components involved in a test session. While the sample demonstrates a capability specifically designed for the testing framework, any component can expose and implement extensions that inherit from `ICapability`. + +It's evident that not all details can be communicated through an interface. Considering the previous example, what should the extension expect if the `CanProvidePerTestCpuConsumption` is supported? What kind of custom information is expected to be transmitted via the [IMessageBus](./unit-testing-platform-architecture-services.md#the-imessagebus-service) by the testing framework? The solution is **documentation of the capability**. It's the responsibility of the capability *owner* to design, ship, and document it as clearly as possible to assist implementors who want to effectively *collaborate* with the extension that requires the specific capability. + +For instance, the TRX report extension enables the testing framework to implement the necessary capability to accurately generate a TRX report. The extension to register is included in the NuGet package, but the capability to implement is found in the *contract only* NuGet package. + +In conclusion, let's summarize the primary aspects of the capability system: + +* It's essential for facilitating clear and stable communication between components. +* All capabilities should inherit from `ICapability` or an interface that inherits from it, and are exposed through a collection with the `ICapabilities` interface. +* It aids in the evolution of features without causing breaking changes. If a certain capability isn't supported, appropriate action can be taken. +* The responsibility of designing, shipping, and documenting the usage of a capability lies with the *capability owner*. The testing platform can also *own* a capability in the same way as any other extension. + +## Framework capabilities + +The platform exposes a specialized interface named `ITestFrameworkCapability` that is the base of all capabilities exposed for test frameworks. These capabilities are provided when [registering the test framework to the platform](./unit-testing-platform-architecture-extensions.md#registering-a-testing-framework). + +### `IBannerMessageOwnerCapability` + +An optional [test framework capability](#framework-capabilities) that allows the test framework to provide the banner message to the platform. If the message is `null` or if the capability is n't present, the platform will use its default banner message. + +This capability implementation allows you to abstract away the various conditions that the test framework may need to consider when deciding whether or not the banner message should be displayed. + +The platform exposes the [`IPlatformInformation` service](./unit-testing-platform-architecture-services.md#the-iplatforminformation-service) to provide some information about the platform that could be useful when building your custom banner message. diff --git a/docs/core/testing/unit-testing-platform-architecture-extensions.md b/docs/core/testing/unit-testing-platform-architecture-extensions.md new file mode 100644 index 0000000000000..1a2c518a30fbc --- /dev/null +++ b/docs/core/testing/unit-testing-platform-architecture-extensions.md @@ -0,0 +1,1351 @@ +--- +title: Microsoft.Testing.Platform extensions architecture overview +description: Learn about how to extend Microsoft.Testing.Platform. +author: MarcoRossignoli +ms.author: mrossignoli +ms.date: 07/11/2024 +--- + +# Microsoft.Testing.Platform extensibility + +The testing platform consists of a [testing framework](#test-framework-extension) and any number of [extensions](#other-extensibility-points) that can operate *in-process* or *out-of-process*. + +As outlined in the [architecture](./unit-testing-platform-architecture.md) section, the testing platform is designed to accommodate a variety of scenarios and extensibility points. The primary and essential extension is undoubtedly the [testing framework](#test-framework-extension) that your tests will utilize. Failing to register this results in startup error. **The [testing framework](#test-framework-extension) is the sole mandatory extension required to execute a testing session.** + +To support scenarios such as generating test reports, code coverage, retrying failed tests, and other potential features, you need to provide a mechanism that allows other extensions to work in conjunction with the [testing framework](#test-framework-extension) to deliver these features not inherently provided by the [testing framework](#test-framework-extension) itself. + +In essence, the [testing framework](#test-framework-extension) is the primary extension that supplies information about each test that makes up the test suite. It reports whether a specific test has succeeded, failed, skipped, and can provide additional information about each test, such as a human-readable name (referred to as the display name), the source file, and the line where our test begins, among other things. + +The extensibility point enables the utilization of information provided by the [testing framework](#test-framework-extension) to generate new artifacts or to enhance existing ones with additional features. A commonly used extension is the TRX report generator, which subscribes to the [TestNodeUpdateMessage](#the-testnodeupdatemessage-data) and generates an XML report file from it. + +As discussed in the [architecture](./unit-testing-platform-architecture.md), there are certain extension points that *cannot* operate within the same process as the [testing framework](#test-framework-extension). The reasons typically include: + +* The need to modify the *environment variables* of the *test host*. Acting within the test host process itself is *too late*. +* The requirement to *monitor* the process from the outside because the *test host*, where tests and user code run, might have some *user code bugs* that render the process itself *unstable*, leading to potential *hangs* or *crashes*. In such cases, the extension would crash or hang along with the *test host* process. + +Due to these reasons, the extension points are categorized into two types: + +1. *In-process extensions*: These extensions operate within the same process as the [testing framework](#test-framework-extension). + + You can register *in-process extensions* via the `ITestApplicationBuilder.TestHost` property: + + ```csharp + // ... + var builder = await TestApplication.CreateBuilderAsync(args); + builder.TestHost.AddXXX(...); + // ... + ``` + +1. *Out-of-process extensions*: These extensions function in a separate process, allowing them to monitor the test host without being influenced by the test host itself. + + You can register *out-of-process extensions* via the `ITestApplicationBuilder.TestHostControllers`. + + ```csharp + var builder = await TestApplication.CreateBuilderAsync(args); + builder.TestHostControllers.AddXXX(...); + ``` + + Lastly, some extensions are designed to function in both scenarios. These common extensions behave identically in both *hosts*. You can register these extensions either through the *TestHost* and *TestHostController* interfaces or directly at the `ITestApplicationBuilder` level. An example of such an extension is the [ICommandLineOptionsProvider](#the-icommandlineoptionsprovider-extensions). + +## The `IExtension` interface + +The `IExtension` interface serves as the foundational interface for all extensibility points within the testing platform. It's primarily used to obtain descriptive information about the extension and, most importantly, to enable or disable the extension itself. + +Consider the following `IExension` interface: + +```csharp +public interface IExtension +{ + string Uid { get; } + string Version { get; } + string DisplayName { get; } + string Description { get; + Task IsEnabledAsync(); +} +``` + +* `Uid`: Represents the unique identifier for the extension. It's crucial to choose a unique value for this string to avoid conflicts with other extensions. + +* `Version`: Represents the version of the interface. Requires [**semantic versioning**](https://semver.org/). + +* `DisplayName`: A user-friendly name representation that will appear in logs and when you request information using the `--info` command line option. + +* `Description`: The description of the extension, that appears when you request information using the `--info` command line option. + +* `IsEnabledAsync()`: This method is invoked by the testing platform when the extension is being instantiated. If the method returns `false`, the extension will be excluded. This method typically makes decisions based on the [configuration file](./unit-testing-platform-architecture-services.md#the-iconfiguration-service) or some [custom command line options](./unit-testing-platform-architecture-services.md#the-icommandlineoptions-service). Users often specify `--customExtensionOption` in the command line to opt into the extension itself. + +## Test Framework extension + +The test framework is the primary extension that provides the testing platform with the ability to discover and execute tests. The test framework is responsible for communicating the results of the tests back to the testing platform. The test framework is the only mandatory extension required to execute a testing session. + +### Register a testing framework + +This section explains how to register the test framework with the testing platform. You register only one testing framework per test application builder using the `TestApplication.RegisterTestFramework` API as shown in [the testing platform architecture](./unit-testing-platform-architecture.md) documentation. + +The registration API is defined as follows: + +```csharp +ITestApplicationBuilder RegisterTestFramework( + Func capabilitiesFactory, + Func adapterFactory); +``` + +The `RegisterTestFramework` API expects two factories: + +1. `Func`: This is a delegate that accepts an object implementing the [`IServiceProvider`](./unit-testing-platform-architecture-services.md) interface and returns an object implementing the [`ITestFrameworkCapabilities`](./unit-testing-platform-architecture-capabilities.md) interface. The [`IServiceProvider`](./unit-testing-platform-architecture-services.md#the-imessagebus-service) provides access to platform services such as configurations, loggers, and command line arguments. + + The [`ITestFrameworkCapabilities`](./unit-testing-platform-architecture-capabilities.md) interface is used to announce the capabilities supported by the testing framework to the platform and extensions. It allows the platform and extensions to interact correctly by implementing and supporting specific behaviors. For a better understanding of the [concept of capabilities](./unit-testing-platform-architecture-capabilities.md), refer to the respective section. + +1. `Func`: This is a delegate that takes in an [ITestFrameworkCapabilities](./unit-testing-platform-architecture-capabilities.md) object, which is the instance returned by the `Func`, and an [IServiceProvider](./unit-testing-platform-architecture-services.md#the-imessagebus-service) to provide access to platform services once more. The expected return object is one that implements the [ITestFramework](#test-framework-extension) interface. The `ITestFramework` serves as the execution engine that discovers and runs tests, and then communicates the results back to the testing platform. + +The need for the platform to separate the creation of the [`ITestFrameworkCapabilities`](./unit-testing-platform-architecture-capabilities.md) and the creation of the [ITestFramework](#test-framework-extension) is an optimization to avoid creating the test framework if the supported capabilities are not sufficient to execute the current testing session. + +Consider the following user code example, which demonstrates a test framework registration that returns an empty capability set: + +```csharp +internal class TestingFrameworkCapabilities : ITestFrameworkCapabilities +{ + public IReadOnlyCollection Capabilities => []; +} + +internal class TestingFramework : ITestFramework +{ + public TestingFramework(ITestFrameworkCapabilities capabilities, IServiceProvider serviceProvider) + { + // ... + } + // Omitted for brevity... +} + +public static class TestingFrameworkExtensions +{ + public static void AddTestingFramework(this ITestApplicationBuilder builder) + { + builder.RegisterTestFramework( + _ => new TestingFrameworkCapabilities(), + (capabilities, serviceProvider) => new TestingFramework(capabilities, serviceProvider)); + } +} + +// ... +``` + +Now, consider the corresponding entry point of this example with the registration code: + +```csharp +var testApplicationBuilder = await TestApplication.CreateBuilderAsync(args); +// Register the testing framework +testApplicationBuilder.AddTestingFramework(); +using var testApplication = await testApplicationBuilder.BuildAsync(); +return await testApplication.RunAsync(); +``` + +> [!NOTE] +> Returning empty [ITestFrameworkCapabilities](./unit-testing-platform-architecture-capabilities.md) shouldn't prevent the execution of the test session. All test frameworks should be capable of discovering and running tests. The impact should be limited to extensions that may opt-out if the test framework lacks a certain feature. + +### Create a testing framework + +The `Microsoft.Testing.Platform.Extensions.TestFramework.ITestFramework` is implemented by extensions that provide a test framework: + +```csharp +public interface ITestFramework : IExtension +{ + Task CreateTestSessionAsync(CreateTestSessionContext context); + Task ExecuteRequestAsync(ExecuteRequestContext context); + Task CloseTestSessionAsync(CloseTestSessionContext context); +} +``` + +The `ITestFramework` interface inherits from the [IExtension](#the-iextension-interface) interface, which is an interface that all extension points inherit from. `IExtension` is used to retrieve the name and description of the extension. The `IExtension` also provides a way to dynamically enable or disable the extension in setup, through `Task IsEnabledAsync()`. Please make sure that you return `true` from this method if you have no special needs. + +#### The `CreateTestSessionAsync` method + +The `CreateTestSessionAsync` method is called at the start of the test session and is used to initialize the test framework. The API accepts a `CloseTestSessionContext` object and returns a `CloseTestSessionResult`. + +```csharp +public sealed class CreateTestSessionContext : TestSessionContext +{ + public SessionUid SessionUid { get; } + public ClientInfo Client { get; } + public CancellationToken CancellationToken { get; } +} + +public readonly struct SessionUid +{ + public string Value { get; } +} + +public sealed class ClientInfo +{ + public string Id { get; } + public string Version { get; } +} +``` + +The `SessionUid` serves as the unique identifier for the current test session, providing a logical connection to the session's results. +The `ClientInfo` provides details about the entity invoking the test framework. This information can be utilized by the test framework to modify its behavior. For example, as of the time this document was written, a console execution would report a client name such as "testingplatform-console". +The `CancellationToken` is used to halt the execution of `CreateTestSessionAsync`. + +The return object is a `CloseTestSessionResult`: + +```csharp +public sealed class CreateTestSessionResult +{ + public string? WarningMessage { get; set; } + public string? ErrorMessage { get; set; } + public bool IsSuccess { get; set; } +} +``` + +The `IsSuccess` property is used to indicate whether the session creation was successful. When it returns `false`, the test execution is halted. + +#### The `CloseTestSessionAsync` method + +The `CloseTestSessionAsync` method is juxtaposed to the `CreateTestSessionAsync` in functionality, with the only difference being the object names. For more information, see the `CreateTestSessionAsync` section. + +#### The `ExecuteRequestAsync` method + +The `ExecuteRequestAsync` method accepts an object of type `ExecuteRequestContext`. This object, as suggested by its name, holds the specifics about the action that the test framework is expected to perform. +The `ExecuteRequestContext` definition is: + +```csharp +public sealed class ExecuteRequestContext +{ + public IRequest Request { get; } + public IMessageBus MessageBus { get; } + public CancellationToken CancellationToken { get; } + public void Complete(); +} +``` + +`IRequest`: This is the base interface for any type of request. You should think about the test framework as an **in-process stateful server** where the lifecycle is: + + +:::image type="content" source="./media/test-framework-sequence-diagram.png" lightbox="./media/test-framework-sequence-diagram.png" alt-text="A sequence diagram representing the lifecycle of the test framework."::: + +The preceding diagram illustrates that the testing platform issues three requests after creating the test framework instance. The test framework processes these requests and utilizes the `IMessageBus` service, which is included in the request itself, to deliver the result for each specific request. Once a particular request has been handled, the test framework invokes the `Complete()` method on it, indicating to the testing platform that the request has been fulfilled. +The testing platform monitors all dispatched requests. Once all requests have been fulfilled, it invokes `CloseTestSessionAsync` and disposes of the instance (if `IDisposable/IAsyncDisposable` is implemented). +It's evident that the requests and their completions can overlap, enabling concurrent and asynchronous execution of requests. + +> [!NOTE] +> Currently, the testing platform does not send overlapping requests and waits for the completion of a request >> before sending the next one. However, this behavior may change in the future. The support for concurrent requests will be determined through the [capabilities](./unit-testing-platform-architecture-capabilities.md) system. + +The `IRequest` implementation specifies the precise request that needs to be fulfilled. The test framework identifies the type of request and handles it accordingly. If the request type is unrecognized, an exception should be raised. + +You can find details about the available requests in the [IRequest](#handling-requests) section. + +`IMessageBus`: This service, linked with the request, allows the test framework to *asynchronously* to publish information about the ongoing request to the testing platform. +The message bus serves as the central hub for the platform, facilitating asynchronous communication among all platform components and extensions. +For a comprehensive list of information that can be published to the testing platform, refer to the [IMessageBus](./unit-testing-platform-architecture-services.md#the-imessagebus-service) section. + +`CancellationToken`: This token is utilized to interrupt the processing of a particular request. + +`Complete()`: As depicted in the previous sequence, the `Complete` method notifies the platform that the request has been successfully processed and all relevant information has been transmitted to the [IMessageBus](./unit-testing-platform-architecture-services.md#the-imessagebus-service). + +> [!WARNING] +> Neglecting to invoke `Complete()` on the request will result in the test application becoming unresponsive. + +To customize your test framework according to your requirements or those of your users, you can use a personalized section inside the [configuration](./unit-testing-platform-architecture-services.md#the-iconfiguration-service) file or with custom [command line options](#the-icommandlineoptionsprovider-extensions). + +### Handling requests + +The subsequent section provides a detailed description of the various requests that a test framework may receive and process. + +Before proceeding to the next section, it's crucial to thoroughly comprehend the concept of the [IMessageBus](./unit-testing-platform-architecture-services.md#the-imessagebus-service), which is the essential service for conveying test execution information to the testing platform. + +#### TestSessionContext + +The `TestSessionContext` is a shared property across all requests, providing information about the ongoing test session: + +```csharp +public class TestSessionContext +{ + public SessionUid SessionUid { get; } + public ClientInfo Client { get; } +} + +public readonly struct SessionUid(string value) +{ + public string Value { get; } +} + +public sealed class ClientInfo +{ + public string Id { get; } + public string Version { get; } +} +``` + +The `TestSessionContext` consists of the `SessionUid`, a unique identifier for the ongoing test session that aids in logging and correlating test session data. It also includes the `ClientInfo` type, which provides details about the *initiator* of the test session. The test framework may choose different routes or publish varying information based on the identity of the test session's *initiator*. + +#### DiscoverTestExecutionRequest + +```csharp +public class DiscoverTestExecutionRequest +{ + // Detailed in the custom section below + public TestSessionContext Session { get; } + + // This is experimental and intended for future use, please disregard for now. + public ITestExecutionFilter Filter { get; } +} +``` + +The `DiscoverTestExecutionRequest` instructs the test framework **to discover** the tests and communicate this information thought to the [IMessageBus](./unit-testing-platform-architecture-services.md#the-imessagebus-service). + +As outlined in the previous section, the property for a discovered test is `DiscoveredTestNodeStateProperty`. Here is a generic code snippet for reference: + +```csharp +var testNode = new TestNode +{ + Uid = GenerateUniqueStableId(), + DisplayName = GetDisplayName(), + Properties = new PropertyBag( + DiscoveredTestNodeStateProperty.CachedInstance), +}; + +await context.MessageBus.PublishAsync( + this, + new TestNodeUpdateMessage( + discoverTestExecutionRequest.Session.SessionUid, + testNode)); + +// ... +``` + +#### RunTestExecutionRequest + +```csharp +public class RunTestExecutionRequest +{ + // Detailed in the custom section below + public TestSessionContext Session { get; } + + // This is experimental and intended for future use, please disregard for now. + public ITestExecutionFilter Filter { get; } +} +``` + +The `RunTestExecutionRequest` instructs the test framework **to execute** the tests and communicate this information thought to the [IMessageBus](./unit-testing-platform-architecture-services.md#the-imessagebus-service). + +Here is a generic code snippet for reference: + +```csharp +var skippedTestNode = new TestNode() +{ + Uid = GenerateUniqueStableId(), + DisplayName = GetDisplayName(), + Properties = new PropertyBag( + SkippedTestNodeStateProperty.CachedInstance), +}; + +await context.MessageBus.PublishAsync( + this, + new TestNodeUpdateMessage( + runTestExecutionRequest.Session.SessionUid, + skippedTestNode)); + +// ... + +var successfulTestNode = new TestNode() +{ + Uid = GenerateUniqueStableId(), + DisplayName = GetDisplayName(), + Properties = new PropertyBag( + PassedTestNodeStateProperty.CachedInstance), +}; + +await context.MessageBus.PublishAsync( + this, + new TestNodeUpdateMessage( + runTestExecutionRequest.Session.SessionUid, + successfulTestNode)); + +// ... + +var assertionFailedTestNode = new TestNode() +{ + Uid = GenerateUniqueStableId(), + DisplayName = GetDisplayName(), + Properties = new PropertyBag( + new FailedTestNodeStateProperty(assertionException)), +}; + +await context.MessageBus.PublishAsync( + this, + new TestNodeUpdateMessage( + runTestExecutionRequest.Session.SessionUid, + assertionFailedTestNode)); + +// ... + +var failedTestNode = new TestNode() +{ + Uid = GenerateUniqueStableId(), + DisplayName = GetDisplayName(), + Properties = new PropertyBag( + new ErrorTestNodeStateProperty(ex.InnerException!)), +}; + +await context.MessageBus.PublishAsync( + this, + new TestNodeUpdateMessage( + runTestExecutionRequest.Session.SessionUid, + failedTestNode)); +``` + +### The `TestNodeUpdateMessage` data + +As mentioned in the [IMessageBus](./unit-testing-platform-architecture-services.md#the-imessagebus-service) section, before utilizing the message bus, you must specify the type of data you intend to supply. The testing platform has defined a well-known type, `TestNodeUpdateMessage`, to represent the concept of a *test update information*. + +This part of the document will explain how to utilize this payload data. Let's examine the surface: + +```csharp +public sealed class TestNodeUpdateMessage( + SessionUid sessionUid, + TestNode testNode, + TestNodeUid? parentTestNodeUid = null) +{ + public TestNode TestNode { get; } + public TestNodeUid? ParentTestNodeUid { get; } +} + +public class TestNode +{ + public required TestNodeUid Uid { get; init; } + public required string DisplayName { get; init; } + public PropertyBag Properties { get; init; } = new(); +} + +public sealed class TestNodeUid(string value) + +public sealed partial class PropertyBag +{ + public PropertyBag(); + public PropertyBag(params IProperty[] properties); + public PropertyBag(IEnumerable properties); + public int Count { get; } + public void Add(IProperty property); + public bool Any(); + public TProperty? SingleOrDefault(); + public TProperty Single(); + public TProperty[] OfType(); + public IEnumerable AsEnumerable(); + public IEnumerator GetEnumerator(); + ... +} + +public interface IProperty +{ +} +``` + +* `TestNodeUpdateMessage`: The `TestNodeUpdateMessage` consists of two properties: a `TestNode` and a `ParentTestNodeUid`. The `ParentTestNodeUid` indicates that a test may have a parent test, introducing the concept of a **test tree** where `TestNode`s can be arranged in relation to each other. This structure allows for future enhancements and features based on the *tree* relationship between the nodes. If your test framework doesn't require a test tree structure, you can opt not to use it and simply set it to null, resulting in a straightforward flat list of `TestNode`s. + +* `TestNode`: The `TestNode` is composed of three properties, one of which is the `Uid` of type `TestNodeUid`. This `Uid` serves as the **UNIQUE STABLE ID** for the node. The term **UNIQUE STABLE ID** implies that the same `TestNode` should maintain an **IDENTICAL** `Uid` across different runs and operating systems. The `TestNodeUid` is an **arbitrary opaque string** that the testing platform accepts as is. + +> [!IMPORTANT] +> The stability and uniqueness of the ID are crucial in the testing domain. They enable the precise targeting of a single test for execution and allow the ID to serve as a persistent identifier for a test, facilitating powerful extensions and features. + +The second property is `DisplayName`, which is the human-friendly name for the test. For example, this name is displayed when you execute the `--list-tests` command line. + +The third attribute is `Properties`, which is a `PropertyBag` type. As demonstrated in the code, this is a specialized property bag that holds generic properties about the `TestNodeUpdateMessage`. This implies that you can append any property to the node that implements the placeholder interface `IProperty`. + +***The testing platform identifies specific properties added to a `TestNode.Properties` to determine whether a test has passed, failed, or been skipped.*** + +You can find the current list of available properties with the relative description in the section [TestNodeUpdateMessage.TestNode](#the-testnodeupdatemessage-data) + +The `PropertyBag` type is typically accessible in every `IData` and is utilized to store miscellaneous properties that can be queried by the platform and extensions. This mechanism allows us to enhance the platform with new information without introducing breaking changes. If a component recognizes the property, it can query it; otherwise, it will disregard it. + +Finally this section makes clear that you test framework implementation needs to implement the `IDataProducer` that produces `TestNodeUpdateMessage`s like in the sample below: + +```csharp +internal sealed class TestingFramework + : ITestFramework, IDataProducer +{ + // ... + + public Type[] DataTypesProduced => + [ + typeof(TestNodeUpdateMessage) + ]; + + // ... +} +``` + +If your test adapter requires the publication of *files* during execution, you can find the recognized properties in this source file: . As you can see, you can provide file assets in a general manner or associate them with a specific `TestNode`. Remember, if you intend to push a `SessionFileArtifact`, you must declare it to the platform in advance, as shown below: + +```csharp +internal sealed class TestingFramework + : ITestFramework, IDataProducer +{ + // ... + + public Type[] DataTypesProduced => + [ + typeof(TestNodeUpdateMessage), + typeof(SessionFileArtifact) + ]; + + // ... +} +``` + +#### Well-known properties + +As detailed in the [requests section](#handling-requests), the testing platform identifies specific properties added to the `TestNodeUpdateMessage` to determine the status of a `TestNode` (e.g., successful, failed, skipped, etc.). This allows the runtime to accurately display a list of failed tests with their corresponding information in the console, and to set the appropriate exit code for the test process. + +In this segment, we'll elucidate the various well-known `IProperty` options and their respective implications. + +If you're looking for a comprehensive list of well-known properties, you can find it [here](https://github.com/microsoft/testfx/blob/main/src/Platform/Microsoft.Testing.Platform/Messages/TestNodeProperties.cs). If you notice that a property description is missing, please don't hesitate to file an issue. + +These properties can be divided in the following categories: + +1. [*Generic information*](#generic-information): Properties that can be included in any kind of request. +1. [*Discovery information*](#discovery-information): Properties that are supplied during a `DiscoverTestExecutionRequest` discovery request. +1. [*Execution information*](#execution-information): Properties that are supplied during a test execution request `RunTestExecutionRequest`. + +Certain properties are **required**, while others are optional. The mandatory properties are required to provide basic testing functionality, such as reporting failed tests and indicating whether the entire test session was successful or not. + +Optional properties, on the other hand, enhance the testing experience by providing additional information. They are particularly useful in IDE scenarios (like VS, VSCode, etc.), console runs, or when supporting specific extensions that require more detailed information to function correctly. However, these optional properties do not affect the execution of the tests. + +> [!NOTE] +> Extensions are tasked with alerting and managing exceptions when they require specific information to operate correctly. If an extension lacks the necessary information, it should not cause the test execution to fail, but rather, it should simply opt-out. + +##### Generic information + +```csharp +public record KeyValuePairStringProperty( + string Key, + string Value) + : IProperty; +``` + +The `KeyValuePairStringProperty` stands for a general key/value pair data. + +```csharp +public record struct LinePosition( + int Line, + int Column); + +public record struct LinePositionSpan( + LinePosition Start, + LinePosition End); + +public abstract record FileLocationProperty( + string FilePath, + LinePositionSpan LineSpan) + : IProperty; + +public sealed record TestFileLocationProperty( + string FilePath, + LinePositionSpan LineSpan) + : FileLocationProperty(FilePath, LineSpan); +``` + +`TestFileLocationProperty` is used to pinpoint the location of the test within the source file. This is particularly useful when the initiator is an IDE like Visual Studio or Visual Studio Code. + +```csharp +public sealed record TestMethodIdentifierProperty( + string AssemblyFullName, + string Namespace, + string TypeName, + string MethodName, + string[] ParameterTypeFullNames, + string ReturnTypeFullName) +``` + +`TestMethodIdentifierProperty` is a unique identifier for a test method, adhering to the ECMA-335 standard. + +> [!NOTE] +> The data needed to create this property can be conveniently obtained using the .NET reflection feature, using types from the `System.Reflection` namespace. + +```csharp +public sealed record TestMetadataProperty( + string Key, + string Value) +``` + +`TestMetadataProperty` is utilized to convey the characteristics or *traits* of a `TestNode`. + +##### Discovery information + +```csharp +public sealed record DiscoveredTestNodeStateProperty( + string? Explanation = null) +{ + public static DiscoveredTestNodeStateProperty CachedInstance { get; } +} +``` + +The `DiscoveredTestNodeStateProperty` indicates that this TestNode has been discovered. It is utilized when a `DiscoverTestExecutionRequest` is sent to the test framework. +Take note of the handy cached value offered by the `CachedInstance` property. +This property is **required**. + +##### Execution information + +```csharp +public sealed record InProgressTestNodeStateProperty( + string? Explanation = null) +{ + public static InProgressTestNodeStateProperty CachedInstance { get; } +} +``` + +The `InProgressTestNodeStateProperty` informs the testing platform that the `TestNode` has been scheduled for execution and is currently in progress. +Take note of the handy cached value offered by the `CachedInstance` property. + +```csharp +public readonly record struct TimingInfo( + DateTimeOffset StartTime, + DateTimeOffset EndTime, + TimeSpan Duration); + +public sealed record StepTimingInfo( + string Id, + string Description, + TimingInfo Timing); + +public sealed record TimingProperty : IProperty +{ + public TimingProperty(TimingInfo globalTiming) + : this(globalTiming, []) + { + } + + public TimingProperty( + TimingInfo globalTiming, + StepTimingInfo[] stepTimings) + { + GlobalTiming = globalTiming; + StepTimings = stepTimings; + } + + public TimingInfo GlobalTiming { get; } + + public StepTimingInfo[] StepTimings { get; } +} +``` + +The `TimingProperty` is utilized to relay timing details about the `TestNode` execution. It also allows for the timing of individual execution steps via `StepTimingInfo`. This is particularly useful when your test concept is divided into multiple phases such as initialization, execution, and cleanup. + +***One and only one*** of the following properties is **required** per `TestNode` and communicates the result of the `TestNode` to the testing platform. + +```csharp +public sealed record PassedTestNodeStateProperty( + string? Explanation = null) + : TestNodeStateProperty(Explanation) +{ + public static PassedTestNodeStateProperty CachedInstance + { get; } = new PassedTestNodeStateProperty(); +} +``` + +`PassedTestNodeStateProperty` informs the testing platform that this `TestNode` is passed. +Take note of the handy cached value offered by the `CachedInstance` property. + +```csharp +public sealed record SkippedTestNodeStateProperty( + string? Explanation = null) + : TestNodeStateProperty(Explanation) +{ + public static SkippedTestNodeStateProperty CachedInstance + { get; } = new SkippedTestNodeStateProperty(); +} +``` + +`SkippedTestNodeStateProperty` informs the testing platform that this `TestNode` was skipped. +Take note of the handy cached value offered by the `CachedInstance` property. + +```csharp +public sealed record FailedTestNodeStateProperty : TestNodeStateProperty +{ + public FailedTestNodeStateProperty() + : base(default(string)) + { + } + + public FailedTestNodeStateProperty(string explanation) + : base(explanation) + { + } + + public FailedTestNodeStateProperty( + Exception exception, + string? explanation = null) + : base(explanation ?? exception.Message) + { + Exception = exception; + } + + public Exception? Exception { get; } +} +``` + +`FailedTestNodeStateProperty` informs the testing platform that this `TestNode` is failed after an assertion. + +```csharp +public sealed record ErrorTestNodeStateProperty : TestNodeStateProperty +{ + public ErrorTestNodeStateProperty() + : base(default(string)) + { + } + + public ErrorTestNodeStateProperty(string explanation) + : base(explanation) + { + } + + public ErrorTestNodeStateProperty( + Exception exception, + string? explanation = null) + : base(explanation ?? exception.Message) + { + Exception = exception; + } + + public Exception? Exception { get; } +} +``` + +`ErrorTestNodeStateProperty` informs the testing platform that this `TestNode` has failed. This type of failure is different from the `FailedTestNodeStateProperty`, which is used for assertion failures. For example, you can report issues like test initialization errors with `ErrorTestNodeStateProperty`. + +```csharp +public sealed record TimeoutTestNodeStateProperty : TestNodeStateProperty +{ + public TimeoutTestNodeStateProperty() + : base(default(string)) + { + } + + public TimeoutTestNodeStateProperty(string explanation) + : base(explanation) + { + } + + public TimeoutTestNodeStateProperty( + Exception exception, + string? explanation = null) + : base(explanation ?? exception.Message) + { + Exception = exception; + } + + public Exception? Exception { get; } + + public TimeSpan? Timeout { get; init; } +} +``` + +`TimeoutTestNodeStateProperty` informs the testing platform that this `TestNode` is failed for a timeout reason. You can report the timeout using the `Timeout` property. + +```csharp +public sealed record CancelledTestNodeStateProperty : TestNodeStateProperty +{ + public CancelledTestNodeStateProperty() + : base(default(string)) + { + } + + public CancelledTestNodeStateProperty(string explanation) + : base(explanation) + { + } + + public CancelledTestNodeStateProperty( + Exception exception, + string? explanation = null) + : base(explanation ?? exception.Message) + { + Exception = exception; + } + + public Exception? Exception { get; } +} +``` + +`CancelledTestNodeStateProperty` informs the testing platform that this `TestNode` has failed due to cancellation. + +## Other extensibility points + +The testing platform provides additional extensibility points that allow you to customize the behavior of the platform and the test framework. These extensibility points are optional and can be used to enhance the testing experience. + +### The `ICommandLineOptionsProvider` extensions + +> [!NOTE] +> When extending this API, the custom extension will exists both in and out of the test host process. + +As discussed in the [architecture](./unit-testing-platform-architecture.md) section, the initial step involves creating the `ITestApplicationBuilder` to register the testing framework and extensions with it. + +```csharp +var builder = await TestApplication.CreateBuilderAsync(args); +``` + +The `CreateBuilderAsync` method accepts an array of strings (`string[]`) named `args`. These arguments can be used to pass command-line options to all components of the testing platform (including built-in components, testing frameworks, and extensions), allowing for customization of their behavior. + +Typically, the arguments passed are those received in the standard `Main(string[] args)` method. However, if the hosting environment differs, any list of arguments can be supplied. + +Arguments **must be prefixed** with a double dash `--`. For example, `--filter`. + +If a component such as a testing framework or an extension point wishes to offer custom command-line options, it can do so by implementing the `ICommandLineOptionsProvider` interface. This implementation can then be registered with the `ITestApplicationBuilder` via the registration factory of the `CommandLine` property, as shown: + +```csharp +builder.CommandLine.AddProvider( + static () => new CustomCommandLineOptions()); +``` + +In the example provided, `CustomCommandLineOptions` is an implementation of the `ICommandLineOptionsProvider` interface, This interface comprises the following members and data types: + +```csharp +public interface ICommandLineOptionsProvider : IExtension +{ + IReadOnlyCollection GetCommandLineOptions(); + + Task ValidateOptionArgumentsAsync( + CommandLineOption commandOption, + string[] arguments); + + Task ValidateCommandLineOptionsAsync( + ICommandLineOptions commandLineOptions); +} + +public sealed class CommandLineOption +{ + public string Name { get; } + public string Description { get; } + public ArgumentArity Arity { get; } + public bool IsHidden { get; } + + // ... +} + +public interface ICommandLineOptions +{ + bool IsOptionSet(string optionName); + + bool TryGetOptionArgumentList( + string optionName, + out string[]? arguments); +} +``` + +As observed, the `ICommandLineOptionsProvider` extends the [`IExtension`](#the-iextension-interface) interface. Therefore, like any other extension, you can choose to enable or disable it using the `IExtension.IsEnabledAsync` API. + +The order of execution of the `ICommandLineOptionsProvider` is: + + +:::image type="content" source="./media/icommandlineoptionsprovider-sequence-diagram.png" lightbox="./media/icommandlineoptionsprovider-sequence-diagram.png" alt-text="A diagram representing the order of execution of the 'ICommandLineOptionsProvider' interface."::: + +Let's examine the apis and their mean: + +`ICommandLineOptionsProvider.GetCommandLineOptions()`: This method is utilized to retrieve all the options offered by the component. Each `CommandLineOption` requires the following properties to be specified: + +`string name`: This is the option's name, presented without a dash. For example, *filter* would be used as `--filter` by users. + +`string description`: This is a description of the option. It will be displayed when users pass `--help` as an argument to the application builder. + +`ArgumentArity arity`: The arity of an option is the number of values that can be passed if that option or command is specified. Current available arities are: + +* `Zero`: Represents an argument arity of zero. +* `ZeroOrOne`: Represents an argument arity of zero or one. +* `ZeroOrMore`: Represents an argument arity of zero or more. +* `OneOrMore`: Represents an argument arity of one or more. +* `ExactlyOne`: Represents an argument arity of exactly one. + +For examples, refer to the [System.CommandLine arity table](https://learn.microsoft/dotnet/standard/commandline/syntax#argument-arity). + +`bool isHidden`: This property signifies that the option is available for use but will not be displayed in the description when `--help` is invoked. + +`ICommandLineOptionsProvider.ValidateOptionArgumentsAsync`: This method is employed to *validate* the argument provided by the user. + +For instance, if you have a parameter named `--dop` that represents the degree of parallelism for our custom testing framework, a user might input `--dop 0`. In this scenario, the value `0` would be invalid because it is expected to have a degree of parallelism of `1` or more. By using `ValidateOptionArgumentsAsync`, you can perform upfront validation and return an error message if necessary. + +A possible implementation for the sample above could be: + +```csharp +public Task ValidateOptionArgumentsAsync( + CommandLineOption commandOption, + string[] arguments) +{ + if (commandOption.Name == "dop") + { + if (!int.TryParse(arguments[0], out int dopValue) || dopValue <= 0) + { + return ValidationResult.InvalidTask("--dop must be a positive integer"); + } + } + + return ValidationResult.ValidTask; +} +``` + +`ICommandLineOptionsProvider.ValidateCommandLineOptionsAsync`: This method is called as last one and allows to do global coherency check. + +For example, let's say our testing framework has the capability to generate a test result report and save it to a file. This feature is accessed using the `--generatereport` option, and the filename is specified with `--reportfilename myfile.rep`. In this scenario, if a user only provides the `--generatereport` option without specifying a filename, the validation should fail because the report cannot be generated without a filename. +A possible implementation for the sample above could be: + +```csharp +public Task ValidateCommandLineOptionsAsync(ICommandLineOptions commandLineOptions) +{ + bool generateReportEnabled = commandLineOptions.IsOptionSet(GenerateReportOption); + bool reportFileName = commandLineOptions.TryGetOptionArgumentList(ReportFilenameOption, out string[]? _); + + return (generateReportEnabled || reportFileName) && !(generateReportEnabled && reportFileName) + ? ValidationResult.InvalidTask("Both `--generatereport` and `--reportfilename` need to be provided simultaneously.") + : ValidationResult.ValidTask; +} +``` + +Please note that the `ValidateCommandLineOptionsAsync` method provides the [`ICommandLineOptions`](./unit-testing-platform-architecture-services.md#the-icommandlineoptions-service) service, which is used to fetch the argument information parsed by the platform itself. + +### The `ITestSessionLifetimeHandler` extensions + +The `ITestSessionLifeTimeHandler` is an *in-process* extension that enables the execution of code *before* and *after* the test session. + +To register a custom `ITestSessionLifeTimeHandler`, utilize the following API: + +```csharp +var builder = await TestApplication.CreateBuilderAsync(args); + +// ... + +builder.TestHost.AddTestSessionLifetimeHandle( + static serviceProvider => new CustomTestSessionLifeTimeHandler()); +``` + +The factory utilizes the [IServiceProvider](./unit-testing-platform-architecture-services.md#the-imessagebus-service) to gain access to the suite of services offered by the testing platform. + +> [!IMPORTANT] +> The sequence of registration is significant, as the APIs are called in the order they were registered. + +The `ITestSessionLifeTimeHandler` interface includes the following methods: + +```csharp +public interface ITestSessionLifetimeHandler : ITestHostExtension +{ + Task OnTestSessionStartingAsync( + SessionUid sessionUid, + CancellationToken cancellationToken); + + Task OnTestSessionFinishingAsync( + SessionUid sessionUid, + CancellationToken cancellationToken); +} + +public readonly struct SessionUid(string value) +{ + public string Value { get; } = value; +} + +public interface ITestHostExtension : IExtension +{ +} +``` + +The `ITestSessionLifetimeHandler` is a type of `ITestHostExtension`, which serves as a base for all *test host* extensions. Like all other extension points, it also inherits from [IExtension](#the-iextension-interface). Therefore, like any other extension, you can choose to enable or disable it using the `IExtension.IsEnabledAsync` API. + +Consider the following details for this API: + +`OnTestSessionStartingAsync`: This method is invoked prior to the commencement of the test session and receives the `SessionUid` object, which provides an opaque identifier for the current test session. + +`OnTestSessionFinishingAsync`: This method is invoked after the completion of the test session, ensuring that the [testing framework](#test-framework-extension) has finished executing all tests and has reported all relevant data to the platform. Typically, in this method, the extension employs the [`IMessageBus`](./unit-testing-platform-architecture-services.md#the-imessagebus-service) to transmit custom assets or data to the shared platform bus. This method can also signal to any custom *out-of-process* extension that the test session has concluded. + +Finally, both APIs take a `CancellationToken` which the extension is expected to honor. + +If your extension requires intensive initialization and you need to use the async/await pattern, you can refer to the [`Async extension initialization and cleanup`](#asynchronous-initialization-and-cleanup-of-extensions). If you need to *share state* between extension points, you can refer to the [`CompositeExtensionFactory`](#the-compositeextensionfactoryt) section. + +### The `ITestApplicationLifecycleCallbacks` extensions + +The `ITestApplicationLifecycleCallbacks` is an *in-process* extension that enables the execution of code before everything, it's like to have access to the first line of the hypothetical *main* of the *test host*. + +To register a custom `ITestApplicationLifecycleCallbacks`, utilize the following api: + +```csharp +var builder = await TestApplication.CreateBuilderAsync(args); + +// ... + +builder.TestHost.AddTestApplicationLifecycleCallbacks( + static serviceProvider + => new CustomTestApplicationLifecycleCallbacks()); +``` + +The factory utilizes the [IServiceProvider](./unit-testing-platform-architecture-services.md#the-imessagebus-service) to gain access to the suite of services offered by the testing platform. + +> [!IMPORTANT] +> The sequence of registration is significant, as the APIs are called in the order they were registered. + +The `ITestApplicationLifecycleCallbacks` interface includes the following methods: + +```csharp +public interface ITestApplicationLifecycleCallbacks : ITestHostExtension +{ + Task BeforeRunAsync(CancellationToken cancellationToken); + + Task AfterRunAsync( + int exitCode, + CancellationToken cancellation); +} + +public interface ITestHostExtension : IExtension +{ +} +``` + +The `ITestApplicationLifecycleCallbacks` is a type of `ITestHostExtension`, which serves as a base for all *test host* extensions. Like all other extension points, it also inherits from [IExtension](#the-iextension-interface). Therefore, like any other extension, you can choose to enable or disable it using the `IExtension.IsEnabledAsync` API. + +`BeforeRunAsync`: This method serves as the initial point of contact for the *test host* and is the first opportunity for an *in-process* extension to execute a feature. It's typically used to establish a connection with any corresponding *out-of-process* extensions if a feature is designed to operate across both environments. + +*For example, the built-in hang dump feature is composed of both *in-process* and *out-of-process* extensions, and this method is used to exchange information with the *out-of-process* component of the extension.* + +`AfterRunAsync`: This method is the final call before exiting the [`int ITestApplication.RunAsync()`](./unit-testing-platform-architecture.md) and it provides the [`exit code`](./unit-testing-platform-exit-codes.md). It should be used solely for cleanup tasks and to notify any corresponding *out-of-process* extension that the *test host* is about to terminate. + +Finally, both APIs take a `CancellationToken` which the extension is expected to honor. + +### The `IDataConsumer` extensions + +The `IDataConsumer` is an *in-process* extension capable of subscribing to and receiving `IData` information that is pushed to the [IMessageBus](./unit-testing-platform-architecture-services.md#the-imessagebus-service) by the [testing framework](#test-framework-extension) and its extensions. + +*This extension point is crucial as it enables developers to gather and process all the information generated during a test session.* + +To register a custom `IDataConsumer`, utilize the following api: + +```csharp +var builder = await TestApplication.CreateBuilderAsync(args); + +// ... + +builder.TestHost.AddDataConsumer( + static serviceProvider => new CustomDataConsumer()); +``` + +The factory utilizes the [IServiceProvider](./unit-testing-platform-architecture-services.md#the-imessagebus-service) to gain access to the suite of services offered by the testing platform. + +> [!IMPORTANT] +> The sequence of registration is significant, as the APIs are called in the order they were registered. + +The `IDataConsumer` interface includes the following methods: + +```csharp +public interface IDataConsumer : ITestHostExtension +{ + Type[] DataTypesConsumed { get; } + + Task ConsumeAsync( + IDataProducer dataProducer, + IData value, + CancellationToken cancellationToken); +} + +public interface IData +{ + string DisplayName { get; } + string? Description { get; } +} +``` + +The `IDataConsumer` is a type of `ITestHostExtension`, which serves as a base for all *test host* extensions. Like all other extension points, it also inherits from [IExtension](#the-iextension-interface). Therefore, like any other extension, you can choose to enable or disable it using the `IExtension.IsEnabledAsync` API. + +`DataTypesConsumed`: This property returns a list of `Type` that this extension plans to consume. It corresponds to `IDataProducer.DataTypesProduced`. Notably, an `IDataConsumer` can subscribe to multiple types originating from different `IDataProducer` instances without any issues. + +`ConsumeAsync`: This method is triggered whenever data of a type to which the current consumer is subscribed is pushed onto the [`IMessageBus`](./unit-testing-platform-architecture-services.md#the-imessagebus-service). It receives the `IDataProducer` to provide details about the data payload's producer, as well as the `IData` payload itself. As you can see, `IData` is a generic placeholder interface that contains general informative data. The ability to push different types of `IData` implies that the consumer needs to *switch* on the type itself to cast it to the correct type and access the specific information. + +A sample implementation of a consumer that wants to elaborate the [`TestNodeUpdateMessage`](#the-testnodeupdatemessage-data) produced by a [testing framework](#test-framework-extension) could be: + +```csharp +internal class CustomDataConsumer : IDataConsumer, IOutputDeviceDataProducer +{ + public Type[] DataTypesConsumed => new[] { typeof(TestNodeUpdateMessage) }; + ... + public Task ConsumeAsync( + IDataProducer dataProducer, + IData value, + CancellationToken cancellationToken) + { + var testNodeUpdateMessage = (TestNodeUpdateMessage)value; + + switch (testNodeUpdateMessage.TestNode.Properties.Single()) + { + case InProgressTestNodeStateProperty _: + { + ... + break; + } + case PassedTestNodeStateProperty _: + { + ... + break; + } + case FailedTestNodeStateProperty failedTestNodeStateProperty: + { + ... + break; + } + case SkippedTestNodeStateProperty _: + { + ... + break; + } + ... + } + + return Task.CompletedTask; + } +... +} +``` + +Finally, the API takes a `CancellationToken` which the extension is expected to honor. + +> [!IMPORTANT] +> It's crucial to process the payload directly within the `ConsumeAsync` method. The [IMessageBus](./unit-testing-platform-architecture-services.md#the-imessagebus-service) can manage both synchronous and asynchronous processing, coordinating the execution with the [testing framework](#test-framework-extension). Although the consumption process is entirely asynchronous and doesn't block the [IMessageBus.Push](./unit-testing-platform-architecture-services.md#the-imessagebus-service) at the time of writing, this is an implementation detail that may change in the future due to future requirements. However, the platform ensures that this method is always called once, eliminating the need for complex synchronization, as well as managing the scalability of the consumers. + + + +> [!WARNING] +> When using `IDataConsumer` in conjunction with [ITestHostProcessLifetimeHandler](#the-itestsessionlifetimehandler-extensions) within a [composite extension point](#the-compositeextensionfactoryt), **it's crucial to disregard any data received post the execution of [ITestSessionLifetimeHandler.OnTestSessionFinishingAsync](#the-itestsessionlifetimehandler-extensions)**. The `OnTestSessionFinishingAsync` is the final opportunity to process accumulated data and transmit new information to the [IMessageBus](./unit-testing-platform-architecture-services.md#the-imessagebus-service), hence, any data consumed beyond this point will not be *utilizable* by the extension. + +If your extension requires intensive initialization and you need to use the async/await pattern, you can refer to the [`Async extension initialization and cleanup`](#asynchronous-initialization-and-cleanup-of-extensions). If you need to *share state* between extension points, you can refer to the [`CompositeExtensionFactory`](#the-compositeextensionfactoryt) section. + +### The `ITestHostEnvironmentVariableProvider` extensions + +The `ITestHostEnvironmentVariableProvider` is an *out-of-process* extension that enables you to establish custom environment variables for the test host. Utilizing this extension point ensures that the testing platform will initiate a new host with the appropriate environment variables, as detailed in the [architecture](./unit-testing-platform-architecture.md) section. + +To register a custom `ITestHostEnvironmentVariableProvider`, utilize the following API: + +```csharp +var builder = await TestApplication.CreateBuilderAsync(args); + +// ... + +builder.TestHostControllers.AddEnvironmentVariableProvider( + static serviceProvider => new CustomEnvironmentVariableForTestHost()); +``` + +The factory utilizes the [IServiceProvider](./unit-testing-platform-architecture-services.md#the-imessagebus-service) to gain access to the suite of services offered by the testing platform. + +> [!IMPORTANT] +> The sequence of registration is significant, as the APIs are called in the order they were registered. + +The `ITestHostEnvironmentVariableProvider` interface includes the following methods and types: + +```csharp +public interface ITestHostEnvironmentVariableProvider : ITestHostControllersExtension, IExtension +{ + Task UpdateAsync(IEnvironmentVariables environmentVariables); + + Task ValidateTestHostEnvironmentVariablesAsync( + IReadOnlyEnvironmentVariables environmentVariables); +} + +public interface IEnvironmentVariables : IReadOnlyEnvironmentVariables +{ + void SetVariable(EnvironmentVariable environmentVariable); + void RemoveVariable(string variable); +} + +public interface IReadOnlyEnvironmentVariables +{ + bool TryGetVariable( + string variable, + [NotNullWhen(true)] out OwnedEnvironmentVariable? environmentVariable); +} + +public sealed class OwnedEnvironmentVariable : EnvironmentVariable +{ + public IExtension Owner { get; } + + public OwnedEnvironmentVariable( + IExtension owner, + string variable, + string? value, + bool isSecret, + bool isLocked); +} + +public class EnvironmentVariable +{ + public string Variable { get; } + public string? Value { get; } + public bool IsSecret { get; } + public bool IsLocked { get; } +} +``` + +The `ITestHostEnvironmentVariableProvider` is a type of `ITestHostControllersExtension`, which serves as a base for all *test host controller* extensions. Like all other extension points, it also inherits from [IExtension](#the-iextension-interface). Therefore, like any other extension, you can choose to enable or disable it using the `IExtension.IsEnabledAsync` API. + +Consider the details for this API: + +`UpdateAsync`: This update API provides an instance of the `IEnvironmentVariables` object, from which you can call the `SetVariable` or `RemoveVariable` methods. When using `SetVariable`, you must pass an object of type `EnvironmentVariable`, which requires the following specifications: + +* `Variable`: The name of the environment variable. +* `Value`: The value of the environment variable. +* `IsSecret`: This indicates whether the environment variable contains sensitive information that should not be logged or accessible via the `TryGetVariable`. +* `IsLocked`: This determines whether other `ITestHostEnvironmentVariableProvider` extensions can modify this value. + +`ValidateTestHostEnvironmentVariablesAsync`: This method is invoked after all the `UpdateAsync` methods of the registered `ITestHostEnvironmentVariableProvider` instances have been called. It allows you to *verify* the correct setup of the environment variables. It takes an object that implements `IReadOnlyEnvironmentVariables`, which provides the `TryGetVariable` method to fetch specific environment variable information with the `OwnedEnvironmentVariable` object type. After validation, you return a `ValidationResult` containing any failure reasons. + +> [!NOTE] +> The testing platform, by default, implements and registers the `SystemEnvironmentVariableProvider`. This provider loads all the *current* environment variables. As the first registered provider, it executes first, granting access to the default environment variables for all other `ITestHostEnvironmentVariableProvider` user extensions. + +If your extension requires intensive initialization and you need to use the async/await pattern, you can refer to the [`Async extension initialization and cleanup`](#asynchronous-initialization-and-cleanup-of-extensions). If you need to *share state* between extension points, you can refer to the [`CompositeExtensionFactory`](#the-compositeextensionfactoryt) section. + +### The `ITestHostProcessLifetimeHandler` extensions + +The `ITestHostProcessLifetimeHandler` is an *out-of-process* extension that allows you to observe the test host process from an external standpoint. This ensures that your extension remains unaffected by potential crashes or hangs that could be induced by the code under test. Utilizing this extension point will prompt the testing platform to initiate a new host, as detailed in the [architecture](./unit-testing-platform-architecture.md) section. + +To register a custom `ITestHostProcessLifetimeHandler`, utilize the following API: + +```csharp +var builder = await TestApplication.CreateBuilderAsync(args); + +// ... + +builder.TestHostControllers.AddProcessLifetimeHandler( + static serviceProvider => new CustomMonitorTestHost()); +``` + +The factory utilizes the [IServiceProvider](./unit-testing-platform-architecture-services.md#the-imessagebus-service) to gain access to the suite of services offered by the testing platform. + +> [!IMPORTANT] +> The sequence of registration is significant, as the APIs are called in the order they were registered. + +The `ITestHostProcessLifetimeHandler` interface includes the following methods: + +```csharp +public interface ITestHostProcessLifetimeHandler : ITestHostControllersExtension +{ + Task BeforeTestHostProcessStartAsync(CancellationToken cancellationToken); + + Task OnTestHostProcessStartedAsync( + ITestHostProcessInformation testHostProcessInformation, + CancellationToken cancellation); + + Task OnTestHostProcessExitedAsync( + ITestHostProcessInformation testHostProcessInformation, + CancellationToken cancellation); +} + +public interface ITestHostProcessInformation +{ + int PID { get; } + int ExitCode { get; } + bool HasExitedGracefully { get; } +} +``` + +The `ITestHostProcessLifetimeHandler` is a type of `ITestHostControllersExtension`, which serves as a base for all *test host controller* extensions. Like all other extension points, it also inherits from [IExtension](#the-iextension-interface). Therefore, like any other extension, you can choose to enable or disable it using the `IExtension.IsEnabledAsync` API. + +Consider the following details for this API: + +`BeforeTestHostProcessStartAsync`: This method is invoked prior to the testing platform initiating the test hosts. + +`OnTestHostProcessStartedAsync`: This method is invoked immediately after the test host starts. This method offers an object that implements the `ITestHostProcessInformation` interface, which provides key details about the test host process result. +> [!IMPORTANT] +> The invocation of this method does not halt the test host's execution. If you need to pause it, you should register an [*in-process*](#microsofttestingplatform-extensibility) extension such as [`ITestApplicationLifecycleCallbacks`](#the-itestapplicationlifecyclecallbacks-extensions) and synchronize it with the *out-of-process* extension. + +`OnTestHostProcessExitedAsync`: This method is invoked when the test suite execution is complete. This method supplies an object that adheres to the `ITestHostProcessInformation` interface, which conveys crucial details about the outcome of the test host process. + +The `ITestHostProcessInformation` interface provides the following details: + +* `PID`: The process ID of the test host. +* `ExitCode`: The exit code of the process. This value is only available within the `OnTestHostProcessExitedAsync` method. Attempting to access it within the `OnTestHostProcessStartedAsync` method will result in an exception. +* `HasExitedGracefully`: A boolean value indicating whether the test host has crashed. If true, it signifies that the test host did not exit gracefully. + +## Extensions execution order + +The testing platform consists of a [testing framework](#test-framework-extension) and any number of extensions that can operate [*in-process*](#microsofttestingplatform-extensibility) or [*out-of-process*](#microsofttestingplatform-extensibility). This document outlines the **sequence of calls** to all potential extensibility points to provide clarity on when a feature is anticipated to be invoked: + +1. [ITestHostEnvironmentVariableProvider.UpdateAsync](#the-itesthostenvironmentvariableprovider-extensions) : Out-of-process +1. [ITestHostEnvironmentVariableProvider.ValidateTestHostEnvironmentVariablesAsync](#the-itesthostenvironmentvariableprovider-extensions) : Out-of-process +1. [ITestHostProcessLifetimeHandler.BeforeTestHostProcessStartAsync](#the-itestsessionlifetimehandler-extensions) : Out-of-process +1. Test host process start +1. [ITestHostProcessLifetimeHandler.OnTestHostProcessStartedAsync](#the-itestsessionlifetimehandler-extensions) : Out-of-process, this event can intertwine the actions of *in-process* extensions, depending on race conditions. +1. [ITestApplicationLifecycleCallbacks.BeforeRunAsync](#the-itestsessionlifetimehandler-extensions): In-process +1. [ITestSessionLifetimeHandler.OnTestSessionStartingAsync](#the-itestsessionlifetimehandler-extensions): In-process +1. [ITestFramework.CreateTestSessionAsync](#test-framework-extension): In-process +1. [ITestFramework.ExecuteRequestAsync](#test-framework-extension): In-process, this method can be called one or more times. At this point, the testing framework will transmit information to the [IMessageBus](./unit-testing-platform-architecture-services.md#the-imessagebus-service) that can be utilized by the [IDataConsumer](#the-idataconsumer-extensions). +1. [ITestFramework.CloseTestSessionAsync](#test-framework-extension): In-process +1. [ITestSessionLifetimeHandler.OnTestSessionFinishingAsync](#the-itestsessionlifetimehandler-extensions): In-process +1. [ITestApplicationLifecycleCallbacks.AfterRunAsync](#the-itestsessionlifetimehandler-extensions): In-process +1. In-process cleanup, involves calling dispose and [IAsyncCleanableExtension](#asynchronous-initialization-and-cleanup-of-extensions) on all extension points. +1. [ITestHostProcessLifetimeHandler.OnTestHostProcessExitedAsync](#the-itestsessionlifetimehandler-extensions) : Out-of-process +1. Out-of-process cleanup, involves calling dispose and [IAsyncCleanableExtension](#asynchronous-initialization-and-cleanup-of-extensions) on all extension points. + +## Extensions helpers + +The testing platform provides a set of helper classes and interfaces to simplify the implementation of extensions. These helpers are designed to streamline the development process and ensure that the extension adheres to the platform's standards. + +### Asynchronous initialization and cleanup of extensions + +The creation of the testing framework and extensions through factories adheres to the standard .NET object creation mechanism, which uses synchronous constructors. If an extension requires intensive initialization (such as accessing the file system or network), it cannot employ the *async/await* pattern in the constructor because constructors return void, not `Task`. + +Therefore, the testing platform provides a method to initialize an extension using the async/await pattern through a simple interface. For symmetry, it also offers an async interface for cleanup that extensions can implement seamlessly. + +```csharp +public interface IAsyncInitializableExtension +{ + Task InitializeAsync(); +} + +public interface IAsyncCleanableExtension +{ + Task CleanupAsync(); +} +``` + +`IAsyncInitializableExtension.InitializeAsync`: This method is assured to be invoked following the creation factory. + +`IAsyncCleanableExtension.CleanupAsync`: This method is assured to be invoked *at least one time* during the termination of the testing session, prior to the default `DisposeAsync` or `Dispose`. + +> [!IMPORTANT] +> Similar to the standard `Dispose` method, `CleanupAsync` may be invoked multiple times. If an object's `CleanupAsync` method is called more than once, the object must ignore all calls after the first one. The object must not throw an exception if its `CleanupAsync` method is called multiple times. +> [!NOTE] +> By default, the testing platform will call `DisposeAsync` if it's available, or `Dispose` if it's implemented. It's important to note that the testing platform will not call both dispose methods but will prioritize the async one if implemented. + +### The CompositeExtensionFactory + +As outlined in the [extensions](#microsofttestingplatform-extensibility) section, the testing platform enables you to implement interfaces to incorporate custom extensions both in and out of process. + +Each interface addresses a particular feature, and according to .NET design, you implement this interface in a specific object. You can register the extension itself using the specific registration API `AddXXX` from the `TestHost` or `TestHostController` object from the `ITestApplicationBuilder` as detailed in the corresponding sections. + +However, if you need to *share state* between two extensions, the fact that you can implement and register different objects implementing different interfaces makes sharing a challenging task. Without any assistance, you would need a way to pass one extension to the other to share information, which complicates the design. + +Hence, the testing platform provides a sophisticated method to implement multiple extension points using the same type, making data sharing a straightforward task. All you need to do is utilize the `CompositeExtensionFactory`, which can then be registered using the same API as you would for a single interface implementation. + +For instance, consider a type that implements both `ITestSessionLifetimeHandler` and `IDataConsumer`. This is a common scenario because you often want to gather information from the [testing framework](#test-framework-extension) and then, when the testing session concludes, you'll dispatch your artifact using the [`IMessageBus`](./unit-testing-platform-architecture-services.md#the-imessagebus-service) within the `ITestSessionLifetimeHandler.OnTestSessionFinishingAsync`. + +What you should do is to normally implement the interfaces: + +```csharp +internal class CustomExtension : ITestSessionLifetimeHandler, IDataConsumer, ... +{ + ... +} +``` + +Once you've created the `CompositeExtensionFactory` for your type, you can register it with both the `IDataConsumer` and `ITestSessionLifetimeHandler` APIs, which offer an overload for the `CompositeExtensionFactory`: + +```csharp +var builder = await TestApplication.CreateBuilderAsync(args); + +// ... + +var factory = new CompositeExtensionFactory(serviceProvider => new CustomExtension()); + +builder.TestHost.AddTestSessionLifetimeHandle(factory); +builder.TestHost.AddDataConsumer(factory); +``` + +The factory constructor employs the [IServiceProvider](./unit-testing-platform-architecture-services.md) to access the services provided by the testing platform. + +The testing platform will be responsible for managing the lifecycle of the composite extension. + +It's important to note that due to the testing platform's support for both *in-process* and *out-of-process* extensions, you can't combine any extension point arbitrarily. The creation and utilization of extensions are contingent on the host type, meaning you can only group *in-process* (TestHost) and *out-of-process* (TestHostController) extensions together. + +The following combinations are possible: + +* For `ITestApplicationBuilder.TestHost`, you can combine `IDataConsumer` and `ITestSessionLifetimeHandler`. +* For `ITestApplicationBuilder.TestHostControllers`, you can combine `ITestHostEnvironmentVariableProvider` and `ITestHostProcessLifetimeHandler`. diff --git a/docs/core/testing/unit-testing-platform-architecture-services.md b/docs/core/testing/unit-testing-platform-architecture-services.md new file mode 100644 index 0000000000000..7790e6a95e974 --- /dev/null +++ b/docs/core/testing/unit-testing-platform-architecture-services.md @@ -0,0 +1,411 @@ +--- +title: Microsoft.Testing.Platform services overview +description: Learn about the Microsoft.Testing.Platform available services. +author: MarcoRossignoli +ms.author: mrossignoli +ms.date: 07/11/2024 +--- + +# Microsoft.Testing.Platform Services + +The testing platform offers valuable services to both the testing framework and extension points. These services cater to common needs such as accessing the configuration, parsing and retrieving command-line arguments, obtaining the logging factory, and accessing the logging system, among others. `IServiceProvider` implements the _service locator pattern_ for the testing platform. + +The `IServiceProvider` is derived directly from the base class library. + +```csharp +namespace System +{ + public interface IServiceProvider + { + object? GetService(Type serviceType); + } +} +``` + +The testing platform offers handy extension methods to access well-known service objects. All these methods are housed in a static class within the `Microsoft.Testing.Platform.Services` namespace. + +```csharp +public static class ServiceProviderExtensions +{ + public static TService GetRequiredService( + this IServiceProvider provider) + + public static TService? GetService( + this IServiceProvider provider) + + public static IMessageBus GetMessageBus( + this IServiceProvider serviceProvider) + + public static IConfiguration GetConfiguration( + this IServiceProvider serviceProvider) + + public static ICommandLineOptions GetCommandLineOptions( + this IServiceProvider serviceProvider) + + public static ILoggerFactory GetLoggerFactory( + this IServiceProvider serviceProvider) + + public static IOutputDevice GetOutputDevice( + this IServiceProvider serviceProvider) + + // ... and more +} +``` + +Most of the registration factories exposed by extension points provide access to the `IServiceProvider`: For example, when [registering the testing framework](./unit-testing-platform-architecture-extensions.md#registering-a-testing-framework), the `IServiceProvider` is passed as a parameter to the factory method. + +```csharp +ITestApplicationBuilder RegisterTestFramework( + Func capabilitiesFactory, + Func adapterFactory); +``` + +In the preceding code, both the `capabilitiesFactory` and the `adapterFactory` supply the `IServiceProvider` as a parameter. + +## The `IConfiguration` service + +The `IConfiguration` interface can be retrieved using the [`IServiceProvider`](#microsofttestingplatform-services) and provides access to the configuration settings for the testing framework and any extension points. By default, these configurations are loaded from: + +* Environment variables +* A JSON file named `[assemblyName].testingplatformconfig.json` located near the entry point assembly. + +**The order of precedence is maintained, which means that if a configuration is found in the environment variables, the JSON file will not be processed.** + +The interface is a straightforward key-value pair of strings: + +```csharp +public interface IConfiguration +{ + string? this[string key] { get; } +} +``` + +### JSON configuration file + +The JSON file follows a hierarchical structure. To access child properties, you need to use the `:` separator. For example, consider a configuration for a potential testing framework like: + +```json +{ + "CustomTestingFramework": { + "DisableParallelism": true + } +} +``` + +The code snippet would look something like this: + +```csharp +IServiceProvider serviceProvider = null; // Get the service provider... + +var configuration = serviceProvider.GetConfiguration(); + +if (bool.TryParse(configuration["CustomTestingFramework:DisableParallelism"], out var value) && value is true) +{ + // ... +} +``` + +In the case of an array, such as: + +```json +{ + "CustomTestingFramework": { + "Engine": [ + "ThreadPool", + "CustomThread" + ] + } +} +``` + +The syntax to access to the fist element ("ThreadPool") is: + +```csharp +IServiceProvider serviceProvider = null; // Get the service provider... + +var configuration = serviceProvider.GetConfiguration(); + +var fistElement = configuration["CustomTestingFramework:Engine:0"]; +``` + +### Environment variables + +The `:` separator doesn't work with environment variable hierarchical keys on all platforms. `__`, the double underscore, is: + +* Supported by all platforms. For example, the `:` separator is not supported by [Bash](https://linuxhint.com/bash-environment-variables/), but `__` is. +* Automatically replaced by a `:` + +For instance, the environment variable can be set as follows (This example is applicable for Windows): + +```bash +setx CustomTestingFramework__DisableParallelism=True +``` + +You can choose not to use the environment variable configuration source when creating the `ITestApplicationBuilder`: + +```csharp +var options = new TestApplicationOptions(); + +options.Configuration.ConfigurationSources.RegisterEnvironmentVariablesConfigurationSource = false; + +var builder = await TestApplication.CreateBuilderAsync(args, options); +``` + +## The `ICommandLineOptions` service + +The `ICommandLineOptions` service is utilized to fetch details regarding the command-line options that the platform has parsed. The APIs available include: + +```csharp +public interface ICommandLineOptions +{ + bool IsOptionSet(string optionName); + + bool TryGetOptionArgumentList( + string optionName, + out string[]? arguments); +} +``` + +The `ICommandLineOptions` can be obtained through certain APIs, such as the [ICommandLineOptionsProvider](./unit-testing-platform-architecture-extensions.md#the-icommandlineoptionsprovider-extensions), or you can retrieve an instance of it from the [IServiceProvider](#microsofttestingplatform-services) via the extension method `serviceProvider.GetCommandLineOptions()`. + +`ICommandLineOptions.IsOptionSet(string optionName)`: This method allows you to verify whether a specific option has been specified. When specifying the `optionName`, omit the `--` prefix. For example, if the user inputs `--myOption`, you should simply pass `myOption`. + +`ICommandLineOptions.TryGetOptionArgumentList(string optionName, out string[]? arguments)`: This method enables you to check whether a specific option has been set and, if so, retrieve the corresponding value or values (if the arity is more than one). Similar to the previous case, the `optionName` should be provided without the `--` prefix. + +### The `ILoggerFactory` service + +The testing platform comes with an integrated logging system that generates a log file. You can view the logging options by running the `--help` command. +The options you can choose from include: + +```dotnetcli +--diagnostic Enable the diagnostic logging. The default log level is 'Trace'. The file will be written in the output directory with the name log_[MMddHHssfff].diag +--diagnostic-filelogger-synchronouswrite Force the built-in file logger to write the log synchronously. Useful for scenario where you don't want to lose any log (i.e. in case of crash). Note that this is slowing down the test execution. +--diagnostic-output-directory Output directory of the diagnostic logging, if not specified the file will be generated inside the default 'TestResults' directory. +--diagnostic-output-fileprefix Prefix for the log file name that will replace '[log]_.' +--diagnostic-verbosity Define the level of the verbosity for the --diagnostic. The available values are 'Trace', 'Debug', 'Information', 'Warning', 'Error', and 'Critical' +``` + +From a coding standpoint, to log information, you need to obtain the `ILoggerFactory` from the [`IServiceProvider`](#microsofttestingplatform-services). +The `ILoggerFactory` API is as follows: + +```csharp +public interface ILoggerFactory +{ + ILogger CreateLogger(string categoryName); +} + +public static class LoggerFactoryExtensions +{ + public static ILogger CreateLogger(this ILoggerFactory factory); +} +``` + +The logger factory allows you to create an `ILogger` object using the `CreateLogger` API. There's also a convenient API that accepts a generic argument, which will be used as the category name. + +```csharp +public interface ILogger +{ + Task LogAsync( + LogLevel logLevel, + TState state, + Exception? exception, + Func formatter); + + void Log( + LogLevel logLevel, + TState state, + Exception? exception, + Func formatter); + + bool IsEnabled(LogLevel logLevel); +} + +public interface ILogger : ILogger +{ +} + +public static class LoggingExtensions +{ + public static Task LogCriticalAsync(this ILogger logger, string message); + public static Task LogDebugAsync(this ILogger logger, string message); + public static Task LogErrorAsync(this ILogger logger, Exception ex); + public static Task LogErrorAsync(this ILogger logger, string message, Exception ex); + public static Task LogErrorAsync(this ILogger logger, string message); + public static Task LogInformationAsync(this ILogger logger, string message); + public static Task LogTraceAsync(this ILogger logger, string message); + public static Task LogWarningAsync(this ILogger logger, string message); + public static void LogCritical(this ILogger logger, string message); + public static void LogDebug(this ILogger logger, string message); + public static void LogError(this ILogger logger, Exception ex); + public static void LogError(this ILogger logger, string message, Exception ex); + public static void LogError(this ILogger logger, string message); + public static void LogInformation(this ILogger logger, string message); + public static void LogTrace(this ILogger logger, string message); + public static void LogWarning(this ILogger logger, string message); +} +``` + +The `ILogger` object, which is created by the `ILoggerFactory`, offers APIs for logging information at various levels. These logging levels include: + +```csharp +public enum LogLevel +{ + Trace, + Debug, + Information, + Warning, + Error, + Critical, + None, +} +``` + +Here's an example of how you might use the logging API: + +```csharp +... +IServiceProvider provider = null; // Get the service provider... + +var factory = provider.GetLoggerFactory(); + +var logger = factory.CreateLogger(); + +// ... + +if (logger.IsEnabled(LogLevel.Information)) +{ + await logger.LogInformationAsync( + $"Executing request of type '{context.Request}'"); +} + +// ... +``` + +Keep in mind that to prevent unnecessary allocation, you should check if the level is *enabled* using the `ILogger.IsEnabled(LogLevel)` API. + +## The `IMessageBus` service + +The message bus service is the central mechanism that facilitates information exchange between the test framework and its extensions. + +The message bus of the testing platform employs the _publish-subscribe pattern_. + +The overarching structure of the shared bus is as follows: + +:::image type="content" source="./media/message-bus.png" lightbox="./media/message-bus.png" alt-text="A picture representing the interactions of the various extensions with the message bus."::: + +As illustrated in the diagram, which includes an extensions and a test framework, there are two potential actions: pushing information to the bus or consuming information from the bus. + +The `IMessageBus` satisfied the *pushing action* to the bus and the API is: + +```csharp +public interface IMessageBus +{ + Task PublishAsync( + IDataProducer dataProducer, + IData data); +} + +public interface IDataProducer : IExtension +{ + Type[] DataTypesProduced { get; } +} + +public interface IData +{ + string DisplayName { get; } + string? Description { get; } +} +``` + +Consider the following details about the parameters: + +* `IDataProducer`: The `IDataProducer` communicates to the message bus the `Type` of information it can supply and establishes ownership through inheritance from the base interface [IExtension](./unit-testing-platform-architecture-extensions.md#the-iextension-interface). This implies that you can't indiscriminately push data to the message bus; you must declare the data type produced in advance. If you push unexpected data, an exception will be triggered. + +* `IData`: This interface serves as a placeholder where you only need to provide descriptive details such as the name and a description. The interface doesn't reveal much about the data's nature, which is intentional. It implies that the test framework and extensions can push any type of data to the bus, and this data can be consumed by any registered extension or the test framework itself. + +This approach facilitates the evolution of the information exchange process, preventing breaking changes when an extension is unfamiliar with new data. **It allows different versions of extensions and the test framework to operate in harmony, based on their mutual understanding**. + +The opposite end of the bus is referred to as a [consumer](./unit-testing-platform-architecture-extensions.md#the-idataconsumer-extensions), which is subscribed to a specific type of data and can thus consume it. + +> [!IMPORTANT] +> Always use *await* the call to `PublishAsync`. If you don't, the `IData` might not be processed correctly by the testing platform and extensions, which could lead to subtle bugs. It's only after you've returned from the *await* that you can be assured that the `IData` has been queued for processing on the message bus. Regardless of the extension point you're working on, ensure that you've awaited all `PublishAsync` calls before exiting the extension. For example, if you're implementing the [`testing framework`](./unit-testing-platform-architecture-extensions.md#creating-a-testing-framework), you should not call `Complete` on the [requests](./unit-testing-platform-architecture-extensions.md#handling-requests) until you've awaited all `PublishAsync` calls for that specific request. + +## The `IOutputDevice` service + +The testing platform encapsulates the idea of an *output device*, allowing the testing framework and extensions to *present* information by transmitting any kind of data to the currently utilized display system. + +The most traditional example of an *output device* is the console output. + +> [!NOTE] +> While the testing platform is engineered to support custom *output devices*, currently, this extension point is not available. + +To transmit data to the *output device*, you must obtain the `IOutputDevice` from the [`IServiceProvider`](#microsofttestingplatform-services). + +The API consists of: + +```csharp +public interface IOutputDevice +{ + Task DisplayAsync( + IOutputDeviceDataProducer producer, + IOutputDeviceData data); +} + +public interface IOutputDeviceDataProducer : IExtension +{ +} + +public interface IOutputDeviceData +{ +} +``` + +The `IOutputDeviceDataProducer` extends the [`IExtension`](./unit-testing-platform-architecture-extensions.md#the-iextension-interface) and provides information about the sender to the *output device*. + +The `IOutputDeviceData` serves as a placeholder interface. The concept behind `IOutputDevice` is to accommodate more intricate information than just colored text. For instance, it could be a complex object that can be graphically represented. + +The testing platform, by default, offers a traditional colored text model for the `IOutputDeviceData` object: + +```csharp +public class TextOutputDeviceData : IOutputDeviceData +{ + public TextOutputDeviceData(string text) + public string Text { get; } +} + +public sealed class FormattedTextOutputDeviceData : TextOutputDeviceData +{ + public FormattedTextOutputDeviceData(string text) + public IColor? ForegroundColor { get; init; } + public IColor? BackgroundColor { get; init; } +} + +public sealed class SystemConsoleColor : IColor +{ + public ConsoleColor ConsoleColor { get; init; } +} +``` + +Here's an example of how you might use the colored text with the *active* output device: + +```csharp +IServiceProvider provider = null; // Get the service provider... + +var outputDevice = provider.GetOutputDevice(); + +await outputDevice.DisplayAsync( + this, + new FormattedTextOutputDeviceData($"TestingFramework version '{Version}' running tests with parallelism of {_dopValue}") + { + ForegroundColor = new SystemConsoleColor + { + ConsoleColor = ConsoleColor.Green + } + }); +``` + +Beyond the standard use of colored text, the main advantage of `IOutputDevice` and `IOutputDeviceData` is that the *output device* is entirely independent and unknown to the user. This allows for the development of complex user interfaces. For example, it's entirely feasible to implement a *real-time* web application that displays the progress of tests. + +## The `IPlatformInformation` service + +Provides information about the platform such as: name, version, commit hash and build date. diff --git a/docs/core/testing/unit-testing-platform-architecture.md b/docs/core/testing/unit-testing-platform-architecture.md new file mode 100644 index 0000000000000..137bcc844b93d --- /dev/null +++ b/docs/core/testing/unit-testing-platform-architecture.md @@ -0,0 +1,167 @@ +--- +title: Microsoft.Testing.Platform architecture overview +description: Learn about Microsoft.Testing.Platform architecture. +author: MarcoRossignoli +ms.author: mrossignoli +ms.date: 07/11/2024 +--- + +# Microsoft.Testing.Platform architecture + +Welcome to our new test platform! To help you get acquainted with its capabilities, we'll start with a simple example that demonstrates how to register and run a test. This foundational example will give you a solid understanding of the core functionality and how to get started quickly. + +> [!NOTE] +> All of the concepts in this article are exemplified in the Microsoft Test Framework repository as a complete sample. For more information, see the [Sample code](https://github.com/microsoft/testfx/tree/main/samples/public/TestingPlatformExamples). + +[Step 1: Register and Run a simple test application](#step-1-register-and-run-a-simple-test-application) + +In this initial example, you walk through the basic steps to declare and run a test application. This straightforward approach ensures that you can immediately start using the platform with minimal setup. + +[Step 2: Extending the Platform](#step-2-extending-the-platform) + +After you've discovered how to create your first test application, you explore an example of extension to cover partially the concepts surrounding the test application extensions. + +[Step 3: Comprehensive Overview of Extension Points](#step-3-comprehensive-overview-of-extension-points) + +Once you're comfortable with the basics, you delve into the various extension points. This will include: + +1. **Platform and Test Framework Capabilities**: Understanding the full range of capabilities provided by the platform and the test framework, allowing you to leverage them effectively. + +1. **Custom Test Framework**: How to write and register your custom test framework, enabling you to tailor the testing environment to your specific requirements. + +1. **In-Process and Out-of-Process Extensions**: Detailed instructions on how to write and register both in-process and out-of-process extensions, offering flexibility in how you extend the platform. + +1. **Order of Execution**: Clarifying the order of execution for the various extension points to ensure seamless integration and operation of your custom extensions. + +[Step 4: Available Services and Helpers](#step-4-available-services) + +Finally, you review an exhaustive list of the available services and helper functions within the platform. This section will serve as a reference to help you leverage all the tools at your disposal for creating robust and efficient test extensions. + +By following this structured approach, you will gain a comprehensive understanding of our test platform, from basic usage to advanced customization. Let's get started and explore the full potential of what our platform can offer! + +## Step 1: Register and Run a simple test application + +To introduce the architecture of the testing platform, this document will use the classic console application (for Windows) as the host. The samples in this document are written in C#, but you can use the testing platform with any language that supports the .NET Ecma specification, and run on any OS supported by .NET. To use the platform, simply reference the `Microsoft.Testing.Platform.dll` assembly, which can be consumed through the official NuGet package available at . + +In a console project `Contoso.UnitTests.exe` the following `Main` method defines the entry point: + +```csharp +public static async Task Main(string[] args) +{ + ITestApplicationBuilder builder = await TestApplication.CreateBuilderAsync(args); + + builder.RegisterTestFramework(); + + using ITestApplication testApp = await builder.BuildAsync(); + + return await testApp.RunAsync(); +} +``` + +This code includes everything needed to execute a test session, except for registering a test framework such as MSTest through `RegisterTestFramework`. This code is shown and explained in later sections. + +> [!NOTE] +> The default behavior is to have the entry point automatically generated by the testing platform MSBuild task. This is done by adding the `Microsoft.Testing.Platform.MSBuild` package to your project. + +When `Contoso.UnitTests.exe` application is started a standard Windows process is created, and the testing platform interacts with the registered testing framework to execute the testing session. + +A single process is created to carry out this work: + + +:::image type="content" source="./media/platform-testhost.png" lightbox="./media/platform-testhost.png" alt-text="A diagram representing the test host process."::: + +The testing platform includes a built-in display device that writes the testing session information in the terminal, similar to: + +```bash +Microsoft(R) Testing Platform Execution Command Line Tool +Version: 1.1.0+8c0a8fd8e (UTC 2024/04/03) +RuntimeInformation: win-x64 - .NET 9.0.0-preview.1.24080.9 +Copyright(c) Microsoft Corporation.  All rights reserved. +Passed! - Failed: 0, Passed: 1, Skipped: 0, Total: 1, Duration: 5ms - Contoso.UnitTests.dll (win-x64 - .NET 9.0.0-preview.1.24080.9) +``` + +> [!NOTE] +> The known exit codes returned by the `ITestApplication.RunAsync()` call are detailed in [platform exit codes](./unit-testing-platform-exit-codes.md). + +## Step 2: Extending the platform + +Test runs commonly collect code coverage information, or similar information to evaluate code quality. Such workloads may require configuration before the test host process starts, for example setting environment variables. + +The testing platform accommodates this by having **out-of-process** extensions. When running with an out-of-process extensions, the testing platform will start multiple processes and it will manage them appropriately. + +The following example demonstrates how to register a code coverage feature using a **TestHostController** extension. + +```csharp +ITestApplicationBuilder builder = await TestApplication.CreateBuilderAsync(args); + +builder.RegisterTestFramework(); +builder.AddCodeCoverage(); + +using ITestApplication testApp = await builder.BuildAsync(); + +return await testApp.RunAsync(); +``` + +The `builder.AddCodeCoverage();` internally uses the `TestHostController` extensibility point, which is an out-of-process extensibility point. + +```csharp +public static class TestApplicationBuilderExtensions +{ + public static ITestApplicationBuilder AddCodeCoverage( + this ITestApplicationBuilder builder) + { + builder.TestHostControllers + .AddEnvironmentVariableProvider(/* ... */); + + // ... + + return builder; + } +} +``` + +The parameters for the api `AddEnvironmentVariableProvider` will be explained in later sections. + +When running `Contoso.UnitTests.exe` this time, the testing platform detects that a `TestHostController` extension is registered. As a result, it starts another instance of the `Contoso.UnitTests.exe` process as a child process. This is done to properly set the environment variables as required by the extension registered with the `AddEnvironmentVariableProvider` API. + +The process layout looks like this: + + +:::image type="content" source="./media/platform-testhostcontroller-testhost.png" lightbox="./media/platform-testhostcontroller-testhost.png" alt-text="A diagram representing the process layout of the test host and test host controller."::: + +> [!NOTE] +> The provided example assumes a console application layout, which handles the start process correctly and propagates all command line arguments to the child process. +> If you are using a different host, you need to ensure that the entry point code correctly forwards the process entry point (the "Main") to the appropriate code block. +> The runtime simply starts itself with the same command line arguments. + +The above section provides a brief introduction to the architecture of the testing platform. The current extensibility points are divided into two categories: + +1. **In process** extensions can be accessed through the `TestHost` property of the test application builder. In process means that they will run in the same process as the test framework. + + ```csharp + ITestApplicationBuilder builder = await TestApplication.CreateBuilderAsync(args); + + builder.RegisterTestFramework(); + + builder.TestHost.AddXXX(/* ... */); + ``` + + As observed, the most crucial extension point is the in-process *testing framework* (`RegisterTestFramework`), which is the only **mandatory** one. + +1. **Out of process** extensions can be accessed through the `TestHostControllers` property of the test application builder. These extensions run in a separate process from the test framework to "observe" it. + + ```csharp + ITestApplicationBuilder builder = await TestApplication.CreateBuilderAsync(args); + + builder.TestHostControllers.AddXXX(/* ... */); + ``` + +## Step 3: Comprehensive Overview of Extension points + +Let's start by getting familiar with the concept of [capabilities](./unit-testing-platform-architecture-capabilities.md) before diving into the various [extensions points](./unit-testing-platform-architecture-extensions.md). + +## Step 4: Available services + +The testing platform offers valuable services to both the testing framework and extension points. These services cater to common needs such as accessing the configuration, parsing and retrieving command-line arguments, obtaining the logging factory, and accessing the logging system, among others. `IServiceProvider` implements the _service locator pattern_ for the testing platform. + +All the services, helpers and technical information about how to access and use these services is listed [here](./unit-testing-platform-architecture-services.md). diff --git a/docs/core/testing/unit-testing-platform-exit-codes.md b/docs/core/testing/unit-testing-platform-exit-codes.md index 2b29df0d99c38..7bdf4210bfa30 100644 --- a/docs/core/testing/unit-testing-platform-exit-codes.md +++ b/docs/core/testing/unit-testing-platform-exit-codes.md @@ -19,7 +19,7 @@ ms.topic: reference | `3` | The exit code `3` indicates that the test session was aborted. A session can be aborted using Ctrl+C, as an example. | | `4` | The exit code `4` indicates that the setup of used extensions is invalid and the tests session cannot run. | | `5` | The exit code `5` indicates that the command line arguments passed to the test app are invalid. | -| `6` | The exit code `6` indicates that the test session is using a nonimplemented feature. | +| `6` | The exit code `6` indicates that the test session is using a non-implemented feature. | | `7` | The exit code `7` indicates that a test session was unable to complete successfully, and likely crashed. It's possible that this was caused by a test session that was run via a test controller's extension point. | | `8` | The exit code `8` indicates that the test session ran zero tests. | | `9` | The exit code `9` indicates that the minimum execution policy for the executed tests was violated. | diff --git a/docs/core/testing/unit-testing-platform-intro.md b/docs/core/testing/unit-testing-platform-intro.md index 722a484aa09ef..34a38c549584c 100644 --- a/docs/core/testing/unit-testing-platform-intro.md +++ b/docs/core/testing/unit-testing-platform-intro.md @@ -12,6 +12,30 @@ Microsoft.Testing.Platform is a lightweight and portable alternative to [VSTest] `Microsoft.Testing.Platform` is open source. You can find `Microsoft.Testing.Platform` code in [microsoft/testfx](https://github.com/microsoft/testfx/tree/main/src/Platform/Microsoft.Testing.Platform) GitHub repository. +## Microsoft.Testing.Platform pillars + +This new testing platform is built on the .NET Developer Experience Testing team's experience and aims to address the challenges encountered since the release of .NET Core in 2016. While there's a high level of compatibility between the .NET Framework and the .NET Core/.NET, some key features like the plugin-system and the new possible form factors of .NET compilations have made it complex to evolve or fully support the new runtime feature with the current [VSTest platform](https://github.com/microsoft/vstest) architecture. + +The main driving factors for the evolution of the new testing platform are detailed in the following: + +* **Determinism**: Ensuring that running the same tests in different contexts (local, CI) will produce the same result. The new runtime does not rely on reflection or any other dynamic .NET runtime feature to coordinate a test run. + +* **Runtime transparency**: The test runtime does not interfere with the test framework code, it does not create isolated contexts like `AppDomain` or `AssemblyLoadContext`, and it does not use reflection or custom assembly resolvers. + +* **Compile-time registration of extensions**: Extensions, such as test frameworks and in/out-of-process extensions, are registered during compile-time to ensure determinism and to facilitate detection of inconsistencies. + +* **Zero dependencies**: The core of the platform is a single .NET assembly, `Microsoft.Testing.Platform.dll`, which has no dependencies other than the supported runtimes. + +* **Hostable**: The test runtime can be hosted in any .NET application. While a console application is commonly used to run tests, you can create a test application in any type of .NET application. This allows you to run tests within special contexts, such as devices or browsers, where there may be limitations. + +* **Support all .NET form factors**: Support current and future .NET form factors, including Native AOT. + +* **Performant**: Finding the right balance between features and extension points to avoid bloating the runtime with non-fundamental code. The new test platform is designed to "orchestrate" a test run, rather than providing implementation details on how to do it. + +* **Extensible enough**: The new platform is built on extensibility points to allow for maximum customization of runtime execution. It allows you to configure the test process host, observe the test process, and consume information from the test framework within the test host process. + +* **Single module deploy**: The hostability feature enables a single module deploy model, where a single compilation result can be used to support all extensibility points, both out-of-process and in-process, without the need to ship different executable modules. + ## Supported test frameworks * MSTest. In MSTest, the support of `Microsoft.Testing.Platform` is done via [MSTest runner](unit-testing-mstest-runner-intro.md). diff --git a/docs/navigate/devops-testing/toc.yml b/docs/navigate/devops-testing/toc.yml index 81c46cd43cf37..ccbecd49da1f4 100644 --- a/docs/navigate/devops-testing/toc.yml +++ b/docs/navigate/devops-testing/toc.yml @@ -179,8 +179,16 @@ items: href: ../../core/testing/unit-testing-platform-exit-codes.md - name: Integration with dotnet test href: ../../core/testing/unit-testing-platform-integration-dotnet-test.md - - name: Testing platform SDK - href: https://github.com/microsoft/testfx/blob/main/docs/testingplatform/Index.md + - name: Testing platform Architecture + items: + - name: Overview + href: ../../core/testing/unit-testing-platform-architecture.md + - name: Capabilities + href: ../../core/testing/unit-testing-platform-architecture-capabilities.md + - name: Extensions + href: ../../core/testing/unit-testing-platform-architecture-extensions.md + - name: Services + href: ../../core/testing/unit-testing-platform-architecture-services.md - name: Run selective unit tests href: ../../core/testing/selective-unit-tests.md - name: Order unit tests