From 8e7dc5722e6416aa3bebca72593af8ac1edec999 Mon Sep 17 00:00:00 2001 From: adam-saucelabs <159158755+adam-saucelabs@users.noreply.github.com> Date: Wed, 14 Aug 2024 10:57:45 -0600 Subject: [PATCH 1/5] Create overview.md First iteration of job overview page --- docs/insights/overview.md | 65 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 65 insertions(+) create mode 100644 docs/insights/overview.md diff --git a/docs/insights/overview.md b/docs/insights/overview.md new file mode 100644 index 0000000000..2de5871b65 --- /dev/null +++ b/docs/insights/overview.md @@ -0,0 +1,65 @@ +--- +id: overview +title: Analyzing Test Case Health +sidebar_label: Job Overview +description: Gain insight into test case health, test summaries and breakdowns across Real or Virtual devices and analyze errors to identify where and why errors occurred over time. +--- + +import useBaseUrl from '@docusaurus/useBaseUrl'; + +The **Job Overview** page, accessible from the **Insights** heading, offers two frames of reference to view health of the aggregate of your tests. First is the Overview tab, which offers a visual overview of test case health over time, job summary, and job breakdown. Job Overview also gives you access to the Errors tab which details errors for all your tests over time. + +:::note +The Extended Debugging feature offers access to HAR files and JavaScript console logs, assisting in the identification of flaky tests. For more details, refer to [Debugging Tests with JavaScript Console Logs and HAR Files (Extended Debugging)](/insights/debug). +::: + +## Accessing the Job Overview Page + +1. Click the **Insights** tab to expand its submenu. +1. Click **Job Overview**. +1. Apply [filters](scope.md#using-filters-to-adjust-the-scope-of-your-data) to narrow down the list of tests within a specific range. + +## The **Job Overview** page offers three data visualizations on the Overview tab to give you insight into your tests over time. + +### Job Overview +The Job Overview visualization gives you a bar chart with a few different categories reprsented to give you an overview of how your test suite has performed over a given period of time. This bar chart is usually broken down into five categories. + +- Consistently Failing tests +- Consistently Passing tests +- Consistently Erroring tests +- Tests with Missing Status +- Inconsistent Resulting tests + +overview data visualization + +Using this data, you can identify if your test suite is performing as expected and where to debug if necessary. Clicking each of the headings will take you to that filter within the **Job History** page to drill down into the tests that are comprised in that filter. Use the filters above the visualization to identify builds, browsers, or platforms and the time-frame to surface test data that is most important to your organization. + +### Job Summary +The Job Summary section gives you a pie chart, and associated raw numbers with percentages, of the fundamentals of your testing strategy over the selected period of time. This visualization also compares usage for the same time interval previous. For example, if you select 30 days from the filter you will see your current 30 days of tests compared to your last 30 days of tests in this visualization. Job Summary shows you proportions of: + +- Total Runs +- Passed tests (Passed) +- Errored tests (Error) +- Failed tests (Failures) +- Completed tests (Completed) + +:::note +Completed tests are sometimes aggregated due to not receiving a pass or failure declaration or intent. Be sure to [send all your pass/fail results](/basics/test-config-annotation/test-annotation#setting-passfail) to Sauce Labs to take advantage of Insights. +::: + +job summary data visualization + +### Job Breakdown +Job Breakdown gives you insight into your test suite by categorizing all your individual jobs by OS, Browser version, Framework and Device type. You can use this section to understand where your testing coverage is currently, where you may need more testing based on current analytics, and deep dive into OS or Browser versions to ensure proper coverage across your testing suite. + +job breakdown data visualization + +:::note +Read more about how your testing strategy might benefit from our [Virtual and Real device clouds](/mobile-apps/supported-devices.md) to ensure proper coverage across your user base. +::: + +## The **Job Overview** page also offers insight into your testing errors over time via the Errors tab. + +Using the Errors tab from the Job Overview page you can use the same familiar filters like OS, test owner, or Framework to analyze your testing errors and understand what actions to take based upon the resulting set of errors. Errors shows visualization of errors over time based upon your selected filters, and then surfaces the most pervasive errors in that data set for you to debug and take action on. + +errors data visualization From ad679713f68fba6861a0712f02e9468053e6e1ab Mon Sep 17 00:00:00 2001 From: adam-saucelabs <159158755+adam-saucelabs@users.noreply.github.com> Date: Wed, 14 Aug 2024 11:39:38 -0600 Subject: [PATCH 2/5] small errors update --- docs/insights/overview.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/insights/overview.md b/docs/insights/overview.md index 2de5871b65..36d1aa5858 100644 --- a/docs/insights/overview.md +++ b/docs/insights/overview.md @@ -60,6 +60,8 @@ Read more about how your testing strategy might benefit from our [Virtual and Re ## The **Job Overview** page also offers insight into your testing errors over time via the Errors tab. +### Errors + Using the Errors tab from the Job Overview page you can use the same familiar filters like OS, test owner, or Framework to analyze your testing errors and understand what actions to take based upon the resulting set of errors. Errors shows visualization of errors over time based upon your selected filters, and then surfaces the most pervasive errors in that data set for you to debug and take action on. errors data visualization From bf36899c75f226914deec32fa5d34a56839ccc36 Mon Sep 17 00:00:00 2001 From: adam-saucelabs <159158755+adam-saucelabs@users.noreply.github.com> Date: Thu, 15 Aug 2024 09:06:28 -0600 Subject: [PATCH 3/5] updating sidebar for overview --- sidebars.js | 1 + 1 file changed, 1 insertion(+) diff --git a/sidebars.js b/sidebars.js index c103d3e6d3..7f0c1d172f 100644 --- a/sidebars.js +++ b/sidebars.js @@ -1659,6 +1659,7 @@ module.exports = { items: [ 'insights', 'insights/scope', + 'insights/overview', 'insights/history', 'insights/trends', 'insights/usage-report', From bdf04a65c85a55c564553cc02d41aaad3519a268 Mon Sep 17 00:00:00 2001 From: Adam E <159158755+adam-saucelabs@users.noreply.github.com> Date: Thu, 15 Aug 2024 09:24:29 -0600 Subject: [PATCH 4/5] small formatting changes to overview.md --- docs/insights/overview.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/insights/overview.md b/docs/insights/overview.md index 36d1aa5858..f21b25db6e 100644 --- a/docs/insights/overview.md +++ b/docs/insights/overview.md @@ -19,7 +19,8 @@ The Extended Debugging feature offers access to HAR files and JavaScript console 1. Click **Job Overview**. 1. Apply [filters](scope.md#using-filters-to-adjust-the-scope-of-your-data) to narrow down the list of tests within a specific range. -## The **Job Overview** page offers three data visualizations on the Overview tab to give you insight into your tests over time. +## Job Overview Tab +The **Job Overview** page offers three data visualizations on the Overview tab to give you insight into your tests over time. ### Job Overview The Job Overview visualization gives you a bar chart with a few different categories reprsented to give you an overview of how your test suite has performed over a given period of time. This bar chart is usually broken down into five categories. @@ -58,7 +59,8 @@ Job Breakdown gives you insight into your test suite by categorizing all your in Read more about how your testing strategy might benefit from our [Virtual and Real device clouds](/mobile-apps/supported-devices.md) to ensure proper coverage across your user base. ::: -## The **Job Overview** page also offers insight into your testing errors over time via the Errors tab. +## Errors Tab +The **Job Overview** page also offers insight into your testing errors over time via the Errors tab. ### Errors From 66d93e89d76b55ba8a83ac13c8357b850b420609 Mon Sep 17 00:00:00 2001 From: Adam E <159158755+adam-saucelabs@users.noreply.github.com> Date: Tue, 3 Sep 2024 10:06:37 -0600 Subject: [PATCH 5/5] naming consistency updates --- docs/insights/overview.md | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/docs/insights/overview.md b/docs/insights/overview.md index f21b25db6e..c084ae0a2f 100644 --- a/docs/insights/overview.md +++ b/docs/insights/overview.md @@ -7,7 +7,7 @@ description: Gain insight into test case health, test summaries and breakdowns a import useBaseUrl from '@docusaurus/useBaseUrl'; -The **Job Overview** page, accessible from the **Insights** heading, offers two frames of reference to view health of the aggregate of your tests. First is the Overview tab, which offers a visual overview of test case health over time, job summary, and job breakdown. Job Overview also gives you access to the Errors tab which details errors for all your tests over time. +The **Job Overview** page, accessible from the **Insights** heading, offers two frames of reference to view health of the aggregate of your jobs. First is the Overview tab, which offers a visual overview of job case health over time, job summary, and job breakdown. Job Overview also gives you access to the Errors tab which details errors for all your jobs over time. :::note The Extended Debugging feature offers access to HAR files and JavaScript console logs, assisting in the identification of flaky tests. For more details, refer to [Debugging Tests with JavaScript Console Logs and HAR Files (Extended Debugging)](/insights/debug). @@ -17,32 +17,32 @@ The Extended Debugging feature offers access to HAR files and JavaScript console 1. Click the **Insights** tab to expand its submenu. 1. Click **Job Overview**. -1. Apply [filters](scope.md#using-filters-to-adjust-the-scope-of-your-data) to narrow down the list of tests within a specific range. +1. Apply [filters](scope.md#using-filters-to-adjust-the-scope-of-your-data) to narrow down the list of jobs within a specific range. ## Job Overview Tab -The **Job Overview** page offers three data visualizations on the Overview tab to give you insight into your tests over time. +The **Job Overview** page offers three data visualizations on the Overview tab to give you insight into your jobs over time. ### Job Overview -The Job Overview visualization gives you a bar chart with a few different categories reprsented to give you an overview of how your test suite has performed over a given period of time. This bar chart is usually broken down into five categories. +The Job Overview visualization gives you a bar chart with a few different categories reprsented to give you an overview of how your suite has performed over a given period of time. This bar chart is usually broken down into five categories. -- Consistently Failing tests -- Consistently Passing tests -- Consistently Erroring tests -- Tests with Missing Status -- Inconsistent Resulting tests +- Consistently Failing +- Consistently Passing +- Consistently Erroring +- Jobs with Missing Status +- Inconsistent Resulting overview data visualization Using this data, you can identify if your test suite is performing as expected and where to debug if necessary. Clicking each of the headings will take you to that filter within the **Job History** page to drill down into the tests that are comprised in that filter. Use the filters above the visualization to identify builds, browsers, or platforms and the time-frame to surface test data that is most important to your organization. ### Job Summary -The Job Summary section gives you a pie chart, and associated raw numbers with percentages, of the fundamentals of your testing strategy over the selected period of time. This visualization also compares usage for the same time interval previous. For example, if you select 30 days from the filter you will see your current 30 days of tests compared to your last 30 days of tests in this visualization. Job Summary shows you proportions of: +The Job Summary section gives you a pie chart, and associated raw numbers with percentages, of the fundamentals of your testing strategy over the selected period of time. This visualization also compares usage for the same time interval previous. For example, if you select 30 days from the filter you will see your current 30 days of jobs compared to your last 30 days of jobs in this visualization. Job Summary shows you proportions of: - Total Runs -- Passed tests (Passed) -- Errored tests (Error) -- Failed tests (Failures) -- Completed tests (Completed) +- Passed +- Errored +- Failed +- Completed :::note Completed tests are sometimes aggregated due to not receiving a pass or failure declaration or intent. Be sure to [send all your pass/fail results](/basics/test-config-annotation/test-annotation#setting-passfail) to Sauce Labs to take advantage of Insights.