Terminology and Formulas

TestOps Terminology

Refer to the following Katalon Studio key terms that are used throughout the TestOps product documentation.

​Katalon Platform​ organizes users into four management levels: Account, Organization, Team, and Project. The Account level is the highest, followed by the Organization, Team, and Project levels. For more information, see: Administrative Roles and Permissions.

Term	Description
Administrator	A Katalon Platform​ user who can make changes that can affect all or users of a specific management level.
User	A ​Katalon Platform​ user. A user has the privileges and permissions based on his or her assigned management level.
Billing manager	​A Katalon Platform​ user with privileges or permissions that pertains to managing the Account subscription and billing, but is not an administrator user.
Member	A ​Katalon Platform​ user assigned to an Organization, Project, and/or Team. A member does not have the privileges of a billing manager or an administrator.

Agent

An agent connects a local machine to the TestOps servers for test executions.

API performance anomalies

The API performance anomalies identify web services that take more/less time to respond than usual by applying the local outlier factor (LOF) on 30 latest execution requests.

Assertions

An assertion determines whether the automated test case succeeds or not.

Katalon TestOps provides you with a specific view of assertions in each test case to evaluate the quality of test cases and ensure your tested application/software is working correctly.

Assertions help to check whether a condition is true (e.g., whether labels, data, API responses are rendered correctly).

BDD Test Result

A BDD Test Result represents the result of a test following the Behavior Driven Development (BDD) Framework.

In Katalon TestOps, the Features of BDD tests are displayed as Requirements, while Scenarios are displayed as Test Cases.

Build

A build represents a group of features/tasks in a release, used for management or testing purposes.

In Katalon TestOps, a build can consist of one or more test cases.

When users map a test result to a build, the test result is automatically mapped to the corresponding release. However, if users map a test result to a release, the test result is not mapped to any specific build.

Build Label

A build label represents a piece of code committed to the source code.

Defect

A defect identifies a failed test run and links the test result to a Jira issue (Jira bug).

Interval (Trigger)

An interval is a time period between each scheduled execution.

In Katalon TestOps, the Interval Unit values are Minute, Hour, Day, Week. For example, if you set Interval Unit as Hour, Interval as 5, it means the execution is scheduled for every 5 hours.

Key (Jira)

A key represents the ID of a Jira issue you have linked a test case/test result to.

In Katalon TestOps, you link a test case to Jira Requirements and link a test result (if the test run fails) to a Jira Defect.

Last Ping (Agent)

The last ping is the last time an Agent connects to a server to ask for a job to execute.

Maintainer

A maintainer is responsible for all test results of a test case.

In Katalon TestOps, within an organization, you can assign a person to maintain a test case (the maintainer) and the history of its test results.

Organization

An organization is created in Katalon TesOps to brings together all users, teams, and projects of a company.

In TestOps, the organization level is the highest level of management, followed by the team level and the project level.

Path

A path is the location of a test in Katalon Studio.

In Katalon TestOps, each test case has a path if it is uploaded from Katalon Studio.

Profile

A profile is an execution profile that is created in Katalon Studio (KS).

In Katalon TestOps, the KS execution profiles are displayed as Profiles. A test run can have multiple profiles. This helps cover multiple and different environments to execute automated test scripts with ease.

Properties (Visual Testing)

A property represents the ID of a baseline image.

Release

A release represents a version or a milestone of your tested software.

In Katalon TestOps, a release consists of one or more test cases.

Request (API testing)

A request in API testing represents the action of calling a server and asking it to perform a task.

Requirement (Jira)

A requirement represents the Jira issue that is linked to a test case.

Script Repository

A script repository stores all the test assets for a test automation project (tests, data files, scripts files, macros, etc.)

In Katalon TestOps, you can create a script repository to manage all test scripts and decide which one to be executed along with the given test environment. You can upload files to a script repository from an external source.

Session

A session is created once a test run is executed.

In Katalon TestOps, each test run can be executed as one or many test sessions depending on whether the tests run in parallel or not.

A session can contain:

• one or more test cases
• a test environment
• test scripts (in a script repository) used for the execution of each test case.

Test Case

A test case consists of multiple test results once a test is executed.

In Katalon TestOps, you can see detailed information of test cases to see if your software is free of bugs and whether it is working as expected. You can link test cases to Jira Requirements to ensure good test coverage. A test case can be linked to one or more Requirements.

Test Environment

A test environment is a place where a test run could be executed with the command-line interface.

In Katalon TestOps, you can use your local machine as a test environment. This test environment must be controlled by an agent.

Test Execution

A test execution provides the results of test cases executed by TestOps scheduler.

Katalon TestOps does not include reports uploaded from Katalon Studio or other tools.

Test Run

A test run consists of one or more test cases to be executed (execution of more test cases depends on whether a test run is conducted in parallel).

In Katalon TestOps, a test run keeps track of all test results, status, duration, assignee, etc.

Test Run List

A test run list is a collection of all test runs that have the same configurations.

Test Suite

A test suite is a collection of test cases.

Test Suite Collection

A test suite collection is a list of test suites that allows users to manage and plan test executions better.

Threshold (Agent)

A threshold indicates the maximum number of sessions that an agent can execute at the same time.

Total Assigned Sessions (Agent)

The total assigned sessions represents the total number of sessions assigned to an agent to execute tests.

In Katalon TestOps, the maximum number of assigned sessions is also called Threshold.

Total Executing Sessions (Agent)

The total executing sessions represents the number of sessions an agent executes in real time.

Traceability Matrix

A traceability matrix manages the relationships across requirements, test cases and defects.

Trigger

A trigger determines when a test run is executed.

In Katalon TestOps, this function helps leverage remote execution for complete control of the testing plan.

User Management

The User Management page is an administration tool to manage users.

In Katalon TestOps, this feature allows you to:

• invite users to your organization (you can also set up permission access for other Katalon products).
• delete pending invitations.
• set and edit a user's role.
• remove users from your organization.

UUID (Agent)

An UUID is an identifier that is generated when an agent is set up successfully.

Visual Baseline (Visual Testing)

A visual baseline is a standard reference image used to compare with the Checkpoint screenshot(s) during a test execution.

TestOps Formulas

% change of Test Case (Dashboard)

= (Total test cases this week/Total test cases last week) x 100%

% change of Test Result (Dashboard)

= (Total test results this week/Total test results last week) x 100%

% fail (Build)

= (Total failed test cases/Total test cases) x 100%

% flakiness

= (Number of times the status of test results changes/Total number of test results) x 100%

The total number of test results represents 30 latest test results (sorted by execution start time).

% pass (Build)

= (Total passed test cases/Total test cases) x 100%

Active/Archived Release

A new release is active by default. You can track it when it's active.

In Katalon TestOps, you can also archive a release to stop tracking it.

Active Test Case

An active test case is a test case that has been run in the last 2 months, based on the start time of the latest execution.

Average Duration (Test Case)

= (Total duration of all test results/Total number of test results) x 100%

The calculation is based on the last 100 test results.

Average Duration (Web Services)

The average duration represents the average amount of time based on the last 30 execution requests (sorted by the most recent execution start time).

Development Progress (Dashboard)

= (Resolved Jira issues/Total Jira issues) x 100%

Duration (Session)

= (End time - Start time)

Start time = when an agent starts receiving a job.

End time = when uploading reports to TestOps is done.

Duration (Test Result)

= (End time - Start time)

Duration (Test Run)

= (End time - Start time)

This formula shows the actual time of running tests, including test runs via TestOps Scheduler and Reports Upload.

Execution Time (Dashboard)

The execution time represents the total duration of test results.

Flaky Test Case

A flaky test case represents the test case that fails to produce the same results each time the same analytics is run.

Max/Min Duration (Web Services)

The maximum/minimum duration is based on the last 30 execution requests.

Offline Agent

An offline agent is the agent whose last ping is longer than the active time.

By default, the active time is 5 minutes. If the agent does not run any job for over 5 minutes, it is considered offline.

Platform Coverage (Test Case)

A platform coverage shows the quality of test cases by operating system (OS) and browser-basis.

The color of the dot indicates that the test has passed or failed.

• Red dot = failed test
• Green dot = passed test

The size of the dot represents the number of tests (e.g., the bigger the dot is, the more tests are).

Run Frequency (Test Run)

A run frequency is a statistic of the scheduled test runs in a day.

The color of the dot indicates that the test has passed or failed.

The size of the dot represents the number of test results (e.g., the bigger the dot is, the more test results are).

Similar Failures

The similar failures represents the test runs that have at least 70% of error similarities.

Slowest Test Case

The slowest test case is a test case that has the longest average duration.

Katalon TestOps ranks active test cases by their average duration. The shorter an average duration is, the more active/faster a test execution is.

Stale Test Case

A stale test case is a test case that has not been run for at least 2 months.

In real-time, the 2 month-point is calculated every time you click to view the report.

Status (Release)

A status in Release indicates the status of a release including:

• Ready: all test cases have passed.
• Not Ready: at least one test case has failed.
• Empty: there's no test case linked to the release.

Status (Session)

A status in Session is the status of a session, including:

• Queued: session has been created, waiting to be executed.
• Running: session is in progress.
• Failed: session has failed.
• Success: session has succeeded.
• Canceled: session is canceled manually or session timeout.

Status (Test Case)

A status in Test Case is the status of a test case, including:

• Passed: all test results have passed.
• Failed: one of the test results has not passed.

The status of a test case is defined by its latest execution.

Status (Test Result)

A status in Test Result is the status of a test result, including:

• Passed: test case runs successfully.
• Failed: test case runs unsuccessfully.
• Error: an error occurs during the execution.
• Incomplete: test case predefined in Katalon Studio.

Status (Test Run)

A status in Test Run is the status of a test run, including:

• Passed: all test results have passed.
• Failed: one of the test results has failed.

Status (Test Suite)

A status in Test Suite is the status of a test suite, including:

• Passed: all test cases have passed.
• Failed: one of the test cases has failed.

Status (Visual Checkpoint)

A status in Visual Checkpoint is the status of a visual checkpoint, including:

• Pass: checkpoint image has matched the baseline, or it's manually marked as Pass.
• Fail: checkpoint image is marked manually as Fail.
• Unresolved: checkpoint image has mismatched the baseline. You can compare with the baseline image, then mark it as Pass or Fail.

Status (Visual Test Run)

A status in Visual Test Run is the status of a visual test run, including:

• Pass: all new checkpoint images match the baseline image.
• Fail: one or more checkpoints have failed but no unresolved checkpoints.
• Unresolved: one or more checkpoints are unresolved.

Test Progress (Dashboard)

= (Total passed test results in release/Total test results in release) x 100%

Test Run Coverage

A test run coverage shows the quality of each requirement based on the status of the corresponding test result.

• The size of each block indicates the number of test runs associated with each requirement.

• The color of each block indicates the overall status of the total test cases in each requirement.

Time (Test Suite)

Time in Test Suite represents the duration of the last run of a test suite. It also indicates the start time of the last run of a test suite.

Time (Visual Test Run)

= (End time - Start time of a visual test run)

Total Duration in Summary (Test Results)

The total duration represents a sum of all test result durations in a day.

Total Duration (Release)

The total duration in Release represents the total duration of all test runs in a release.