Use TestOps Visual Testing

You can compare images captured during test executions with TestOps Visual Testing.

Set up Visual Testing

To set up Visual Testing, follows these steps:

1. Enable screenshot capture in Katalon Studio. See Capture Screenshots.
   Note:

   • You can only apply visual testing for screenshots taken as checkpoints. For new users, we highly recommend using these keywords:

     • [WebUI] Take Screenshot As Checkpoint
     • [Mobile] Take Screenshot As Checkpoint

   • You can also use this Visual testing sample project on Katalon Studio for basic setup. To learn more about using the WebUI sample project, see: Sample WebUI tests project.

2. Run a test suite in Katalon Studio.
   Note:

   • If you have enabled Katalon Studio Integration, Katalon Studio automatically uploads Test Results to Katalon TestOps.

3. Sign in to Katalon TestOps and go to your project.
4. Go to Visual Testing > Visual Test Runs.
5. Click on the ID of a Visual Test Run.
   
   The Results page appears. You can see the screenshots (known as Visual Checkpoints) captured during a test execution.
6. Select one of the screenshots to see the details.

   • If you run a Test Suite for the first time, there is no baseline image.

   • If you have existing visual data, it is stored in the System-generated baseline collection folder.

You have successfully set up Visual Testing in Katalon TestOps.

Configure a baseline image

You can set up a baseline image to compare it with the screenshot of the next Test Run. Follow these steps:

1. Select a screenshot to see the details.
   
2. Select Mark as Passed at the top right corner of a screenshot, then close the screenshot.

   An Unsaved label now appears on the Results page.

   

3. Click Save to baseline.
   Note:

   • If you run a Test Suite for the first time and the Test Suite has passed, the screenshots are automatically marked as Passed and there is no Save to baseline button. You can open a screenshot and click Failed, then switch back to Passed so that the Save to baseline button appears.

Access a baseline collection

To access a baseline collection, follow these steps:

1. Sign in to Katalon TestOps and go to your project.
2. Go to Visual Testing > Visual Baseline Collections.
   You will see all baseline collection IDs and their details (latest version, number of the baseline images in each baseline collection, and the test schedule that links to them).
   
   
3. Click on the Baseline Collection ID you want to check.
   To customize the baseline collection, you can:

   1. Click on Pencil icon to edit the name of the baseline collection.
   2. Click History to see the older version of the baseline collection.
   3. Edit or delete each baseline image of this baseline collection.
   4. Select Pixel-based or Layout-based as the default comparison method for the baseline collection.
      Note:

      • This feature is available on Visual Testing Professional (VTP) package.
      • The default comparison method will determine the checkpoint status (such as Match or Mismatch) for all visual test runs that use the same baseline collection.

Unresolved images

If you run the Test Suite again and the new screenshots of this test execution are different from the baseline images, the status of this Visual Test Run is then marked as Unresolved. To resolve these images, follow these steps:

1. Click on the ID of the unresolved Visual Test Run.
2. Select the unresolved checkpoint.
   
   
   

   You see a drop-down list in the Comparison Method section. The three options include: Pixel, Layout, Text Content. For detailed information of each comparison method, see: TestOps Visual Testing.

3. Select one of the three options to start comparing the new screenshot (on the right) with the baseline image (on the left).
   Note:

   • Comparison Method is enabled only if the status of the screenshot is mismatched.
   • If you select Pixel, check the Diffs between two images (the differences are highlighted in red). See: Pixel-based comparison.
   • If you select Layout, check the Identical, Distinguishable and Missing/New boxes to see the differences highlighted in green/red/pink color respectively. See: Layout-based comparison.
   • If you select Text Content, check the Identical, Shifted and Missing/New boxes to see the differences highlighted in green/blue/pink color respectively. See: Content-based comparison.
4. Click Mark as Passed or Mark as Failed to resolve.
   Note:

   • The Unsaved label appears whenever you change the status (Passed or Failed) of a screenshot. If you click Save to baseline, the new Passed screenshot replace the older baseline image. Katalon TestOps then compares between the new baseline image and screenshots of the next Visual Test Run.

Ignored zones

Specifying ignored zones is especially useful when testing a website with some dynamic and changing content (such as pop-up ads on specific areas). Visual testing can ignore such content while comparing captured images with the baseline one.

In TestOps Visual Testing, you can configure an ignore zone in a baseline image and apply this ignored zone to all baseline images in the baseline collection. You can also delete the ignored zone(s) in all baseline images.

Add an ignored zone to a baseline image

To configure an ignored zone, follow these steps:

1. Sign in to Katalon TestOps and go to your project.
2. Go to Visual Testing > Visual Baseline Collections.
3. Select the Baseline Collection ID you want to check.
   The list of baseline images appears.

4. Click on the Edit image icon of the baseline image you want to configure (e.g., the baseline image image10).
   The page displays as below.
   
   
5. Click on the Configure Ignored Zones button.
   The button is then highlighted in blue, which indicates the edit mode.
   
   
   
6. Draw (in rectangles) to specify the area you want to ignore.
   The ignored zones are highlighted in purple. You can draw up to 20 ignored zones (i.e. 20 rectangles).

   Note:

   • If you want to apply an ignored zone to all baseline images, see: Apply an ignored zone to all baseline images.

7. Click Save.
   You have temporarily saved the ignored zones on this baseline image.
   
   
   

   Note:

   • You have not permanently saved the configuration of ignored zones at this stage.
8. Click Save Changes on the top right corner to permanently save the ignored zones you have configured.
   
   
   

The baseline image has now had ignored zones. Visual testing then ignores these zones when comparing captured images with this baseline image.

When comparing the checkpoint image with the baseline image, you can show or hide the ignored zones by ticking/un-ticking the Ignored Zones checkbox.

Apply an ignored zone to all baseline images

Adding an ignored zone to the same area in multiple baseline images could become time-consuming if there are more than one baseline image. With TestOps Visual Testing, you can apply an ignored zone to multiple baseline images in a baseline collection.

To apply an ignored zone to all baseline images within one baseline collection, follow these steps:

1. Configure an ignored zone. See: Add an ignored zone to a baseline image.
   The ignored zone is displayed in purple.
2. Click on the ignored zone, then select the Apply the zone to all images icon.
   
   You then see the Acknowledgement pop-up box.
3. Click Acknowledge.
   The ignored zone is now displayed in yellow because it has been applied to all other baseline images in the baseline collection.
4. Click Save to temporarily save the configuration.
5. Click Save Changes on the top right corner to permanently save the configuration.

You have applied an ignored zone to all baseline images within the selected baseline collection.

Delete ignored zones

You can delete an ignored zone in one baseline image and/or all baseline images in the baseline collection.

An ignored zone applied to one baseline image only is displayed in purple. To delete it, click on the ignored zone and select the Delete ignored zone icon.

An ignored zone applied to all baseline images is displayed in yellow. To delete it, follow these steps:

1. Click on the ignored zone and select the Delete ignored zone icon.
   
   The Delete all ignored zones box pops up.
2. Click Delete all zones to confirm your action.
3. Click Save to temporarily save the configuration.
4. Click Save Changes to permanently save this configuration.
   The ignored zone is now deleted in all baseline images of the baseline collection.