Retry Failed Test Cases in playwright

 Playwright is a powerful tool for automating browser testing, and one of its notable features is the ability to retry failed test cases. This blog will teach you how to use retry mechanism for failed test cases.

Why Use Test Retries?

Automated tests can fail for various reasons that are not necessarily related to the code itself, such as:

1. Flaky tests: Tests that pass or fail inconsistently due to timing issues or other unpredictable factors.
2. Network issues: Temporary connectivity problems can cause tests to fail.
3. Environmental factors: Changes in the testing environment can lead to failures.

By enabling retries, Playwright allows you to automatically re-run failed tests, reducing the need for manual intervention and providing a more robust testing framework.

Configuring Retries in Playwright

To retry failed test cases in Playwright, you can configure retries either globally in the configuration file or per test file or even per test using the test API. To set up retries in Playwright, you need to modify your configuration file (playwright.config.js). Here’s how you can do it:

playwright.config.ts or playwright.config.js

import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: 2, // retry failed tests up to 2 times
  use: {
    // other config options
  },
});

In this example, each failing test will be retried up to two times. If a test fails on the first attempt but passes on any of the retries, it will be categorized as a flaky test in the reports.

Running Tests with Retry Options

You can also specify retry attempts directly from the command line when executing your tests. For example:

npx playwright test --retries=2

This command will initiate the test suite with two retry attempts for any failing tests without needing to alter the configuration file.

Rerunning Only Failed Tests

Since Playwright v1.44, you can rerun only the tests that failed in the last run using the --last-failed flag:

npx playwright test --last-failed

How to check Retry Detection

You can detect if a test is being retried using the testInfo.retry property, which starts at 0 for the first run and increments for each retry:

import { test } from '@playwright/test';

test('example test', async ({ page }, testInfo) => {
  if (testInfo.retry) {
    console.log(`Retrying test, attempt #${testInfo.retry}`);
    // Optional: Clear caches or reset state before retry
  }
  // Test code
});

Retry Specific Test File or Suite

You can also override the retry setting per test file or test suite:

import { test } from '@playwright/test';

test.describe.configure({ retries: 2 });

test('flaky test', async ({ page }) => {
  // test logic
});

custom retry logic handler

For custom retry logic at the step level (e.g., retrying a button click), implement a try-catch loop:

async function clickWithRetry(page, selector, maxRetries = 3) {
  for (let attempt = 1; attempt <= maxRetries; attempt++) {
    try {
      await page.click(selector);
      return; // Success
    } catch (error) {
      if (attempt === maxRetries) throw error;
      console.warn(`Attempt ${attempt} failed, retrying...`);
      await page.waitForTimeout(1000 * Math.pow(2, attempt)); // Exponential backoff
    }
  }
}

Playwright categorizes test results based on their behavior during execution:

Passed: Tests that succeed on the first run.

Flaky: Tests that fail initially but succeed upon retry.

Failed: Tests that continue to fail after all retry attempts.

The retry mechanism in Playwright is an essential feature that enhances the stability and reliability of automated testing. By configuring retries effectively developers can significantly reduce false negatives in their test suites and improve overall confidence in their automated testing processes.