How to test flaky tests in Playwright using command line

Flaky tests often stem from race conditions, timeouts, or environmental differences. These CLI flags help by repeating tests to expose inconsistencies, isolating changed tests to focus debugging, and retrying failures to confirm flakiness. Debugging tools and CPU throttling further aid in reproducing and fixing issues locally.

In Playwright (a Node.js library for browser automation), the flags --repeat-each and --only-changed are used with the Playwright Test Runner (npx playwright test) to control how tests are run. Here's what each one does:

--repeat-each <N>

This flag repeats each test a specified number of times. If You want to verify the stability of your tests by running them multiple times.

npx playwright test test.spec.js --repeat-each 5

This command runs each individual test 5 times, even if they pass each time.

1. Stress Test with More Workers

Sometimes, running the test repeatedly isn’t enough, especially if the failure is triggered by CPU or memory constraints on CI. In this case, you can stress your system by maximizing the number of workers:

npx playwright test test.spec.js --repeat-each 100 --workers=10

2. Stop on First Failure

Running a test 100 times can take a while, but you don’t have to wait for all runs to complete. Use the -x flag to stop execution as soon as a failure occurs:

npx playwright test test.spec.js --repeat-each 100 -x

This will save you time by immediately identifying when a test starts failing under stress.

--only-changed

This flag runs only the tests related to changed files (based on git changes). If You want to save time during development by only running tests related to files you've recently modified.

npx playwright test test.spec.js --only-changed

This command will:

1. Detect changed test files or changed files they depend on.
2. Run only the affected tests.

Note: This requires that your project is in a Git repository, and Playwright uses Git to detect changes.

Also we can run only changed tests and repeat each one twice.

npx playwright test test.spec.js --only-changed --repeat-each 2

This is all about checking flaky test in playwright.