Automated Visual Regression Testing With Playwright

Automated visual testing offers a powerful, albeit sometimes unpredictable, approach to software quality assurance. Playwright simplifies this process for websites, though fine-tuning might be necessary.

Recent system downtime spurred me to address a long-standing issue: the CSS of a website I maintain had become unwieldy due to continuous feature additions. With a clearer understanding of requirements, it's time for internal CSS refactoring to reduce technical debt and leverage modern CSS features (like nested selectors for improved structure). A cleaner codebase will also facilitate the addition of a much-needed dark mode, enhancing user experience by respecting preferred color schemes.

However, hesitant to introduce bugs through significant changes, I needed a robust visual regression testing solution. Traditional snapshot testing proved too slow and fragile.

Snapshot testing, in this context, involves capturing screenshots to establish a baseline for comparison with future results. These screenshots, however, are susceptible to various uncontrollable factors (e.g., timing issues, varying hardware resources, or dynamic content). Managing the state between test runs (saving screenshots) adds complexity and makes it difficult for the test code alone to fully define expectations.

After prolonged procrastination, I finally implemented a solution, initially expecting a quick fix. This wouldn't be part of the main test suite; just a temporary tool for this refactoring task.

Fortunately, I recalled previous research and quickly found Playwright's built-in visual comparison capabilities. Appreciating Playwright's minimal external dependency reliance, I proceeded with the implementation.

Setting Up the Environment

While npm init playwright@latest provides a good starting point, I opted for a manual setup to better understand the components. Given the infrequent use of snapshot testing, I created a dedicated subdirectory, test/visual, as our working directory. The package.json file declares dependencies and helper scripts:

{
  "scripts": {
    "test": "playwright test",
    "report": "playwright show-report",
    "update": "playwright test --update-snapshots",
    "reset": "rm -r ./playwright-report ./test-results ./viz.test.js-snapshots || true"
  },
  "devDependencies": {
    "@playwright/test": "^1.49.1"
  }
}

To avoid cluttering the main project with this rarely-used dependency, consider running npm install --no-save @playwright/test directly in the root directory when needed.

After installing the necessary packages (npm install followed by npx playwright install), we configure the test environment using playwright.config.js:

import { defineConfig, devices } from "@playwright/test";

let BROWSERS = ["Desktop Firefox", "Desktop Chrome", "Desktop Safari"];
let BASE_URL = "http://localhost:8000";
let SERVER = "cd ../../dist && python3 -m http.server";

let IS_CI = !!process.env.CI;

export default defineConfig({
  testDir: "./",
  fullyParallel: true,
  forbidOnly: IS_CI,
  retries: 2,
  workers: IS_CI ? 1 : undefined,
  reporter: "html",
  webServer: {
    command: SERVER,
    url: BASE_URL,
    reuseExistingServer: !IS_CI
  },
  use: {
    baseURL: BASE_URL,
    trace: "on-first-retry"
  },
  projects: BROWSERS.map(ua => ({
    name: ua.toLowerCase().replaceAll(" ", "-"),
    use: { ...devices[ua] }
  }))
});

This configuration assumes a static website in the dist directory, served locally at localhost:8000 (using Python's http.server). The number of browsers can be adjusted for performance. The IS_CI logic is omitted for this specific, non-CI scenario.

Capturing and Comparing Visuals

A minimal test in sample.test.js demonstrates the core functionality:

import { test, expect } from "@playwright/test";

test("home page", async ({ page }) => {
  await page.goto("/");
  await expect(page).toHaveScreenshot();
});

The first run generates baseline snapshots. Subsequent runs compare against these, reporting failures if visual differences are detected. Playwright provides tools to visually compare snapshots and inspect baseline images.

Automated Test Generation

To avoid manual test creation for each page, I implemented a simple web crawler to generate tests dynamically.

Site Map Creation

The playwright.config.js file is extended to include globalSetup and export relevant configuration values:

export let BROWSERS = ["Desktop Firefox", "Desktop Chrome", "Desktop Safari"];
export let BASE_URL = "http://localhost:8000";

// ...rest of the config...

export default defineConfig({
  // ...rest of the config...
  globalSetup: require.resolve("./setup.js")
});

The setup.js file uses a headless browser to crawl the site and create a sitemap:

import { BASE_URL, BROWSERS } from "./playwright.config.js";
import { createSiteMap, readSiteMap } from "./sitemap.js";
import playwright from "@playwright/test";

export default async function globalSetup(config) {
  // only create site map if it doesn't already exist
  try {
    readSiteMap();
    return;
  } catch(err) {}

  // launch browser and initiate crawler
  let browser = playwright.devices[BROWSERS[0]].defaultBrowserType;
  browser = await playwright[browser].launch();
  let page = await browser.newPage();
  await createSiteMap(BASE_URL, page);
  await browser.close();
}

The sitemap.js file contains the crawler logic:

import { readFileSync, writeFileSync } from "node:fs";
import { join } from "node:path";

let ENTRY_POINT = "/topics";
let SITEMAP = join(__dirname, "./sitemap.json");

export async function createSiteMap(baseURL, page) {
  await page.goto(baseURL   ENTRY_POINT);
  let urls = await page.evaluate(extractLocalLinks, baseURL);
  let data = JSON.stringify(urls, null, 4);
  writeFileSync(SITEMAP, data, { encoding: "utf-8" });
}

export function readSiteMap() {
  try {
    var data = readFileSync(SITEMAP, { encoding: "utf-8" });
  } catch(err) {
    if(err.code === "ENOENT") {
      throw new Error("missing site map");
    }
    throw err;
  }
  return JSON.parse(data);
}

function extractLocalLinks(baseURL) {
  let urls = new Set();
  let offset = baseURL.length;
  for(let { href } of document.links) {
    if(href.startsWith(baseURL)) {
      let path = href.slice(offset);
      urls.add(path);
    }
  }
  return Array.from(urls);
}

This uses page.evaluate to extract links within the browser context.

Dynamic Test Generation

The viz.test.js file dynamically generates tests based on the sitemap:

import { readSiteMap } from "./sitemap.js";
import { test, expect } from "@playwright/test";
import { join } from "node:path";

let OPTIONS = {
  stylePath: join(__dirname, "./viz.tweaks.css")
};

let sitemap = [];
try {
  sitemap = readSiteMap();
} catch(err) {
  test("site map", ({ page }) => {
    throw new Error("missing site map");
  });
}

for(let url of sitemap) {
  test(`page at ${url}`, async ({ page }) => {
    await page.goto(url);
    await expect(page).toHaveScreenshot(OPTIONS);
  });
}

readSiteMap needs to be synchronous due to Playwright's current limitations on top-level await in test files. The OPTIONS object allows for custom CSS tweaks.

Handling Exceptions and Flakiness

Visual testing requires handling exceptions. Custom CSS in viz.tweaks.css can suppress specific elements affecting consistency:

/* suppress state */
main a:visited {
  color: var(--color-link);
}

/* suppress randomness */
iframe[src$="/articles/signals-reactivity/demo.html"] {
  visibility: hidden;
}

/* suppress flakiness */
body:has(h1 a[href="/wip/unicode-symbols/"]) {
  main tbody > tr:last-child > td:first-child {
    font-size: 0;
    visibility: hidden;
  }
}

Addressing Viewport Limitations

Initially, the tests failed to detect styling changes because .toHaveScreenshot only captures the viewport. To capture the full page, playwright.config.js is modified:

export let WIDTH = 800;
export let HEIGHT = WIDTH;

// ...rest of the config...

projects: BROWSERS.map(ua => ({
  // ...rest of the config...
  use: {
    // ...rest of the config...
    viewport: {
      width: WIDTH,
      height: HEIGHT
    }
  }
}))

And viz.test.js is updated to determine and set the page height:

import { WIDTH, HEIGHT } from "./playwright.config.js";

// ...rest of the imports...

for(let url of sitemap) {
  test(`page at ${url}`, async ({ page }) => {
    checkSnapshot(url, page);
  });
}

async function checkSnapshot(url, page) {
  await page.setViewportSize({ width: WIDTH, height: HEIGHT });
  await page.goto(url);
  await page.waitForLoadState("networkidle");
  let height = await page.evaluate(getFullHeight);
  await page.setViewportSize({ width: WIDTH, height: Math.ceil(height) });
  await page.waitForLoadState("networkidle");
  await expect(page).toHaveScreenshot(OPTIONS);
}

function getFullHeight() {
  return document.documentElement.getBoundingClientRect().height;
}

This ensures the entire page is captured, though it increases resource consumption and might introduce flakiness due to layout shifts or timeouts.

Conclusion

This visual regression testing solution, while more complex than initially anticipated, effectively addresses the problem. The refactoring and subsequent dark mode implementation remain.

The above is the detailed content of Automated Visual Regression Testing With Playwright. For more information, please follow other related articles on the PHP Chinese website!