Visual regression testing for fun, profit, and peace of mind
A friend of mine recently related a scary story about the lack of automated visual regression testing where he works—a huge international tech firm that you’ve heard of and probably use. He’d added a CSS class to an element to hide it from the user. Unbeknownst to him, and despite using BEM-style naming for CSS classes, this class name was already being used for an important checkbox in the user settings screen of his company’s web app. The result? Users could no longer change the setting represented by the checkbox, because it was invisible!
I asked him why that hadn’t been caught by automated tests. He explained that, although they did have end-to-end tests in place that went through the UI and tested its functionality, the tests didn’t catch the bug. Selenium was still able to check the original checkbox via its selector, so its visibility had no effect on the outcome of the test. The only way this could have been caught is via visual regression testing — or, as it is sometimes called, screenshot testing.
💅 Switching to styled-components
I recalled this story recently while working on a sizable project at Clue: converting all of our “native” CSS to use styled-components. The new helloclue.com website consists of dozens of components. An article page, for example, contains at least ten different React components, all of which have accompanying CSS files. Converting all of this CSS to styled-components means there’s an enormous risk of bugs exactly like the one my friend experienced. So before the project began, I investigated setting up screenshot testing for the site.
📕 Enter Storybook
(Note: if you’re already familiar with Storybook and @storybook/react, feel free to skip to the next section.)
We use Storybook when developing simple presentational components. This way, we can test them in every possible state, without having to reproduce all the logic and so forth required to get them to that state.
For example, on our author pages (the AuthorPage component), there are four possible states:
- Both the author and the author’s articles are loading.
- The author is loaded, but the articles are still loading.
- The articles are loaded, but the author is still loading.
- Both the author and the articles are loaded.
The loading states are reflected in the UI via placeholder elements. The problem is, how do I make sure these elements look right? Using our office’s speedy internet connection, I only get a split second to check the loading state of the author page before the placeholders are replaced with real content.
That’s where Storybook comes in. With Storybook, you can render a component in the specific state you want to test it in, and keep it in that state for as long as you need¹:
// stories/AuthorPage.tsx
// Note that the code samples in this article are written in TypeScript.import { storiesOf } from "@storybook/react"
import AuthorPage from "../components/AuthorPage/AuthorPage"storiesOf("AuthorPage", module)
.add("loading", () => (
<AuthorPage
isLoadingArticles={true}
isLoadingAuthor={true}
/>
))
In the example above, I’ve created a “story” for the author page in which both the articles and the authors are still loading. I can then run yarn run storybook (or npm run storybook), and it starts a Storybook server where I can view the AuthorPage component in its loading state. (Of course, I created additional stories for each of the other states, as well.)
While viewing the component, I can use Chrome Dev Tools to inspect the elements on the page, debug UI issues, and generally have a stable environment in which to develop the loading state of the component.
📸 But what about screenshots?
Since each component can be isolated and locked into a specific state in Storybook, and I wanted to screenshot every possible state of many of the site’s components, our Storybook setup seemed like a great place to do screenshot testing.
A bit of searching turned up an excellent library called storybook-chrome-screenshot, a Storybook addon. Using Storybook Chrome Screenshot, I can add a decorator to my Storybook stories:
// stories/AuthorPage.tsximport { initScreenshot, withScreenshot } from "storybook-chrome-screenshot/lib"
import { addDecorator, storiesOf } from "@storybook/react"import AuthorPage from "../components/AuthorPage/AuthorPage"addDecorator(initScreenshot())storiesOf("AuthorPage", module)
.add("loading", withScreenshot()(() => (
<AuthorPage
isLoadingArticles={true}
isLoadingAuthor={true}
/>
)))
When I run storybook-chrome-screenshot -p 9001 -c .storybook, a Storybook server is once again spun up. But this time, the Storybook Chrome Screenshot addon goes through each story that I’ve added the withScreenshot decorator to and takes a screenshot of it. (The decorator can also be added to the entire Storybook setup to automatically screenshot every story.²)
I can then add the storybook-chrome-screenshot command to the scripts key in my package.json, and automatically run it as a part of my CI process.
🔬 Comparing screenshots: before and after
OK, so I have screenshots of every component in the site in every possible state. But the whole point of taking screenshots is to compare them against master, to make sure nothing unintentionally changed.
That’s where reg-suit comes in. It’s an NPM package that does the actual work of pixel-by-pixel visual regression testing, and then generates an HTML report of the results.
First, I created regconfig.json in our app’s root directory (with a few non-JSON compliant comments added for your benefit):
{
"core": {
// The directory where storybook-chrome-screenshot dumps its screenshots, which reg-suit will use to compare against screenshots from master.
"actualDir": "__screenshots__",
// The directory where reg-suit will dump its HTML report, as well as images showing the visual differences if there are any.
"workingDir": "__screenshots_tmp__",
// This determines how forgiving reg-suit should be of differences between screenshots.
"threshold": 0,
"addIgnore": true,
"thresholdRate": 0,
"ximgdiff": {
"invocationType": "client"
}
},
"plugins": {
// Use CI environment variables to determine which commits to compare screenshots from (in this case, master vs. HEAD).
"reg-simple-keygen-plugin": {
"expectedKey": "${TARGET_GIT_COMMIT}",
"actualKey": "${GIT_COMMIT}"
},
// Notify GitHub of the results of the visual regression test.
"reg-notify-github-plugin": true,
// Publish screenshots and test results to S3. Note that your CI will need to have AWS credentials configured for this to work.
"reg-publish-s3-plugin": {
"bucketName": "<BUCKET_NAME>"
}
}
}Given the above config, reg-suit stores the screenshots for each commit in an S3 bucket, under a directory named for the Git SHA of that commit. It also stores master screenshots in the same bucket under its own Git SHA directory. It then uses the reg-simple-keygen-plugin to identify those directories and run comparisons between the screenshots in each.
🍱 Putting it all together
I mentioned earlier that the impetus for screenshot testing of helloclue.com was the conversion of CSS files to styled-components. Once we got the setup described above working on helloclue.com, I started the conversion process.
What was so cool about having automated screenshot tests during this process was that I could make tons of changes to the styling of components without worrying too much about making mistakes, since I knew they’d be caught.
And of course, I did make mistakes! While converting the author page to styled-components, for example, I mistakenly left out the styling that made the placeholder for the author’s title appear as a gray bar. Since our visual regression tests are integrated with GitHub, reg-suit commented on my PR to inform me that some visual comparisons had failed³: they didn’t match what was on master. My visual regression testing was actually working!
🎁 Fin
There are two things that I hope are clear from this article. First, visual regression testing is important! It can help you catch major UI bugs that your end-to-end tests missed. And second, there are tools available to make this easy in React. It’s just a matter of putting them together to make them work for you!
I’d love to hear from you in the comments:
- Was there anything that could be made clearer about the setup steps in this article? I’m happy to explain in the comments, or even to edit this article to make it easier to understand.
- What other tools, if any, are you using right now to do visual regression testing—particularly for React?
- Any other thoughts or feedback?
¹ Note that this works best with presentational components. Our presentational components simply take properties (such as isLoading) and output DOM. API calls, timeouts, etc. are all handled in container components. If you’d like to read more about separating business logic from presentation logic in React, I highly recommend Dan Abramov’s excellent article on the topic.
² Our actual screenshot configuration (with comments added for your benefit) looks like this:
// stories/index.tsximport { addDecorator } from "@storybook/react"
import { initScreenshot, withScreenshot } from "storybook-chrome-screenshot/lib"addDecorator(initScreenshot())// Rather than wrapping an individual story in `withScreenshot()(...)`, we'll add a decorator to the entire Storybook instance. This way, it'll take screenshots of every single story.
addDecorator(withScreenshot({
// A one-second delay ensures that fonts load before screenshots are taken.
delay: 1000, // We take screenshots at multiple viewport sizes, to ensure that various media queries are covered.
viewport: [
{
width: 320,
height: 568,
isMobile: true,
hasTouch: true,
},
{
width: 768,
height: 1024,
isMobile: true,
hasTouch: true,
},
{
width: 1024,
height: 768,
isMobile: true,
hasTouch: true,
},
{
width: 1280,
height: 800,
},
{
width: 1440,
height: 900,
},
],
}))import "./AuthorPage.tsx"
// [import all other stories...]
³ It’s worth noting that we use a custom GitHub integration which doesn’t actually fail the build when screenshot comparisons “fail.” This is because changes to the UI are often intentional. Instead of failing the build, our GitHub integration simply comments on the PR with a count of how many screenshots changed from master to the PR. If that count is greater than 0, I can manually review the visual regression report and determine whether or not all the changes in the PR were intentional.
FAQs
Does React testing library have snapshot? ›
Snapshot testing
It works well with React components because when you render a component you can view the DOM output and create a “snapshot” at the time of run.
Snapshot tests compare the rendered markup of every story against known baselines. It's a way to identify markup changes that trigger rendering errors and warnings. Storybook is a helpful tool for snapshot testing because every story is essentially a test specification.
Should I use snapshot testing React? ›Snapshot tests are useful when you want to make sure your UI does not change unexpectedly. A typical snapshot test case renders a UI component, takes a snapshot, then compares it to a reference snapshot file stored alongside the test.
Is Storybook good for testing? ›Storybook provides a clean-room environment for testing components in isolation. Stories make it easy to explore a component in all its variations, no matter how complex. That means stories are a pragmatic starting point for your UI testing strategy.
Which library is used for snapshot testing? ›However, snapshots can capture any serializable value and should be used anytime the goal is testing whether the output is correct. The Jest repository contains many examples of testing the output of Jest itself, the output of Jest's assertion library as well as log messages from various parts of the Jest codebase.
How do I create a snapshot using React testing library? ›The first time we run this test, toMatchSnapshot() will create a snapshot of the React tree. This snapshot will be saved in a folder called “__snapshots__” in the same folder as the test file. If no folder already exists, a folder will be automatically created. const { asFragment } = render(<Header text='Hello!
What is called Snapshot? ›A snapshot is a photograph that is "shot" spontaneously and quickly, most often without artistic or journalistic intent and usually made with a relatively cheap and compact camera.
What is snapshot Altus? ›Snapshot by Altus is a short, one-way video response tool, meant to provide an additional opportunity for schools to get to know you. Along with the CASPer test and the Duet, Snapshot is a part of the Altus Suite, a multi-level assessment tool used by professional programs.
What is snapshot testing in Android? ›How does it work? screenshot-tests-for-android generates deterministic screenshots of views during a test run. By deterministic, we mean that every single run of your tests generates a pixel-perfect screenshot of the app as it would appear on a user's device.
What is snapshot test in react? ›Snapshot tests assert that the current output is same as the output before. The main difference between snapshot testing and functional/unit tests is, snapshot tests never assert the correct behavior of the application functionality but does an output comparison instead.
Should I commit Jest snapshots? ›
The snapshot artifact should be committed alongside code changes. Jest uses pretty-format to make snapshots human-readable during code review. On subsequent test runs Jest will simply compare the rendered output with the previous snapshot. If they match, the test will pass.
How do I view Jest snapshots? ›To review your snapshots, run npm run jest-html ( yarn run jest-html ). This launches your default browser and opens the jest-html application. By default, jest-html looks for snapshots under **/*. snap,!
Is Storybook a testing library? ›Storybook provides an instrumented version of testing library in the @storybook/testing-library package. When writing interactions, make sure to use the helper functions from @storybook/testing-library , so that addon-interactions can intercept these helper functions and allow you to step through them when debugging.
Is storybook unit test? ›Storybook allows us to view and interact with our components in an isolated manner. It's just like unit testing but for UI components. According to Storybook's documentation, Storybook is a user interface development environment and playground for UI components.
How do you test a component in storybook? ›Start by writing a story to set up the component's initial state. Then simulate user behavior such as clicks and form entries using the play function. Finally, use the Storybook test-runner to check whether the UI and component state update correctly. Automate testing via the command line or your CI server.
Is Jest and React testing library same? ›Jest provides a great iteration speed combined with powerful features like mocking modules and timers so you can have more control over how the code executes. React Testing Library is a set of helpers that let you test React components without relying on their implementation details.
How do I run test library in React? ›- Step 1: Install. npm install --save-dev @testing-library/react jest @types/jest prettier. ...
- Step 2: Add babelrc file. ...
- Step 3: Add Test and Run.
In these docs we'll demonstrate configuring Jest, but you should be able to do similar things with any testing framework (React Testing Library does not require that you use Jest).
How do you write snapshot Jest? ›When writing snapshot tests for a React component, you first need to have code in a working state. Then, generate a snapshot of its expected output given certain data. The snapshot tests are committed alongside the component. Jest, a testing framework, will compare the snapshot to the rendered output for the test.
What is Snapshotserializers? ›A serializer is an object with two methods test and print . When Jest serializes a value it will try to match against the test function. If it matches, it will serialize the value using print . print also receives a serialization function to allow recursive serialization, and an indentation function.
How do you test for useEffect? ›
To test the component update useEffect hook you'd simply trigger state updates and check for effects in rendered elements. Redux hooks can be tested by mocking them and their implementation.
Why do we need snapshots? ›Usually, snapshots are used to test software updates or for unsafe operations on a VM, and then returned to the initial state if needed — think about it as a bookmark or an undo button. Snapshots are not a full copy of the base disk, therefore, they are not sufficient to restore a VM in case of storage failure.
What is the purpose of snapshot? ›Snapshots are generally created for data protection, but they can also be used for testing application software and data mining. A storage snapshot can be used for disaster recovery (DR) when information is lost due to human error.
What is snapshot in coding? ›A snapshot code transforms a smart code into a regular code. This code preserves the current state of a smart code at the moment of its creation, like taking a photograph that captures the moment, and creates direct links to all quotations. A snapshot code is displayed in the margin area.
Can you practice for snapshot? ›You can practice as many times as you like or only once. Once you've practiced, you can begin your Snapshot interview. Before the recording for the real interview begins, you'll be able to read the first question and have 30 seconds to reflect on your response.
Can you take snapshot after 14 days? ›Do I have to take all parts of Altus Suite at once? No. While you have to take the Casper test at a specific time, Snapshot and Duet can be completed anytime after booking your Casper test and no later than 14 days after taking the Casper test.
Is snapshot required for Casper? ›Snapshot and Duet are not required. Casper is optional but encouraged.
What is snapshot testing IOS? ›Snapshot testing involves taking a screenshot of your user interface and storing it as a reference image. Then, doing tests to take screenshots of the same UI and comparing the layout to the reference images pixel by pixel.
What is snapshot testing used in React? ›- [Instructor] Snapshot tests are useful for making sure that the UI will not change unexpectedly. An additional Facebook package needs to be added to the project via npm to help us do that. It's called react-test-renderer, and it allows us to render React components as pure JavaScript objects.
What is snapshot in React JS? ›Snapshot testing has been created due to the need for an easier way to write tests for React components. Many React developers reported that they spend more time writing tests than the actual component. Therefore, snapshot testing allows React developers to quickly generate tests using its simple syntax.
What is snapshot in testing? ›
In snapshot testing, the output of a function is saved in a file (the “snapshot”), and when the test runs, it compares this saved output with the output of the function when it is run each time in the test suite.
Is Jest and React testing library same? ›Jest provides a great iteration speed combined with powerful features like mocking modules and timers so you can have more control over how the code executes. React Testing Library is a set of helpers that let you test React components without relying on their implementation details.
What is snapshot testing IOS? ›Snapshot testing involves taking a screenshot of your user interface and storing it as a reference image. Then, doing tests to take screenshots of the same UI and comparing the layout to the reference images pixel by pixel.
What is Snapshotserializers? ›A serializer is an object with two methods test and print . When Jest serializes a value it will try to match against the test function. If it matches, it will serialize the value using print . print also receives a serialization function to allow recursive serialization, and an indentation function.
What is reconciliation in React? ›The “reconciliation” algorithm in React is how the decision to re-render the component is made. In the browser, DOM manipulation is expensive and time consuming, both in mounting and unmounting. Part of what makes React very performant is its reconciliation algorithm.
Are snapshot tests unit tests? ›The main difference between snapshot testing and functional/unit tests is, snapshot tests never assert the correct behavior of the application functionality but does an output comparison instead. There are two types of tools that support frontend snapshot testing.
How do you write a test case for react? ›- Step 1: Create a new react app.
- Step 2: Create a component.
- Step 3: Write a unit test for the react component.
- Note: In order to let jest know about this test file, it's important to use the extension . test. js.
- Step 4: Run the test.
- Conclusion.
To test the component update useEffect hook you'd simply trigger state updates and check for effects in rendered elements. Redux hooks can be tested by mocking them and their implementation.
What is the purpose of snapshot? ›Snapshots are generally created for data protection, but they can also be used for testing application software and data mining. A storage snapshot can be used for disaster recovery (DR) when information is lost due to human error.
How do I update snapshot Jest React? ›- Run the snapshot test, make sure it passes — if it does not pass, find the problem and fix it, don't change the UI until the test passes.
- Change the UI.
How do you debug a Jest test? ›
- Open the unit test file you want to debug.
- Set breakpoints or the debugger statement where you want to stop.
- Press Ctrl + Shift + D , or click on the Debug icon in the left panel.
- Select DEBUG ‣ Jest: current file option in the top panel.
- Press F5 to start debugging.
React Testing Library is not specific to any testing framework; we can use it with any other testing library, although Jest is recommended and preferred by many developers.
What is E2E testing in React? ›End-to-end tests simulate actual user actions and are designed to test how a real user would likely use the application. In React, E2E testing helps to ensure that the code you wrote is functional and your app works as intended, allowing you to catch bugs in your code before your app is live.
Which is best React testing library or Enzyme? ›If you want mimic real-world user interactions, the React Testing Library is the way to go because you can do the same with fireEvent functions. Meanwhile, Enzyme is better suited to situations where you have to match the state of React or some other function with state.