Download - Implementing a visual css testing framework
![Page 1: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/1.jpg)
IMPLEMENTING A VISUAL CSS TESTING FRAMEWORK
@jessicard
hi everyone! thanks for having me heretoday i’m going to talk about implementing a visual css testing framework
![Page 2: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/2.jpg)
using
AUTOMATICscreenshot
COMPARISONto catch style
REGRESSIONSusing automatic screenshot comparison to catch style regressions
![Page 3: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/3.jpg)
@jessicardmy name is jessicard on the internet, and jessica in real life
![Page 4: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/4.jpg)
i work at a company called bugsnag here in san franciscobugsnag is an exception monitoring tooli’m a software engineer there, working primarily in ruby and javascriptbut our stack includes lots of languages, including node and go
![Page 5: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/5.jpg)
(we’re hiring!)
we are currently hiringso please get in touch if you’re interested in working on developer tools at a small company!
![Page 6: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/6.jpg)
IMPLEMENTING A VISUAL CSS TESTING FRAMEWORK
so, back to this whole “implementing a visual css testing framework” thingwhat am i even talking about?
![Page 7: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/7.jpg)
well, at bugsnag, we decided we wanted a way to automatically take two screenshotsof our website, at different times in the app
![Page 8: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/8.jpg)
master feature
for example, since we use git, lets say we had a feature branch that we just pushed a commit to,we’d want to take a screenshot of what our homepage looks like on that branchand a screenshot of how our homepage looks on production, or whats currently running on master
![Page 9: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/9.jpg)
with those screenshots, we’d want to spot those differences
![Page 10: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/10.jpg)
and produce some sort of diff image that highlighted the differenceswhy would we want to do this, you might ask? well,
![Page 11: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/11.jpg)
as we all know, writing, reading, and code reviewing CSS can be really intense
![Page 12: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/12.jpg)
😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭
😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭
😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭
😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭
😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭
😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭
😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭
😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭
😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭
😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭
😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭
😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭and even more intense to refactorat bugsnag, we decided to do a huge organizational and code-style refactor of our CSSand we wanted a way to test that our site looked the same, despite changing all of the code
![Page 13: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/13.jpg)
unfortunately, as you can tell, that didn’t always work out for uswe went through many iterations of refactoring,and we realized we needed a tool to help us test our pages automatically
![Page 14: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/14.jpg)
does thisEXIST?
so, we went on the hunt for a way to test our CSSwe wanted to know if there was a tool already built that did what we wantedwithout knowing exactly what it was that we wanted out of this framework
![Page 15: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/15.jpg)
huxley
we first stumbled upon one of facebook’s open source libraries, huxley
![Page 16: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/16.jpg)
“Watches you browse, takes screenshots, tells you when they change”
in huxley’s readme, it says“watches you browse, takes screenshots, tells you when they change”which sounds amazing!
![Page 17: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/17.jpg)
but… i had noticed that it hadn’t been updated in over a yearthat wasn’t promising, but i decided to give it a shot anyway
![Page 18: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/18.jpg)
👎after a good solid day of fiddling around with huxley, it ended up being a bit too buggyit would have random failures, randomly wouldn’t take screenshotsand i started realizing it wasn’t exactly what i was looking for in a css testing tool
![Page 19: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/19.jpg)
Huxley Phantom CSS GhostStory
Cactus Needle
CSSCritic fighting-layout-bugs
sikuli Mogo
Quixotei kept looking, and it turns out there are a lot frameworks that do some sort of css testingthey work in all different ways - some take screenshots, some aren’t even visuali’d definitely recommend checking some of these out to see if they fit what you’re doing in your apps before building your own
![Page 20: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/20.jpg)
WHAT DO I WANT?
but i really started thinking about itwhat was i looking for in a CSS testing framework?what would fit the way bugsnag is built best?i decided i wanted a visual way to test my CSS, with screenshotsrather than writing out tests describing the visuals
![Page 21: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/21.jpg)
this is where i need a disclaimerat bugsnag, our web dashboard is written in railsthis, mixed with the fact i wanted the tests to take screenshots,affected my decisions in what frameworks i wanted to use
![Page 22: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/22.jpg)
we also use git for our source control at bugsnag, and we use it the “github” way
![Page 23: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/23.jpg)
master
what that means is, we have our master branchwhich is always deployable and production ready(or at least it should be)
![Page 24: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/24.jpg)
master feature
and whenever we want to create a feature,we branch off of master until it’s ready, and then we merge it back in
![Page 25: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/25.jpg)
considering the tools we had at our disposal, and after taking a look at some of the screenshotting framework’s source codes,i realized that there wasn’t actually that much code to a lot of the screenshot librariesso I decided why not, i’m just gonna write one myself
![Page 26: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/26.jpg)
PROCESSi came up with a process of how i thought i wanted my tests to work
![Page 27: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/27.jpg)
VISIT1.number one, i wanted a way to somehow automatically visit pages of our site
![Page 28: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/28.jpg)
so the test would, in an actual browser, hit each page of our site on a local server
![Page 29: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/29.jpg)
SCREENSHOT2.
once the test visited the page, i wanted the test to take a screenshot of that page
![Page 30: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/30.jpg)
the important bit for the screenshotis that i’d want it to take a screenshot of the entire page
![Page 31: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/31.jpg)
not just the current viewport of the browser
![Page 32: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/32.jpg)
for example, if a change happened below the fold of our site, but we weren’t taking full page screenshotswe wouldn’t be able to capture that diff
![Page 33: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/33.jpg)
UPLOAD3.next, i’d need somewhere to store these screenshots,and i’d need a way to upload and download these screenshots from that storage area
![Page 34: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/34.jpg)
upload master
screenshot
so, using git, every time i made a push to a branch, i’d upload a screenshot of the current state of each pageincluding our master branch
![Page 35: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/35.jpg)
download master
screenshot
upload feature
screenshot
and, if i had a screenshot already uploaded to our storage area from our master branch,i’d need a way to upload my current features branch to that storage area,and download my already uploaded master screenshot from there
![Page 36: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/36.jpg)
DIFF4.i’d then need a way to make a diff of my screenshots
![Page 37: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/37.jpg)
master feature
i’d want to diff between a screenshot i took on master, downloaded from our storage area,and the newest screenshot i took, on my feature branch
![Page 38: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/38.jpg)
so if i have the previous screenshot i tookwhich would be the current commit on mastervs my branch’s screenshot
![Page 39: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/39.jpg)
and i’d need a way to mark the differences between these two screenshots visually
![Page 40: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/40.jpg)
VIEW5.finally, after i have diffs for my screenshots, i’d need a way to view the diffseven though i’ll have a place to upload them, i need an accessible way for everyone on the project to view the diff screenshots depending on the commit
![Page 41: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/41.jpg)
BUILDINGour framework
so now that we have a plan, we can start building our framework around these things that we need
![Page 42: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/42.jpg)
TESTINGwith rspec
we need a way to write some tests that will run automatically after each pushso we decided to use rspec
![Page 43: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/43.jpg)
rspec
rspec is a testing tool for the ruby programming languagewe already had rspec for tests our rails app, so writing these tests with rspec made sense
![Page 44: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/44.jpg)
we wanted to be able to write specs that looked like thiswhere we would just be able to navigate to a local URL
![Page 45: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/45.jpg)
and save a screenshot of that pagewe also wanted these tests to be separate of our main tests
![Page 46: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/46.jpg)
we also wanted these tests to be separate of our main tests
![Page 47: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/47.jpg)
we pulled out these visual specs into their own rspec tagthat way, these specs wouldn’t run with our main specs when we were running the suite locally, unless we explicitly wanted them tothis also made it so we could break out our specs on our CI
![Page 48: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/48.jpg)
LOCAL speed
we want our visual specs to be separate for a few reasonsnumber one, our local build speedif our local tests were bogged down by waiting for visual specs, that would become a huge issueby having them broken out, we could iterate on our main app specs faster and be able to push more often
![Page 49: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/49.jpg)
CI speed
number two, is speed with our CI testswe wanted our main specs to still be fast on our CI, so that we can merge non-visual pull requests without waiting for our visual specs to finish
![Page 50: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/50.jpg)
CI, or continuous integration, is a way for us to automatically run our test suite when we push things to githuband after our specs run on our CI, we are able to see if our build is passing or not on github
![Page 51: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/51.jpg)
i also learned that github just released a new feature where you can split up your buildsso now, we’ll be able to split out our visual specs from our main specs hereso we can quickly see which build is passing or failing or still runningwithout the builds being combined
![Page 52: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/52.jpg)
buildbox
at bugsnag, we use buildbox for our CI
![Page 53: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/53.jpg)
buildbox allows us to add steps to our teststhat way, we can run our main specs first, apart from our visual specswhen they’re separate, our visual specs don’t slow down our main specsand we can merge non-visual pull requests without waiting for our visual specs to finish
![Page 54: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/54.jpg)
SCREENSHOTwith selenium
next, we needed a way to visit webpages and take screenshots with our rspec testsfor that, we decided to use selenium
![Page 55: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/55.jpg)
selenium
selenium is a tool for automating browsers for testing purposeswe would need to use, specifically, their web driver APIthis would allow us to drive a browser natively on a local or remote machinemore specifically, this provides an API between us, and a browser
![Page 56: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/56.jpg)
browserstack
to use selenium, we’d need to use a service like sauce labs or browserstackwe need to use a service like this because we need access to an actual browsersince we are using a CI, it doesn’t have browsers just built in on the serverwe’d either have to set up our own virtual machines for these browsers or use one of these services. we ended up trying browserstack
![Page 57: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/57.jpg)
before our visual tests, we’d need to start our proxy to browserstack and a forked rails serverand then we’d make an instance of a selenium web driverand then of course, after all of our tests, we would terminate the services
![Page 58: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/58.jpg)
we also had to allow webmock,web mock a library for stubbing and setting expectations on HTTP requests in Ruby,to run real web requestsin order to use our local servers and upload our screenshots
![Page 59: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/59.jpg)
to get our browserstack proxy running, we would just spawn a new browserstack processand terminate the process using its assigned pid
![Page 60: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/60.jpg)
and to get our rails server running, we would spin up a new rails process at port 3000unless one was currently runningand terminate it the same way as our browserstack process
![Page 61: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/61.jpg)
to set up our selenium webdriver, we just have to pass it the capabilities we wantedand a url to hit, which was pointed at browserstack
![Page 62: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/62.jpg)
setting up our selenium driver was easy,but when we were setting it up,we learned some interesting things about taking screenshots with different browsers
![Page 63: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/63.jpg)
with our web driver, we wanted to hit web pages in a real browser
![Page 64: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/64.jpg)
and be able to take screenshots of the full page, not just the current viewport
![Page 65: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/65.jpg)
unfortunately, this feature only worked in firefox, which was not ideal
![Page 66: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/66.jpg)
since internet explorer and chrome didn’t work, we couldn’t really transform this framework we’re making to be used for browser compatibility or anything like thatalthough this wasn’t ideal, for our purposes right now as a refactoring tool, firefox worked fine
![Page 67: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/67.jpg)
DYNAMIC DATA
after writing tests for static pages such as our homepage, we quickly realized that we'd have an issue with the dynamic data on our dashboard
![Page 68: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/68.jpg)
with dynamic data, you can get false positive diffs because data can change between the viewing timesto combat this, we set up fixture data for our rspec tests, and manually adjusted any other data not covered by fixtures using Selenium's Javascript supportso that we don’t get a false positive diff
![Page 69: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/69.jpg)
DIFFINGwith imagemagick
ok, we now have our tests taking screenshotswe need to figure out a way to make a diff between two of our screenshotsimagemagick worked perfectly for this
![Page 70: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/70.jpg)
imagemagick
despite having literally one of the worst sites i’ve ever seen,imagemagick did exactly what we neededimagemagick is a tool to convert, edit, and compose images
![Page 71: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/71.jpg)
imagemagick has a command-line compare tool that, with various options enabled,allows us to shell out and produce diff screenshots based off of two other screenshots
![Page 72: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/72.jpg)
for example, when we’d make a simple change like changing the header
![Page 73: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/73.jpg)
imagemagick would spot those differences
![Page 74: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/74.jpg)
and produce a screenshot like this
![Page 75: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/75.jpg)
imagemagick has a lot of options you can pass to its compare tooland we take advantage of a few of these options in order to make it work
![Page 76: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/76.jpg)
let’s go back to the options we use for a second and go over how they actually work with what we’re doing
![Page 77: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/77.jpg)
imagemagick’s compare tool, from their website, “mathematically and visually annotates the differences between an image and its reconstruction”or, in my terms, takes two images and provides a visual diff
![Page 78: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/78.jpg)
compare lets you provide a metric that outputs to standard error a measure of the differences between images according to the type given metrichere we’re using PAE, where PAE stands for peak absolute we can use the peak absolute to find the size of the “fuzz” factor needed to make all pixels “similar”
![Page 79: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/79.jpg)
1
2so if we had screenshot1 and screenshot2, that were pretty different
![Page 80: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/80.jpg)
it would end up producing a diff like this
![Page 81: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/81.jpg)
and our peak absolute measurement would be outputted as needing a huge fuzz factor to make all pixels similar
![Page 82: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/82.jpg)
the fuzz factor can be important in case we want to ignore pixels which only changed by a small amountwe might want to ignore small changes in case of false positivesfor example, i’ve had false positives before because gradients rendered SLIGHTLY differently between two images
![Page 83: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/83.jpg)
we don’t use this output right now, but it would be important if you wanted to make an assertion in your tests meaningfullike, actually have a failure if a diff was produced by a certain amountwe didn’t end up doing that because it doesn’t necessarily mean something is wrong if a diff is created
![Page 84: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/84.jpg)
a few times we were running our specs, we noticed that diffs weren’t even being produced
![Page 85: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/85.jpg)
we took a look at the screenshots, and realized that they were different heights or sizes for some reason,like if we made a change that accidentally removed the footer.imagemagick wouldn’t let us do a default compare on these images, so we had to use a subimage search
![Page 86: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/86.jpg)
subimage searching is required to have compare search for the best match location of a small image within a larger imagethis option will produce two images (or two frames)the first is the "difference" image and the second will be the "match score" image
![Page 87: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/87.jpg)
the "match-score" image is smaller image containing a pixel for every possible position of the top-left corner of the given sub-imagethe search will try to compare the sub-image at every possible location in the larger imagethis can make sub-image searching very slowas you could guess, the smaller the sub-image is, the faster this search is
![Page 88: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/88.jpg)
that being said, this option doesn’t take effect unless you have two images that are different sizesthis doesn’t happen often to us, so the amount it slows down our visual specs on CI is not meaningfulespecially since those specs aren’t tied to our main specs
![Page 89: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/89.jpg)
another fun thing we ran into was sometimes our screenshots were COMPLETELY different
![Page 90: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/90.jpg)
VS
like, completely different
![Page 91: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/91.jpg)
and imagemagick was NOT into itin fact, it just wouldn’t even give us a diff, because the images were so differentwe found an option called the “dissimilarity-threshold”this threshold determined how different two images could be in order to diff themit defaulted to .2, so we upped that to 1, and it seemed to do the trick
![Page 92: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/92.jpg)
the only caveat to this, as you may have guessed, is doing diffs on completely different images can slow down your tests by a lotlike our previous issue with sub-image searching, this doesn’t seem to happen muchand since the tests are separate from our main specs, we aren’t too worried about it
![Page 93: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/93.jpg)
the last 3 arguments aren’t exciting, these are just where our current screenshot is
![Page 94: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/94.jpg)
where our master screenshot is
![Page 95: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/95.jpg)
and where we want the diff to save to
![Page 96: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/96.jpg)
STORINGwith aws
now that we had our screenshots and diffs,we needed somewhere to throw the screenshots onlineand be able to grab them back out with our rails appso we decided to use aws
![Page 97: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/97.jpg)
aws, or amazon web services, offers cloud storage and has a ruby apiso we’ll store and retrieve our screenshots from one of their buckets
![Page 98: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/98.jpg)
/#{commit-sha}/#{area-of-site}/#{page-name}/
#{image-type}.png
we ended up using a naming pattern of commit-sha, area-of-site, page-name,and image-type
![Page 99: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/99.jpg)
/a1a1a1a/marketing/
index/diff.png
so, for an example, we could have a commit sha of a1a1a1awhere we’re in the marketing part of our siteon the index pageuploading the diff for that page
![Page 100: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/100.jpg)
diff.png current.png master.png
the image-types could be current screenshot we tookor the master screenshot we downloaded from S3or the diff we made from the two screenshots
![Page 101: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/101.jpg)
VIEWINGwith the internet
viewing the screenshots from an amazon bucket was far less than idealso we decided to set up our own custom viewing page in our admin dashboard
![Page 102: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/102.jpg)
we created a page that listed out our current branches with their last 3 commits
![Page 103: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/103.jpg)
and when you would click through, it would show you, by area, all of your screenshots and diffswhich was our end goal!
![Page 104: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/104.jpg)
THE FUTURE
however, our tool is not perfect right nowbut it’s certainly better than what we hadwhich was nothing, but
![Page 105: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/105.jpg)
MEANINGFUL TEST ASSERTIONS
right now, all of our tests pass whether or not there's a diffthey'll only fail if there's an issue executing the testit could be interesting to make our assertions mean something - like fail if there’s a diffwe’d need to think more about that, because a diff doesn’t necessarily mean failure to us
![Page 106: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/106.jpg)
0% DIFFSin the future, it’d be nice to account for 0% diffs
![Page 107: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/107.jpg)
VS
VS
like, maybe we shouldn’t upload a diff image at all if there’s no diff between the imagesthis could save us space on our aws bucketas well as speed up our tests because we’re trying to upload fewer screenshotsand it would make our admin dashboard less noisy
![Page 108: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/108.jpg)
PULL REQUESTS
we also think it’d be nice to automatically link these to a github pull request
![Page 109: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/109.jpg)
so, when a diff is created, maybe automatically attach it onto its relevant pull requestthis one is a little tricky, because maybe we don’t want to create that much noise every time we pushbut it’s an idea
![Page 110: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/110.jpg)
CURRENT commit on
BRANCH
CURRENT commit on
MASTER
DIFF BETWEEN
another thing is that we currently only diff between our current commit on a branchvs our current most recent commit on master
![Page 111: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/111.jpg)
master feature
so, when we push a new commit to our branch, it’ll diff versus the last thing that was pushed to masterthis way, we know what’s changed between the feature that i’m working on and what’s currently running in production
![Page 112: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/112.jpg)
PREVIOUS commit on
MASTER
CURRENT commit on
MASTER
DIFF BETWEEN
it would be nice if we could diff on masterwith the current most recent commitvs its previous commit
![Page 113: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/113.jpg)
master feature
so that way, in case you push a visual change directly to master, you can still see a diff of it
![Page 114: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/114.jpg)
CURRENT commit on
BRANCH
PREVIOUS commit on
BRANCH
DIFF BETWEEN
it would also be nice to see a diff between the previous commit on the current branch as well
![Page 115: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/115.jpg)
master feature
like, if you pushed a commit to your branch that changes some stuff visually,you might want to see a diff between those two, regardless of whats happening on master
![Page 116: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/116.jpg)
i also would really love to get this hooked up to more browsersthat would enable us to make this into a backwards compatibility toolas well as an automatic browser comparisonto make sure things aren’t bonked in ie
![Page 117: Implementing a visual css testing framework](https://reader033.vdocument.in/reader033/viewer/2022052509/55a204f71a28abda648b45ee/html5/thumbnails/117.jpg)
thanks!@jessicard
anyway, that’s all i have!feel free to find me after if you want to talk more about css testing frameworks or just want to say hi!thanks for having me!