Migrating to Jest

I get really excited about new tech I find interesting. When I first “got” the
testing framework AVA, I was soooo excited. I
was on the rooftops of twitter telling people to give AVA a shot. I created
two screencasts
about how to set up and use AVA, and a full blown workshop on how to test React
with AVA:

I actually had used Jest long before, but had a less than awesome experience:

So,
I convinced my team at PayPal
and we started using AVA for all our new React tests. I was getting ready to
start migrating our existing UI tests and our server tests when I started to
notice something about AVA that really put a wrench in the whole thing.

It turns out that AVA was created to be fast for lots of async tests. Not for
large test suites of async and sync tests. So for the use cases in my
application, I found this to be true:

It was just too much. As much as I loved all the other amazing and awesome
features the AVA team had worked on (and there are many), performance was so
poor I couldn’t enjoy them as much. As the testbase grew to hundreds of tests,
my MacBook Pro was brought to its knees. My fans would spin up like an airplane
ready to take flight 🛩 and it wouldn’t respond to me reliably until the tests
were done.

These performance issues led
to my team (myself included) skipping the githooks with --no-verify so we
wouldn’t have to wait to push our code changes. This led to the build breaking
for really simple things (like linting errors and broken tests) which slowed our
team down a lot.

Another major issue we found was with AVA’s testing assertions. They use
power-assert which is an amazingly helpful
utility, but would sometimes lead to reeeeally weird things happening (like
this). I realize that these are
things that can be fixed, but as AVA is a community project built in the spare
time of contributors (thank you and
you so much!), those things getting fixed
often took time. Along with that, engineers on my team sunk hours upon hours of
work trying to figure out where these issues were and working around them. We
don’t have that kind of time.

Why migrate to Jest instead of Mocha?

So, at this point, we had all our React tests with AVA, previous UI tests
(mostly backbone) with Mocha, and server tests with Mocha. The natural choice
was to move all our stuff to Mocha. But then I got involved in this thread:

Which led to
this comment
from Trevor D. Miller which I
responded to
(in which I mention that I’d write this very blog post about me migrating from
AVA to Mocha… I’d already started migrating my React Testing workshop to Mocha
here. Hah). Then
Christoph Pojer responded with
this great comment.
That pretty much won my curiosity back and I decided to give it another go on a
project.

Unfortunately, I didn’t have time to look into it right away, even though I
wanted to:

But a few weeks later I gave it a go:

I was impressed at how fast it was and even more how easy it was to set up.
Especially since I’d tried Jest before and it was soooo slow. And I was really
encouraged by this tweet from Mike Nikles:

I was slowly discovering the truth of this statement:

So I eventually
swapped AVA for Jest
for my open source module generator and now all my open source stuff is Jest and
I love it 💚 .

As is the case for me, when I get excited about new tech, I really like to
share! So I created a bunch of videos for
egghead.io about it, had a
tech chat with
Christoph Pojer
on YouTube
about Jest, and rewrote my React Testing Workshop again in
Jest.

The Migration

Client Side

My coworker Jamund Ferguson did most of the work
for migrating our React tests from AVA. We were really helped by
this codemod by
Kenneth Skovhus. This saved us a ton of
time manually updating our tests to work with Jest.

There were still some places we had to work through and we definitely had to
configure Jest in some special ways to handle our scenario. We have both server
and UI code in a single repo and each side gets its own special .babelrc which
complicated things slightly and we had some odd behaviors with some browser
globals, but for the most part it was pretty solid. We also had to migrate our
usage of proxyquire to Jest’s (superior)
built-in
mocking capabilities.

After getting that merged, a week or two later I migrated the old mocha UI
tests. Luckily Jest actually works with most of mocha’s globals so that
migration was relatively simple. Just some slight config changes and
find/replace some stuff. Pretty easy…

Server Side

And then there was the ~300 server Mocha tests. Surprisingly, this was an
absolute nightmare to migrate. Most of the problems stemmed from some of our
dependencies doing some interesting and archaic things. But some of these were
legitimate bugs in Jest that were reported and fixed. Finding those bugs took a
really long time.

I ended up trying 5 different approaches to the migration. I really wanted to
limit changing the existing tests to just find/replace for globals and stuff,
but ended up manually changing every single test in unique ways. It took about 2
weeks of effort. In the process I learned a TON about how our dependencies work
including Jest.

I’m not going to share everything that I learned (that would take a really long
time and a lot of background info), but
here’s one really interesting one
that tripped me up for a very long time. It involved polyfilling JSON.parse.
And here’s another that caused me
some grief. Oh, and
requireindex ended up not working at all
and we had to remove it from our project. Luckily we weren’t using it heavily.

The reason it took so long for me to get the server tests migrated wasn’t so
much the amount of effort it took to do the code changes (though that did take
quite a bit of time) but the amount of time it took to dig through our project’s
dependencies to figure out which one was the root cause of the issue. Most of
the errors were thrown in bluebird promises which provided pretty much no useful
stacktrace of any kind.

I should also mention that my experience was not at all typical. Most people are
able to migrate to Jest fairly quickly and seamlessly.

Was it worth it?

For the client-side stuff, I can say for sure absolutely positively YES:

As for the server side. Well, my PR for the server changes was just merged
yesterday, so I can only say I think that it was worth it. Jest provides a ton
of great features, not the least of which is amazing performance as the number
of tests grow. It’s exposes a surprisingly powerful API with great and
reasonable defaults. Jest ships with
easy code coverage instrumentation and reporting
right out of the box. It has
an interactive watch mode
that I love. It has the best mocking story of anything I’ve ever seen. It has
snapshot assertsions
which work really well for React components, server responses, and redux state.
And most importantly, I think my co-workers and I will stop running
git push --no-verify like we were before which will help our productivity, so
I think that it’s definitely worth the migration!

Stats

So the stuff people always want to know is: “Is it really faster?” Well…
Here’s some clipped output from running the test commands with time before and
after (note that the before tests are from over a month ago so there are a few
more tests in the after numbers):

If you’re unfamiliar with the time command see
this. “real” is the thing we’re focusing
on here.

Note: This is with the cache in all cases. I ran the tests several times in a
row with my machine otherwise idle to get the best numbers I could for all
tests. I also ran these with code coverage because that’s what our git hook
runs. Also, you’ll notice there are more total tests to run in the “after” case
because “before” is one month ago.

So here are some important stats based on my runs:

UI Before: 144ms per test

UI After: 28ms per test

Server Before: 111ms per test

Server After: 116ms per test

One thing that I found with Jest is that the ms/test stat decreases as you add
tests. Turns out it’s really good at running tests in parallel. So I expect we’d
see a wider gap with more tests.

Conclusion

Jest is awesome and I recommend you give it
a solid look. Seriously. Oh, and don’t forget to
give it a star 🌟!

See you on twitter!