Reviews written by Buckle.com email recipients, 10/21/2011 – 10/09/2012

A year ago, Bazaarvoice conducted a series of tests that eventually lead to as much as 146% increase in email campaign conversion. Success only came, however, after we realized and adapted to a fundamental shift in email technology. This shift affects all of us, so any email marketer could likely achieve similar results. There’s just something you’ve got to know about email.

You won’t tweak your way to a eureka-moment

Email is an extremely effective way to encourage customers to write reviews. Post-purchase emails reach customers at a contextually relevant time, and they allow reviewers to bypass site authentication. In 2011, post-purchase email recipients wrote 55% of Buckle.com’s product reviews. Buckle was an ideal candidate for email optimization testing given their active campaigning.

The email template Buckle had been using worked fairly well: For every 100 emails Buckle sent, 25 customers opened them, three clicked the calls-to-action, and one person submitted a review. Nonetheless, our teams wondered if improvements to the visual design and copy could positively impact conversion. The new templates were heavily branded, incorporated professional photography, and played on various emotional appeals.

BP2
Post-purchase email templates

And they all failed. No single template greatly out-performed the rest, and after three months, we had to either call it quits or try something new. We wanted to discover something that would improve post-purchase email conversion for all our clients, and none of these templates offered anything compelling.

BP3
Reviews written by Buckle.com email recipients, 10/21/2011 – 07/01/2012

If you can’t beat ‘em, pivot

We realized we wouldn’t tweak our way to a eureka-moment: Visual design and copy could only take us so far — but what else was there? In August 2012, we didn’t know, but we were okay with that. We would set aside all our predilections and take a completely naive approach to email campaign design.

Litmus estimates that 38% of email is opened on mobile devices. If email were invented yesterday, email designers would build templates that are readable on smartphones, right? And so emerged a new hypothesis: Smartphone-friendly designs would increase conversion by making the end-to-end experience better for everyone.

BP4
Paid participants tested new designs

Qualitative testing in our user lab led us to design some very boring templates. They were linear, practically image-free, text-based… but they were also highly readable. Smartphone users didn’t have to pinch, pan, zoom, or even scroll down to read the primary message and tap the call-to-action. And this, we found, is what they care about.

BP1
Left: Original template without downloaded images, Right: New template without downloaded images

We also redesigned the landing page, thus ensuring an entirely smartphone-friendly end-to-end experience.

BP5
Left:Original reviews submission form, Right:Smartphone-friendly redesign

Finally, we used fluid styling and responsive techniques to make sure the templates and landing page looked decent on any display size. Even so, we were pleasantly surprised when Buckle permitted us to send these boring, ugly templates to thousands of their customers!

BP6
New designs scaled to any display size

And then this happened.

BP7
Reviews written by Buckle.com email recipients, 10/21/2011 – 10/09/2012

Bam! Our best-performing template drove up the daily average 146%! We were blown away. This was success. We finally disrupted the status quo.

BP8
Performance comparison of default, marketing, and smartphone-friendly templates

The new templates out-performed the others in every way: Open rate (30.4%), click-through rate (11.25%), and overall conversion (3.04%).

Now is all that matters

I could conclude with a list of email design best practices, but the moral of the story is: best practices change. Our initial three months of testing were a distraction. We were well versed in traditional design principles and prescribed recommendations, but we knew nothing about how people use email today. When we pivoted, we learned who our audience really is, we realized that email is a mobile thing, and we figured out what practices are best now. In today’s era of rapidly evolving technology, now is all that matters.

Stay tuned: “Throw out best practices, double email conversion – Part II” discusses unrecognized audiences and their conversion potential.

  • http://mobile.bazaarvoice.com/ A. Litsa

    Hey y’all. I’ve applied to present this topic at SxSW. Please vote for my panel if you would like to hear a lengthier, in-person discussion. Thanks! http://panelpicker.sxsw.com/vote/17907

  • http://mobile.bazaarvoice.com/ A. Litsa

    That’s true. Open rate is a squishy number. Recipients didn’t have to download images in order to read the mobile-friendly template, but the click-through rate was much higher. To your point, the increased open rate is probably a reflection of the increased click-through rate (since image downloads were unnecessary). Thanks for the observation!

  • Cameron Ackbury

    Great test results!

  • JitSalunke

    Wow, thanks for sharing Litsa. I always believed that plain text emails are far more effective than html emails. Specially when >35% of emails are read over mobile. This experiemet validate that.

  • stephenkiers

    Open rates could be higher because there was more feedback given for them to track the opens… more click throughs, more replies, more downloading images, more forwarding to friends.
    It is usual that more people ‘open’ the email than is reported to your esp, they are just never ‘reported’ back through any of the above.
    My 2c

  • http://mobile.bazaarvoice.com/ A. Litsa

    Hi Dagny. It is speculation of course, but I think the jump in open rate is due to our “subtitle” technique. Given so many people check email with mobile devices, the way an email appears in a mobile email client matters. Most such clients display the first line of the message just underneath the subject line (if the subject line is not too long). In our template, the subject line sums up the message and the “subtitle” tells the recipient why he or she should care.

    Please check out my more detailed explanation (with screenshots) here:
    http://mobile.bazaarvoice.com/uxdd/handbook.php?article=optimize-for-the-mobile-inbox

  • Dagny

    How do you account for the increase in open rate? Did you also change the subject lines or the from emails?

  • Russ_Somers

    Great test, results and post! The “you won’t tweak your way to a Eureka moment” echoes something I’ve come to strongly believe: tweaking, testing, data-driven marketers get small incremental gains that, over time, build up like compound interest. That is certainly one way to improve a business. But big, sudden gains require a courageous leap of “We don’t have any data around this because it’s never been done, but let’s try it and succeed or failure spectacularly.”