June 20, 2018 by Michael Greenberg

Optimize It: Website Testing Techniques

 This content was adapted from my talk at SUGCON EU in April 2018.

You finally did it. You published that page on long haired cat memes and when you checked your backlog, it was empty. You caught up!

Now, of course, its time to start optimizing everything on your site, which means its time to start testing. Everything.

Optimizely Refinement Exploration Testing Charts

 Image Source: Optimizely

Most people think of testing as simple A/B tests of one attribute of a web page or component. We think of testing in two areas: exploration and refinement. Exploration looks for new areas that might be more productive than the current choice or solution. In this case, we ask ourselves “Do we have [assumption] right?”

Refinement looks for incremental improvements to the current choice or solution. In this case, we ask ourselves “What might work better than this?” So, for example, you might test different images of workers in a conference room to refine the results, or you might test images of landscapes or infographics instead of people to explore for better options. This idea of testing in an exploratory fashion is critical for finding larger impact ideas.

With this framework as our starting point, here are the first 16 tests you can pursue to improve your web results.

1. Does Page Speed Matter?

Google’s guidance is for two second load times, with new emphasis on speeds on slow bandwidth mobile connections. So many companies are trying to determine how much impact there is. One way would be to test different pages with similar search characteristics and see if the results change. Tools like Lighthouse and GTmetrix can be a help here.

2. Does Mobile First Matter?

Everyone talks about it. But does it make a difference? First, you have to ask what it means for you and your company. Maybe it's just responsive. Or maybe you design everything around mobile interaction – big buttons, few text links, highly edited copy, punchier images, faster load times. This is a fundamental problem for lots of companies – their data says mobile traffic is low, but is it low because you haven’t embraced mobile? This is something to test, potentially with alternative home page designs or fundamentally different experiences for desktop.

3. Full Home Page Version Test

One of the best ways to explore lots of ideas at once is to run different versions of your home page. Even though home page traffic is probably not much more than 20-30% of sessions, it's still the most visited page. It's here you can begin to spot behavioral differences by seeing which new ideas seem to outperform their predecessors. It’s a great way to explore ideas quickly.

4. Persona A vs. B

Persona testing itself could be an entire post. At its most basic, testing reflects two different questions: 

Does this content really reflect persona A or B? Maybe you would test tagging content with different profile keys over different time periods and see if the resulting onsite personas perform better.

Is my visitor persona A or actually persona B? It's common to have visitors with virtually identical content needs but different objectives. A plastics procurement buyer and user will likely behave similarly on a website but one wants $10 million of goods and the other $20 worth.

5. Segment A vs. B

Like personas, segments can be defined in a huge variety of ways. So, segment testing might be around the definitions of the segment, such as where the age cutoff is between segments, or it might be around which segments are more effective for targeting messages and content. Ideally you have a good data analyst on staff who can take a more sophisticated approach to segmentation and build models, but in our experience not many companies have this yet.

In the meantime, test your segment definitions in broad strokes to see if there are obvious differences, looking at geography, source, average spend, recency, frequency, responsiveness, age, gender, income, role, preferences, lifetime value, education, category interest, marital status, family size, and any of hundreds of other factors.

6. Channel A vs. B for My Personas

Let's pretend your personas are rock solid and completely right. You still have to figure out the right way to engage. Is it social? Email? Facebook? Twitter? Snap? WeChat? Native app?

And of course, while you need to be engaging in all of these channels, as you put together customer journeys you need to think about where to focus effort. Should I just put all my attention on email and engage prospects post-visit via email? Or should I put everything into Twitter? Or allocate percentages rationally. These are all things to test regularly. More importantly, if your marketing team doesn’t really have a good reason as to why they use particular channels, suggest you begin testing and start gathering data. Setting up A/B splits in your marketing automation is the likely place to start.

7. Does Accessibility Help or Hurt?

If you follow a WCAG 2.0 Accessibility standard, its hard to test the impact since you really don’t want a version without accessibility. But you might want to test level A vs. level AA and get a sense of what more stringent requirements does to results. As with these images of the Hedgehog website with high contrast mode on and off, there will be appreciable differences.

 Hedgehog Sitecore page header with orange background regular contrast mode Hedgehog Sitecore page header with purple background high contrast mode

8. Does More Personalization Help or Hurt?

Personalization of content requires constant testing and is the subject of a future post. But you might want to test whether more personalization improves results or not. Since this might slow down website performance or have any number of other impacts, the results may be surprising.

9. Does My Design Help or Hurt?

Of course, your creatives will not want to hear this. We are big proponents of this now…that design should be continuously “tested in place” instead of thinking about a major redesign. And that the outcome of a redesign should be regularly tested for improvement. Try simpler versions of existing designs, or removing animation, or minimizing negative space. You should also fix things that look outdated through iterative tests as well. And maybe if you can’t or don’t want to do it on your live site, you can use usability testing to do it with a small group and prototypes.

10. Does My JavaScript Help or Hurt?

Similar to the design question, many modern sites use a lot of front-end JavaScript. Is this good or bad? Sounds like a good thing to test. If you have some fancy JavaScript on your site, it's probably a good idea to do a version without and test them against each other. On hhog.com, for example, we use JavaScript to animate a build of the hero component. Looks interesting, but how much is it impacting results? If you still have interactions like this on your site its another area to explore.

11. Is a Long Page Better?

In this era of long scrolling pages, testing standard length page with longer alternatives make lots of sense. Our favorite example was from Crazy Egg which showed a 50+% improvement when testing a page 6x longer than the original.

12. Number of Navigation Items

This is a more classic area of testing, but one that is often overlooked because not all systems can dynamically test navigation. Virtually every website marketer has gone through the pain of trying to keep the size of navigation from exploding, since everyone wants to make sure their area is accessible with a single click. So here is a good opportunity to test the number and see what works best for your visitors. Through good experiment design, you should be able to figure out what number appears to drive the best results and then stay around that. If you look at the navigation above, we use 4 items on hhog.com, which we probably should retest to see if a different number works better.

13. Navigation Labels

Related to the number of items is their labels. Is “What We Do” better than “Services”? Is it better to have “Learn” or “Knowledge Base”? Even with a good information architect, its virtually guaranteed you didn’t get this right the first time. It’s a really good place to test different word choice, to see how your visitors respond to different options.

14. Navigation Item Order

This is particularly important for mobile, since virtually all navigation is vertically stacked on a phone. Is it better to have “Buy Now” at the top? Is it better to have the most common clicks stack ranked or better to conform to website norms? This is another area that sounds simple but especially as a site ages some of the original thinking around the navigation may not be as relevant and you should start testing new ideas.

For example, now that we have data, we find those buying products from our teamdevelopmentforsitecore.com site go straight to product pages or “Licensing,” and rarely visit other parts of the site (or start from blog posts). Those needing support typically start on a blog post then hit “Help” or “Contact Us”. So, what should be at the top of the list? Maybe “Licensing” for home page visitors and “Help” for blog visitors? Another good thing to test.

15. Items in Second Level Navigation

These days, virtually everyone seems to use two-level navigation to get visitors to their desired content faster. But has it become overloaded? Or maybe stripped down?

Particularly in mobile, a big secondary nav can be a problem, offering a bad experience as people try to navigate the finger swiping necessary to get to the right place. Testing the number of items makes a lot of sense here.

16. Mobile Navigation

Your desktop navigation experience may be just fine, but when it translates to responsive, it might not work nearly as well. So putting together tests, whether usability-based or on the live site, will help you separate the problem of mobile nav from traditional nav. What you will likely find is your mobile nav needs to be fundamentally different than your desktop. This is a trend we are seeing much more often and should be tested sooner rather than later.

I will cover more testing suggestions in my next two posts.

Keep in Touch and Stay Informed

Get updates, industry reports, white papers and more Hedgehog love.