Success Story

Noble Studios Helped a Client Increase Newsletter Subscriptions Through Iterative Testing

385% Sign-ups

About Noble Studios

Noble Studios is a creative digital performance marketing agency based in Nevada, USA. Starting in 2003, they have emerged from Carson City and expanded to Reno, Las Vegas, and the UK. Being a ‘digital original,’ Noble Studios has fostered independent thinking, creativity, and compassion in their team to make meaningful connections, one of the company’s core values. Noble Studios has been recognized with distinguished honors in the digital marketing industry, including becoming a Google Premier Partner and one of the fastest-growing companies by INC 5000.

Noble Studios has a dedicated CRO team that facilitates clients’ websites experimentation. VWO has been their choice of partner for their experimentation requirements due to the competitive pricing, key features, and the simple and effective interfaces it offers for test building.

Testing methodology

Ideas are the lifeblood of a CRO program, and choosing which experiments to run is critical for saving clients time, money, and ultimately success. Noble Studios uses its ideation methodology to focus on the best tests to run given time and traffic are limited. Through this process, using a prioritization model that balances estimated test impact vs. effort, Noble Studios can democratize good ideas and eliminate bias to let ideas prove themselves rather than choosing by subjective preference.

Goals

In our conversation with Noble Studios, we gauged the opportunities their client Travel Nevada had with their newsletter signup rates. They wanted to increase the visibility of the newsletter signup section on the Travel Nevada website and reduce the friction for visitors to complete the signup process. To accomplish this specific goal, Noble Studios prioritized the CRO issues on the website and tackled them one by one. This approach enabled them to tackle one issue at a time and feed learnings from one test into another in an iterative, generational testing model, giving definitive and data-driven conclusions in each test.

Observations

Noble Studios conducted an A/B test using VWO Testing on Travel Nevada’s website to see if expanding the newsletter signup section automatically at 75% scroll depth would translate into increased signups. The variation resulted in increased signups with no negative effect on other engagement metrics.

In the second test, Noble Studios tested the position of the “fixed elements.” Those elements were originally the share and email widgets fixed to the side of the screen. The test had variations positioned on the left-hand side and right-hand side on desktop and at the bottom of the screen on mobile. The variation having right-side elements on desktop performed best for newsletter signups with minimal impact to engagement metrics.

In a subsequent test, acquainted with learnings from the previous tests, Noble Studios went a step further. They ran a test with a variation having a right-side fixed element that auto-opens once 75% scroll depth is reached, and therefore, presenting an opportunity for visitors to complete the signup process staying within the widget itself, without being directed to a dedicated newsletter signup page.

Tests run

The collaborative team of Noble Studios and Travel Nevada hypothesized that expanding the newsletter signup at 75% scroll depth will lead to an increased number of signups.

Test 1: Nest the newsletter signup

The control flyout would pop out, but then disappear after no interaction. In the control version, there was no way to re-activate the pop out.

Control – View 1 (Flyout displays after scrolling 75%)

Control – View 2 (Flyout disappears after no interaction)

Variation – View 1: (Flyout displays after scrolling 75%)

Variation – View 2: (Flyout remains accessible after no interaction through a tab)

The variation received more signups than the control, which declared the variation as the winner with no negative impact on other engagement metrics.

Take a free trial with VWO to understand how easily you can set up A/B tests and get interpretable results.

Test 2:

In the second experiment, Noble Studios tested which position of the fixed elements would lead to increased signups.

The variation with right-side elements on desktop performed best for newsletter signups with minimal impact on engagement metrics.

Test 3: Direct newsletter subscription

The third test combined the learnings from the previous tests to hypothesize that a variation having a right-side fixed element that expands once 75% scroll depth is reached will lead to an increased number of signups. In this test they took it a step further and included the sign up form within the widget for the Variation.

Control view:

Variation view:

The team observed the primary metric that was newsletter signup clicks. They monitored both the original signup button (on a standalone page) and the new newsletter signup (within the fixed widget). The newsletter signup button variation had an increase of +385% at a 100% probability to beat control.

Using combined insights from previous tests (right-side fixed widget + auto-open widget at 75%), Noble Studios was able to dramatically increase overall newsletter signups, with an additional 104 signups from the variation received in 7 days of test running time.

Conclusion

Based on the learnings from these generational tests, Noble Studios identified a need to optimize the newsletter’s thank you page experience. Converted visitors were left stranded on the landing page and unable to continue their site exploration. Additionally, they hope to find new and better ways to trigger the newsletter flyout widget with considerations such as triggering after #X pages, #X scroll depth, #X time on site, etc.

VWO has been an incredible partner and their platform helps us get real results with tangible insights for our clients. The ability to set up as many goals as you want, on the fly, is incredibly valuable to keep an experiment moving and get as much insights as possible.

Mason Furr
Web Analytics / CRO Specialist

Noble Studios currently runs one to five tests per quarter for each of its clients. They are steadily increasing their testing velocity by working towards having ownership over implementation and development capabilities, and are hopeful that this will ensure smooth test timelines with more streamlined communication.

Start a free trial with VWO to understand how A/B testing can help you achieve better testing velocity and increased conversions for your business, or learn more about Noble Studios’ conversion rate optimization services and how they achieved their results.