IMEC Lab: Help Test & Validate Web Marketing Hypotheses

UPDATE: IMEC Labs is now being run by Eric Enge and the folks at Stone Temple Consulting (as I was unable to commit enough bandwidth consistently to continue the experiments). The new signup form is here, and I’ve updated the links below.

Over the last few months, I’ve been running some experiments in Google’s search engine and on social media platforms with friends & colleagues. The results have been really exciting to see, but the more I invest in these efforts, the more I perceive that our small group is not enough. To truly validate, repeat, and scale the kinds of experiments we want to perform, we need help – we need a cooperative.

Hopefully, that’s where you come in 🙂

From a recent test of anchor text’s influence on rankings (covered in this presentation on Slideshare)

Starting today, I’m launching a small, private test pilot of a program I’m tentatively calling IMEC Lab. IMEC stands for “Internet Marketing Experimental Collective.” The goal is to assemble a group of marketers who care about producing high-quality, consistent, repeatable experiments in search engines and on social media to learn more about how these platforms work. If this sounds like it’s potentially up your alley, read on.

Running live tests in search engines and on social media is hard. There’s lots of potential confounding variables, plenty of potential mishaps, and the always-present-but-nearly-impossible-to-prove threat of tampering by players at the engines or networks themselves seeking to prevent transparency. If the IMEC is to succeed, it will need a set of guidelines that accounts for the challenges of live tests. I have proposed an initial set below, and intend to bolster this with suggestions from across the field and from those who join the Lab.

Rules Governing IMEC’s Testing Version 1.0 (April 20th, 2014):

1. Projects will be proposed based on a hypothesis and testing methodology that can confirm or deny the hypothesis.
2. No test result shall be considered valid unless it is conducted a minimum of three times with consistent results.
3. Tests in progress are not to be publicly shared by IMEC members until all data has been collected.
4. Even after completion, specific details of tests (e.g. keyword phrases tested in search results) should be kept private so as to limit future risk.
5. No test should ever be performed that is likely to cause substantive harm to any individual, organization, or business (e.g. keyword phrases for testing in search engines should be non-commercial and low in search volume so as to have minimal impact on searchers and search traffic).
6. To limit the impact changes made by networks or engines may have on test results, tests should be performed and repeated with as little delay as possible. However, IMEC must intelligently balance this with the potential fluctuations that can occur from temporal anomolies, and leave tests running long enough to show that results are not simply due to a temporary effect (unless that is a hypothesis’ specific charter).
7. As much data and evidence as possible should be collected as frequently as is reasonable. For search results, this means a minimum of a record/screenshot each day during testing, and for social networks, depending on the test, every hour.
8. To help control for any potential retaliation, tampering, or issues that arise due to this group’s existence, all members (save a small board) will remain anonymous to both each another and and the broader public.
9. We should expect to get some tests wrong, fail to consider all the potential influencers, and, on occasion, be misled. To combat this, we’ll strive to be open with methodology once a set of tests are complete, and always open to revisiting a test should it become clear that we’ve overlooked major factors.

To do this right, we need the help of many dozens, perhaps even hundreds of web marketers of all stripes. If you have the bandwidth to answer an email every week or two, make changes to your website, post to a social account, or perform a short task on the web, and interest in contributing to greater knowledge and transparency about web marketing platforms, please apply for the IMEC Lab:

This Quick, 13-Question Survey is all it takes!

Benefits of Joining IMEC Lab:

• Be the first to see the results of tests
• Contribute to greater transparency about the inner workings of search engines & social networks
• Rand will owe you a favor 🙂
• If projects go well, I’ll try to find budget at Moz to help say thanks

Unfortunately, because I’m not sure how some search engines or social networks might react to this, IMEC can’t publicly thank or credit you, but I’m confident thousands of us in the field will join me in expressing gratitude for your contributions.

In addition to the group of testers, I’ll be assembling a small IMEC board to help review the structure of experiments, review results, record data, and assist in management of the process. I’ve already got a couple people on board to start this, but if you think it’s up your alley, please drop me an email.

Here’s just a sample of the types of hypotheses I’d love to test in the early stages of the project (all of which would be up for prior review, of course):

• Does Facebook consider the recency of a user’s “liking” of a page when factoring in whether to show posts from that page to a user’s network?
• Is it the case that Google still only counts the first anchor text when multiple links to the same page are present?
• Do +1s of a URL have any impact on ranking in non-personalized search results?
• How many searchers/searches does it take to influence Google’s search suggest?
• Can a group of visits to an unindexed URL in Chrome or Internet Explorer get a URL indexed in Google or Bing (respectively)?
• Can Twitter’s “Who to Follow” suggestions be influenced and how?
• Will second-order impacts of a URL being retweeted or frequently shared on Facebook impact Google’s indexation or results (e.g. because those URLs might be picked up and posted by third-party aggregator sites that Google indexes)?
• Can unlinked mentions of a URL impact Google’s indexation or rankings?
• Can using certain semantically-related words make a page more likely to rank well? How much ranking influence can these types of on-page factors have in comparison to raw links?
• What level of ranking influence do internal links have, and how much does anchor text matter on internal links?

Even if you can’t join, if you believe in the goals of this project, it would be greatly appreciate if you could help this post and the application spread as far and wide as possible. Thanks so much, and hopefully, I’ll have some exciting results to share in the months ahead.

APPLY TO JOIN THE IMEC LAB HERE!

p.s. If you have hypotheses of your own you’d like to see the group test, please put those in the comments below and if there’s substantial interest, we’ll try to get to them.