BrandExSEO™

Keyword Difficulty

00-keyword-research-robot

Keywords Explorer 2.0 by Ahrefs: Setting new standards for keyword research

Tim Soulo
Tim is the guy responsible for marketing and product development at Ahrefs. But most importantly he’s the chief evangelist of the company.
If you’re an existing customer of Ahrefs, you’ve probably noticed our almost unhealthy obsession with collecting the industry’s best data and building industry’s best tools on top of it.

Do we succeed at it? I’ll leave that for you to judge.

But one thing I know for sure — Ahrefs team is extremely competitive and we are never scared to go anthe extra mile where others would prefer to skive. For instance we update our hardware every 10–14 months (we’re soon moving to Xeon E5V4 CPUs and 4TB SSDs) for no particular reason other than to make sure that we’re using the world’s best technology and operating at the highest speeds possible.

So after looking at the usage statistics of our Keywords Explorer tool and realizing that it is was one of the least used tools in Ahrefs, we were immediately challenged to turn things around.

We knew that adding a few cool features here and there wouldn’t really make a difference. The only option was to start from scratch and take a shot at creating the very best keyword research tool in the industry.

So without further ado, I’d like to present you Keywords Explorer 2.0 — our brand new (and extremely powerful) keyword research tool.

(And the whole Ahrefs team is now holding their breath waiting for your assessment of our work.)

[TL;DR] What makes Keywords Explorer 2.0 special

This overview turned out to be pretty long and detailed. So just in case you’re in a hurry here’s a quick TL;DR.

3 Billion keywords database — Keywords Explorer 2.0 is running on an absolutely monstrous database of ~3.1 billion keywords from 100+ countries. This database is updated with fresh data every single month. To the best of my knowledge no other tool will match our size as of today. [Read more]

Accurate “Search Volume” (Local & Global) — we’re processing very large amounts of clickstream data, which allows us to calculate much more accurate search volumes than those in Google Keyword Planner (especially if you’re on a low spending Adwords account). [Read more]

Clicks” metric — searches in Google don’t always result in clicks on any of the search results. For example people search a lot for “donald trump age”, but they don’t click on any results because they see the answer right away.

Thanks to clickstream data we can see if people are actually clicking on any of the search results after performing a search. Which means we can show you quite accurately which percentage of searches result in clicks. [Read more]

A lot of our beta testers called this new metric a “game changer”:

01-clicks-metric-feedback

Tons of keyword ideas — one of the main things that our customers loved about the old version of Keywords Explorer was the list of keyword ideas that effectively dwarfed any other tool:

02-massive-list-of-keyword-ideas

Well, the 2.0 version of the tool with its 3.1 Billion keyword database will give you even more. [Read more]

Parent topic — for any keyword that you put into Keywords Explorer 2.0 we will check the #1 ranking page and determine the best keyword that this page ranks for.

This little trick gives you an idea if your keyword belongs to a broader topic and if you can rank for your keyword while actually optimising your page for this broader topic. [Read more]

Keyword difficulty — our KD score has already won the first prize among the existing solutions. So to be honest we didn’t change anything about it in “2.0” version of Keywords Explorer. But still there are quite a few things you might want to know about it. [Read more]

SERP overview & Positions History — our Keyword Difficulty score is quite good, but it will never beat an opinion of an experienced SEO specialist. For that reason we have designed an all new “SERP overview” report, which brings all the core SEO metrics together to help you make educated decisions about the actual difficulty of a given SERP. [Read more]

And for a subset of the best keywords in our database we’ll also show you a cool graph with ranking history of top5 search results, which gives some extra insights about the toughness of a SERP. [Read more]

Keyword Lists — a very simple, but a highly requested feature. Once you come across some juicy keyword ideas, you can now save them into a list for future reference. This can be a huge time saver if you’re doing keyword research for multiple projects. [Read more]

Upcoming Features — here at Ahrefs we’re super quick to bring the feature requests of our users to life. And thanks to many awesome beta testers, we already have quite a roadmap for 2.1 and 2.2 versions of Keywords Explorer. I’m going to tease some of the features that will be released in just a few weeks from now. [Read more]

That’s it for a quick TL;DR.

And now onto a much more detailed review of Keywords Explorer 2.0.

#

The original version of Keywords Explorer was already running on a massive database of ~300 Million keywords, which our customers seemed to enjoy a lot. But in 2.0 version we have increased that by 10x.

Where did we get so many new keywords?

We have processed huge amounts of clickstream data for the past 12 months (i.e. we analyzed what millions of people around the globe were searching for in Google), so all these new keywords in our database are very fresh and trendy.

What is “clickstream data”?

There’s no point in me explain it here, since there’s this great article by Russ Jones which does it perfectly: “Google’s War on Data and the Clickstream Revolution”.

100+ countries supported

Not only have we massively increased our database in size, we now also support a load of new countries.

Here’s a chart showing you the current top ten countries in our database:

03-top-10-countries-in-keywords-explorer

So if you were looking to perform keyword research in any language other than English, please give our tool a try and let us know how it goes.

Monthly database updates

In the teaser video for Keywords Explorer 2.0 we promised you “2.8 Billion keywords”. But at the time of writing this post we’re at 3.1 Billion already.

And there’s a chance that our keyword database will get even bigger, because we’re now processing fresh clickstream data every month.

This allows us to regularly import new keywords that were never seen before and build fancy looking trends for the keywords that we already have in our database.

For example, here’s our search volume trend for “iPhone 7”:

04-iphone-7-search-volume-trend

If you enter some keyword into Keywords Explorer 2.0 and we return “No data” — in 99% of cases that means that this keyword is simply not popular enough.

And in case this keyword is only starting to gain popularity. You’ll find it in our tool next month after we process fresh clickstream data.

#

SEOs knew about the “dirty secrets” of Google Keyword Planner for quite a while now, but this summer Google managed to surprise us (in a bad way) by limiting access to keyword data for “advertisers with lower monthly spend”.

Which means that (unless you setup a campaign and start running ads) you will now see six lousy ranges instead of the accurate search volumes that you used to get.

These six ranges are: 1–100, 100-1K, 1K-10K, 10K-100K, 100K-1M, 1M+, and they make GKP almost useless for anyone who’s not looking to spend money on ads.

But that is only a half of the problem.

The other half is that GKP is now grouping keywords with similar meaning and showing identical volumes for them:

05-keyword-planner-grouping

These two pieces of news came as an existential threat to every keyword research tool that relied on Google Keyword Planner for search volume data.

But luckily we have clickstream data, which helps to bring the search volume accuracy back.

Ahrefs’ Search Volume

The search volume data that we have collected from clickstream was paired with our existing volume data (from Google Keyword Planner) and after a few iterations we created a model that would give a very accurate search volume for almost any keyword there is.

Keywords Explorer 2.0 shows you both Local and Global search volume, with the figure being an average over the last 12 months.

06-local-and-global-volume

Un-grouping keywords

Back to this very irritating issue with keyword grouping. Let’s try the following keywords in GKP:

  • seo;
  • search engine optimization;
  • seo optimization;
  • define seo;
  • search optimization.

08-gkp-grouping-keywords

They have grouped 5 keywords that I’ve entered into 3. And that is a “paid” account I’m using, not even a free one.

Now let’s put the same 5 keywords into Ahrefs:

09-ungrouping-keywords-in-ahrefs

As you can tell, we’re able to un-group these keywords and show you the real search volume for each of them.

Return Rate

This is our brand new metric that no other tool will show you as of today.

Return Rate” is a relative number that gives you an idea of how often people perform the same search over the course of a month.

1” means that people never search for that keyword again. But “2” doesn’t mean that they search for it two times per month on average. It just means that they search for it quite often.

Yes, it’s super confusing. But give me a chance to explain it with an example:

10-return-rate-in-ahrefs

As you can see, people in the UK tend to search “queen age” only once and they hardly ever search for it again. That’s probably because they memorize the answer instantly.

But then you can see that they return to search for “brexit” quite often. Probably because the search results always contain some fresh news on that topic that are of high interest to them.

And finally, “manchester united” has the biggest return rate, which could be an indication that people in the UK follow football news almost religiously.

So again, RR 2.02 for “brexit” and RR 2.22 for “manchester united” doesn’t mean that people search for one of them 2.02 times on average and 2.22 times for the other.

But it gives you a general idea that people in the UK perform a search for “manchester united” a bit more often than a search for “brexit”.

#

This has been lying on the surface for quite a few years now, but somehow almost everyone in the SEO community was successfully ignoring it.

I’m talking about this simple realization:

Search Volume” is not indicative of true traffic potential of a keyword.

Click to tweet

Here’s a great example that clearly illustrates what we mean by that:

12-with-clicks-and-without

As you can see, the keyword “chauffeur” has a huge search volume of 42,000, but that only results in 10,035 clicks.

People search for it, but don’t seem to click on the search results. Why?

There are two reasons for that:

  • Many people put the word ”chauffeur” into Google just to check if the spelling is right (I do it all the time with some fancy words);
  • Google gives you a Knowledge Card right at the top of the search results with a definition of the word “chauffeur”.

What else would you want to know when searching for the word “chauffeur” besides spelling and definition?

So there you have it. People don’t need to click on any of the search results to get what they wanted — hence the low number of clicks, while the volume is deceptively big.

Now let’s look at the other keyword — “wow chauffeur”.

WoW” here refers to World of Warcraft. So it looks like people are looking for some information about the “chauffeur” character in World of Warcraft.

And once you look at the search results for this keyword, you’ll see that there’s no good information upfront. Which means that you have to click on something in order to learn about “wow chauffeur”.

This is why this keyword gets a high volume of clicks.

So if you compare these two keywords purely by their search volume, it may seem that there’s almost a 10x difference in traffic potential.

But when you compare the number of actual clicks on the search results, you’ll see that there’s only a 2x difference in traffic potential.

Sounds awesome? Read on!

Clicks Per Search (CPS)

Like I just said, the search results for “wow chauffeur” don’t contain any good information upfront, so you don’t know whether or not you’ll get what you’re looking for after clicking a certain search result.

And so you click a few of them.

That is why the CPS (Clicks Per Search) metric for “wow chauffeur” says “1.15”. It means that searchers on average tend to click on more than one search result when searching for this keyword:

13-clicks-per-search-ahrefs

And if you were wondering why the number of clicks for “wow chauffeur” is bigger than it’s actual search volume, the Clicks Per Search metric is the answer.

Knowing the CPS of a keyword is crucial for understanding the traffic potential of pages that rank below the #1 result. The higher the CPS — the more chances that you’ll get some nice traffic even at a slightly lower position like 4 or 5.

But organic results aren’t the only ones getting clicks.

Organic Clicks VS Paid Clicks

When you bid for some keyword, Google will show your ad at the very top of the search results, pushing down all the pages that rank for this keyword organically.

And since Google is now showing up to 4 ads, the organic results are about to go below the fold on my 15” screen:

14-ads-in-google-serp

What I’m trying to illustrate here is that these ads quite effectively steal clicks away from organic search results.

This is why experienced SEOs will always check the actual SERP for ads to determine if that keyword is worth ranking for.

And Keywords Explorer 2.0 can now show you what percentage of all clicks go to ads.

Let’s see on a real example how much value this metric brings to the table.

15-paid-clicks

If we were only making decisions based on pure Search Volume — “iphone 7 cases” would clearly be a much better keyword to go for than “best iphone 7 cases”.

But let’s see how many clicks ads steal away from each of these keywords:

  • iphone 7 cases”: 25,353 * 22% = 5,577
  • best iphone 7 cases”: 7,773 * 2% = 155

Let me put all these numbers into a spreadsheet:

16-difference-in-organic-clicks

And this is how we went from a 5x difference in organic search traffic potential between these keywords to 2.6x difference.

Better understanding of searchers’ behaviour

We were super excited while playing with all these new metrics ourselves and we’re hoping that you’re going to love them too and adopt them in your keyword research.

And for those still puzzled by all this, here’s a simple illustration of the metrics that Keywords Explorer 2.0 shows you:

17-how-people-search

That is a very simplified model, as we’ve only started scratching the surface of the actual complexity of searchers’ behaviour.

But we already have quite a few ideas for new reports and metrics that will give you even better idea of how people search for your target keywords.

#

With ~3.1 Billion keywords in our database (and monthly infusions of fresh ones) it’s no surprise that Keywords Explorer 2.0 will give you massive amounts of keyword ideas to sift through.

For the keyword “blogging” the tool will give you over 14k different ideas:

24-keyword-ideas-in-ahrefs

And because you have Search Volume on all of them and cached Keyword Difficulty on the very best of them — you can use filters to quickly drill down this huge list of suggestions to a small one with only the very best.

Multiple seed keywords

After many years of using Google Keyword Planner tool SEOs got used to entering multiple “seed” keyword for generating a list of relevant keyword ideas.

We couldn’t allow ourselves to disrupt this use case, so in Keywords Explorer 2.0 we can use up to 10 seed keywords for generating keyword ideas for you.

Which means that if I add keywords “snorkeling” and “kitesurfing” to my initial seed keyword “blogging”, I’ll get almost 3x more keyword ideas:

25-more-keyword-ideas

Fun Fact: according to their search volumes, snorkeling is the most popular of these three activities.

4 types of keyword ideas

If you click on the report called “All”, we’re going to just give you all the keyword ideas we have.

But actually there are four ways in which we generate them (we’ll soon add more):

1. Phrase match — we search our huge database for any keyword phrases that would contain the exact seed phrase in them.

So if your seed keyword is “adorable cat”, we’ll give you keywords like:

  • how to get rid of an adorable cat;
  • adorable cat ruined my life;
  • how to know if your adorable cat is plotting to kill you.

2. Having same terms — very similar to “Phrase match”, but this time we’ll search our database for any keyword phrases that contain any terms of your seed keyword in any order.

Let’s continue with our “adorable cat” example:

  • why my cat is not as adorable as advertised;
  • why my adorable wife hates my cat;
  • why all animals but my cat are adorable.

3. Also rank for — this one is my favourite. We pull top 10 ranking pages for your “seed” keyword and see what other keywords these pages rank for.

So if you put “james bond” and go to “Also rank for” report, you’ll see a lot of suggestions associated with the latest movie called “Spectre”:

26-also-rank-for-keyword-ideas

4. Search suggestions — this list of keyword ideas will basically give you the keywords that Google is suggesting via their “autosuggest” feature when you type a search query.

The results of this report will be almost identical to what “Phrase match” shows, with the only exception being that it will contain keyword ideas that include your seed keyword without spaces:

27-search-suggestions

#

When researching organic search traffic of their competitors, these days SEOs look at the “Top Pages” report way more often than “Organic keywords” report.

The reason being that a single page can rank for hundreds of closely related keywords and bring in a ton of traffic from search.

Here’s a cool example that illustrates this “topics over keywords” concept:

18-top-pages-of-moz

On that screenshot above I’m looking at the “Top Pages” of Moz Blog and I see that one of their articles ranks for over 1K different keywords. The most polular among these keywords is “how to choose a domain name” so I consider it the main topic of this post.

So basically the idea of “topics over keywords” is that you don’t need to create an individual article to target each of these keywords. You can rank for all of them with a single post.

In Keywords Explorer 2.0 we tried to reverse-engineer this concept.

So if I take all these “other” keywords that article from Moz ranks for, and put them into Keywords Explorer 2.0, it will show me the “parent topic” that these keywords fall into:

19-parent-topic-in-ahrefs

We believe that this “Parent topic” feature is a huge thing for keyword research, because it can tell you how many pages you need to create in order to target all the keywords that you wish to rank for.

IMPORTANT: we’ve just discovered a few bugs in this feature, so it may not work very well as of today. We’re going to solve this in a about a week from now and remove the “beta” icon so that you know the feature is stable.

#

In this release we didn’t change anything about our KD score. Mostly because we believe it is quite good as it is (which was also confirmed by third party tests).

I have already published a pretty detailed post about our Keyword Difficulty score and how to apply it in your keyword research, but I’m still going to re-hash some of the most important points here:

KD is based on backlink data

Ever since we first introduced our Keyword Difficulty score there has been a ton of dicussion about what factors we should or should not be taking into account when calculating it.

And the fun thing is — there’s no single opinion on how much each individual factor should influence keyword difficulty score. What seemed hyper important for some SEOs was secondary at best for others.

We also ran a study of 2 Million keyword searches to investigate how different on page SEO factors correlate with Google rankings. Here’s what we learned from it:

00-backlink-factors-on-page-factors

As you can tell from this graph, the correlation of backlink factors (both page-level and domain-level) massively outweighs the major on page SEO factors.

And this is how we settled with the decision to calculate our KD score based on backlinks only.

SIDENOTE.

Ahrefs’ Keyword Difficulty won’t rely solely on backlinks forever. We already have quite a few cool ideas on how to make our score even better. And some of them do imply looking at on page factors.

KD scale is not linear

As I was watching screencasts recorded by our beta testers I noticed that many of them tried to compare our KD score with what other tools were reporting.

You can’t compare Ahrefs KD score to what any other tool shows, because our scale is most likely very different from theirs:

00-ahrefs-keyword-difficulty-scale

As you can see from the image above, our scale is not linear and mostly consists of hard and super hard keywords.

So if some other tool gives you KD score of “40” and you read it as “Medium” difficulty, on Ahrefs’ scale “40” means “Hard”.

And because we only rely on backlinks when calculating KD score, our scale is super easy to understand:

KD 0 = 0 Ref. Domains
KD 10 = 10 Ref. Domains
KD 20 = 22
KD 30 = 36
KD 40 = 56
KD 50 = 84
KD 60 = 129
KD 70 = 202
KD 80 = 353
KD 90 = 756

Basically our KD score shows you the average number of referring domains that pages ranking on the front page of Google have. So the odds are, you’re going to need more or less the same number of websites linking to your page if you want to outrank your competition.

Cached KD

Once you start using Keywords Explorer 2.0 you’re going to notice that a lot of the keywords have a grey KD score, while others show nothing until you click the “Get Metrics” button:

20-grey-keyword-difficulty

Here’s how it works:

Automatically updating KD scores for all 3.1 Billion keywords in our database is impossible both technically and financially.

So we filtered ~10% of the very best keywords (~300 Million) and these are the ones that get automated updates of their Keyword Difficulty score.

But once you click “Get metrics” we’ll pull the most recent data and update this number.

Plans to improve KD

We do believe that our KD score measures the difficulty of a keyword quite accurately compared to what other tools show.

But at the same time we do understand that there’s a massive room for improvement and we can make this score a lot better. Which we plan to do in one of the future updates of Keywords Explorer.

So if you happen to have any suggestions of which factors we should be taking into account (and how much influence they should have on resulting KD score) please email me and I’ll be happy to discuss it with you.

#

No matter how far we get in fine-tuning our KD score, the best way to determine the real difficulty of a keyword has always been (and probably will always be) a look at who ranks on the front page of Google.

Nothing beats the experience (and intuition) of a professional SEO and all we can help with at this point is bring all the SEO metics we have to the SERP:

21-serp-analysis-for-keyword-marvel

This is how the front page of Google looks like for the keyword “Marvel“.

And as you can tell, Ahrefs Keywords Explorer gives you a ton of very useful data right away:

  • How many organic results are there;
  • How many different SERP features are there;
  • What are the domain level and page level backlink metrics;
  • What is the estimated organic traffic to each page;
  • How many keywords each page ranks for;
  • What is the top keyword each page ranks for;
  • How many shares on Facebook and Google plus each page has.

There are many more things we could potentially show here, but we’ve saved them for future updates of the tool.

#

I just explained how we update Keyword Difficulty for ~300 Million of the very best keywords in our database and store it in our cache, so that you can see it at a glance.

Well, because we closely monitor these keywords, we can also show you the positions history of pages that currently rank in top5. It looks like this:

22-serp-position-history-ahrefs

This helps a lot in figuring out if the SERP is stable and very hard to penetrate or if Google is mixing pages all the time trying to figure out which combination serves searchers best.

For instance, on the above screenshot you see SERP position history for “keyword research”. The top 4 ranking pages didn’t change since June 2015, but we managed to penetrate at #5 spot with our own article on keyword research.

Here’s another fun graph:

23-serp-position-history

It’s the top 5 for the keyword “iPhone 7”, and as you can probably tell it is super unstable at the moment and all the pages that are currently top5 are brand new.

#

To be honest I’m somewhat embarrassed that we didn’t have keyword lists in the very first release of Keywords Explorer, because this feature is so darn obvious.

But now we’ve fixed our mistake:

28-add-keywords-to-lists

The keyword lists that you create will be available on the home screen of Keywords Explorer 2.0 and also in the left navigational panel within the tool.

#

The fact that we’ve released this tool doesn’t mean that we’re done with it (and I’m pretty sure old Ahrefs customers know that very well).

So let me tease 3 cool features that are going to be released very soon:

Questions” filter — a lot of our users were asking for it, so it’s already in the works.

Filter keywords by “SERP features” they have — whether you would like to see which keywords have many features in their SERP, or find which keywords have a very specific feature — we’re about to let you do that.

Where I rank — enter any domain name and we’ll show you if we saw it ranking in top100 for any of the keywords in a given report. It will look like this:

29-where-i-rank-feature

Tell Us What You Think

As you can probably tell from this review alone, we worked super hard trying to impress you with this new tool.

It turned out to be so powerful and feature-rich that I struggled to keep this overview under 4000 words. In fact I had to omit quite a few cool things we have in this tool for the sake of keeping it short (I’m sure you’ll find them on your own).

So please play with Keywords Explorer 2.0 and tell us what you think about this new tool.

This is actually an “early release” and we’re still polishing some bugs here and there and adding new features. But we are absolutely keen to know your opinion on what we have so far.

Link to original article: