When I get given a new web­site to work on, one of the first things I like to do is scrape and sieve through their com­peti­tors’ backlinks.

As well as flag­ging rel­e­vant prospects you might be able to get links from for the new prospect, it also give you a really good feel of the kind of nat­ural links and forced link build­ing strate­gies that exist in the field you’re about to start toil­ing in.

To man­u­ally put each site into OSE or Majes­tic can take a lot of time and isn’t a very thor­ough means of working.

(Some of the best links are buried deep in back­link reports and can be easy to over­look when you’ve been at the grind­stone all day.)

Here is how I quickly scrape and sieve through my com­peti­tors’ back­link pro­file when I begin link build­ing projects.

A lot of the tech­niques here are bor­rowed from Ontolo’s link build­ing video series. If you’ve not seen these and you’re inter­ested in link build­ing, I rec­om­mend you clear a few  hours in your sched­ule and watch them as soon as possible.

Step 1:

Firstly, you need a list of your site’s competitors.

Look who is rank­ing for all the terms you are opti­mis­ing for as well as many rel­e­vant, related terms that you come up with.

I nor­mally like to cherry pick the sites I want here, but if you want to get every­thing quickly you can use the scraper tool to take the SERP result pages to a Google Doc.



Tip: If you work in an agency, always ask your client who their offline com­peti­tors are as well as online.

Do not dis­miss com­peti­tors who aren’t actively opti­mis­ing their site. Often, the links to their site have all occurred nat­u­rally and, from my expe­ri­ence, are an oppor­tu­nity that has not yet been mined by other SEOs.

Step 2:

Next you need to need a com­pre­hen­sive list of all of these sites’ backlinks.

To get this data, you’ll need Majes­tic or OSE, both of which require paid sub­scrip­tions. Per­son­ally, I pre­fer to use Majes­tic as I find it casts its net a lit­tle wider than OSE.

(Although this does mean you’ll have to wade through a lot more gumpf.)

Export each of your com­peti­tor sites’ back­links to CSV and then stack them on top of each other.

Make sure you remem­ber to delete the top line from each of the exports when you are stack­ing them.

(This is impor­tant for when you remove dupli­cate URLs in step 4.)

Step 3:

Now you know who is link­ing to all your com­peti­tors, you should have a tens of thou­sands of link­ing domains.

As you can imag­ine, a lot of domains will occur more than once. We need to remove these if we want to save time when siev­ing through them.

To do this, you need to use Ontolo’s Remove Dupli­cates tool, which is free.

Copy and paste your prospects into the tool and click remove duplicates.

This will give you a new list of domains that only occur once. Copy this back into Excel.

Step 4:

Next we need to actu­ally review the sites to see which are worth look­ing into and which we can discard.

To copy each each url in man­u­ally to the browser would take an age and be mind numb­ingly tedious.

You can speed up the process using Ontolo’s url reviewer.

This tool allows you to simul­ta­ne­ously open sev­eral urls. You can then skip through them using the “Ctrl +W” key­board short-cut.

Ross Hud­gens rec­om­mended turn­ing off images here to speed up page load time.

Per­son­ally, I pre­fer to keep them on and review less sites in smaller batches, as it means I’m less likely to miss good sites.

This, might be due to expe­ri­ence and I imag­ine the more you do this, the eas­ier it is to iden­tify good and bad sites.

Open­ing sev­eral urls at once really will give your com­puter a work­out, so close all other pro­grammes when you’re doing this.

Step 5:

The final step then is what to do with sites we deem valu­able and we want to try accrue links from.

I use the CRM soft­ware Buz­zStream to man­age the sites that I mark as rep­re­sent­ing a link­ing opportunity.

Each time I decide an oppor­tu­nity exists I add it to Buz­zStream and tag it with what­ever the oppor­tu­nity is, for exam­ple: guest post, sub­mit con­tent, rec­i­p­ro­cal etc.

(See my Bet­ter Link Build­ing with Buz­zStream post for more on this.)

This also makes it eas­ier to organ­ise my time. For exam­ple, if I have  9 hours to work on a site I can spend 3 hours siev­ing through oppor­tu­ni­ties, 3 hours mak­ing the out­reach and then 3 hours fol­low­ing up.


That’s it then, chas­ing your com­peti­tors back­links is just one string of your bow; how­ever, it’s gen­er­ally a pretty good place to start and get ideas.

I appre­ci­ate not every­one will have access to the paid tools here, so if any­one can sug­gest alter­na­tives or has any other tips they’d like to add, please leave them in the com­ment below!

Post By Michael Smith (31 Posts)

Michael works at 9xb Dig­i­tal Agency in Har­ro­gate, Mon­day to Fri­day. Click here to fol­low him on Twit­ter or have a look a his Google+ pro­file.

Web­site: →


Last updated by at .

Tagged with:

One Response to How to Quickly Scrape and Sieve Your Competitors’ Backlinks

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title="" rel=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>