You deleted how many pages? 130M! And here’s why.

Delete your pages and rank higher in search

For years I’ve been telling SEO that if you are looking for a way to optimise your website from a technical SEO point of view you should consider deleting some of your website pages. 

I probably sound crazy to you! But hear me out.

Index Bloat and Crawl Budget

Index bloat and crawl budget are real SEO problems that prevent Google from finding and indexing our most valuable content. When we have an excess of (potentially indexable) pages – most of the time low-quality pages – we are not just wasting our crawl budget but also confusing the search engines on how to treat our pages and opening up avenues outside of our control in which the Googlebot can get easily trapped.

A good example would be poor internal linking, slow load speeds, a vast amount of orphan pages, deindexing of important pages, the list goes on and on. Believe it or not, most of these issues come as a result of having a large number of pages on your website. Believe it or not, most of these problems can be fixed by simply removing the clutter from your website. 

Why do I have so many pages on my website?

1. Duplicate content. Unintentional internal duplicate content is one of the main reasons for having a large number of pages on your website. This is because technical SEO mistakes can generate a large number of URL variations that end up crawled and indexed by Google. For example, very often we see eCommerce websites using faceted navigation that creates billions of useless near-duplicate pages resulting in wasted crawl budget and indexation of low-quality junk pages.
2. Thin content (technical SEO). Unintentional thin content is so easily preventable and yet we see so many websites are struggling with this ‘SEO disease’. Let me give you a few examples: the search function on your website will generate thousands and thousands of thin content pages while helping your visitors to find your products. This is normal and desirable internal search behaviour but you shouldn’t allow Google to index these pages. Fixing this issue is as easy as adding a Robots.txt disallow or a noindex directive. Another good example would be category pages that contain only one product. Recently I worked on a website that had more than a thousand category pages displaying only one product. How difficult is to canonicalise these ’emtpy’ categories to its product page, e.g., if there are less than 2 products, then canonicalise to the displayed product. Job done. 
3. Pagination. Recently Google retired the rel=prev/rel=next for paginated pages so now the search giant is looking at every paginated page as a unique page resulting in pagination becoming one of the most common sources of index bloat. Pagination is a complex subject so if you want to learn how to deal with pagination I strongly recommend this article: Pagination for SEO: how to make your eCommerce site easier to crawl and index. 

So now that we know how having a large number of pages affects our SEO let’s talk about Brighton SEO and David Lewis’ talk on optimising Trainline’s SEO by deleting 130 million pages. 

During this talk at BrightonSEO, David Lewis talks about:

In this talk, David Lewis will tell the story of Trainline’s multi-year crawlability project code-named Black Widow. Watch the video above and listen to the story of why millions and millions of Trainline pages were deleted, the results that followed and what we learnt along the way.

You deleted how many pages? 130M and heres why – brightonSEO Autumn 2021 from David Lewis

Who is David Lewis

David Lewis has worked in SEO for 6 years, 3 of those at Trainline where he is the Senior Technical SEO Manager. During the years he has established himself as the in-house specialist on crawl optimisation, JavaScript SEO and landing page performance. 

David works across the entire business from SEO and engineering through to growth marketing and platform operations to enable growth and drive performance. David is an advocate of personal development, loves working out in the gym, meditating and travelling the world.

About BrightonSEO 

BrightonSEO – is a major search marketing event in the UK. One of our favourite events of the year, This is a superb conference for search marketing professionals, novice or expert. BrightonSEO is a chance to learn from some of the best minds in search, and then rub shoulders with them at one of the friendliest, and largest, gatherings of Digital Marketers in Europe.