How Search Engines
Work
From first projects to enterprise scale, Ahrefs’ plans help your business stay discoverable in search, AI, and beyond—powered by the world’s second-most active crawler and 10+ years of web-scale data.
By Joshua Hardwick
Former Head of Content at Ahrefs
Contributors
Search engines work by crawling billions of pages using web crawlers. Also known as spiders or bots, crawlers navigate the web and follow links to find new pages. These pages are then added to an index that search engines—and even AI assistants, like ChatGPT—pull results from.
Understanding how search engines function is crucial if you’re doing SEO. After all, it’s hard to optimize for something unless you know how it works.
That’s what you’ll learn in this guide.
Search engine basics
Let’s begin by exploring what search engines are, why they exist, and how they make money.
What are search engines?
Search engines are searchable databases of web content. They’re made up of two main parts:
- Search index. A digital library of information about webpages.
- Search algorithm(s). Computer program(s) tasked with matching results from the search index.
What is the aim of search engines?
Every search engine aims to provide the best, most relevant results for users. That’s partly how they gain market share.
How do search engines make money?
Search engines have two types of search results:
- Organic results from the search index.You can’t pay to be here.
- Paid results from advertisers. You can pay to be here.
Each time someone clicks a paid search result, the advertiser pays the search engine. This is known as pay-per-click (PPC) advertising, and it’s why market share matters. More users mean more ad clicks and more revenue.
Search Engines Make Money from Ads
How search engines build their indexes
Each search engine has its own process for building a search index. Below is a simplified version of the process Google uses.1
How Google Builds Its Search Index
Let’s break it down.
URLs
Everything begins with a known list of URLs. Google discovers these in many ways, but the three most common are:
- From backlinks.Google has an index of hundreds of billions of webpages.2 If someone links to a new page from a known page, Google can find it from there.
- From sitemaps.Sitemaps tell Google which pages and files you think are important on your site.3
- From URL submissions.Google lets site owners request crawling of individual URLs in Google Search Console.
Crawling
Processing and rendering
Processing is where Google works to understand and extract key information from crawled pages. To do this, it has to render the page, which is where it runs the page’s code to understand how it looks for users.
Nobody outside of Google knows every detail about this process. But it doesn’t matter. All we really need to know is that it involves extracting links and storing content for indexing
Indexing
Indexing is where processed information from crawled pages gets added to the search index.
The search index is what you search when you use a search engine. AI assistants like ChatGPT, Claude, and Gemini also use search indexes to find webpages. That’s why getting indexed in major search engines like Google and Bing is so important. Users can’t find you unless you’re in the index.
How search engines rank pages
Each search engine has its own process for building a search index. Below is a simplified version of the process Google uses.1
How Google Builds Its Search Index
Let’s break it down.
URLs
Everything begins with a known list of URLs. Google discovers these in many ways, but the three most common are:
- From backlinks.Google has an index of hundreds of billions of webpages.2 If someone links to a new page from a known page, Google can find it from there.
- From sitemaps.Sitemaps tell Google which pages and files you think are important on your site.3
- From URL submissions.Google lets site owners request crawling of individual URLs in Google Search Console.
Crawling
Processing and rendering
Processing is where Google works to understand and extract key information from crawled pages. To do this, it has to render the page, which is where it runs the page’s code to understand how it looks for users.
Nobody outside of Google knows every detail about this process. But it doesn’t matter. All we really need to know is that it involves extracting links and storing content for indexing
Indexing
Indexing is where processed information from crawled pages gets added to the search index.
The search index is what you search when you use a search engine. AI assistants like ChatGPT, Claude, and Gemini also use search indexes to find webpages. That’s why getting indexed in major search engines like Google and Bing is so important. Users can’t find you unless you’re in the index.
How search engines personalize results
Each search engine has its own process for building a search index. Below is a simplified version of the process Google uses.1
How Google Builds Its Search Index
Let’s break it down.
URLs
Everything begins with a known list of URLs. Google discovers these in many ways, but the three most common are:
- From backlinks.Google has an index of hundreds of billions of webpages.2 If someone links to a new page from a known page, Google can find it from there.
- From sitemaps.Sitemaps tell Google which pages and files you think are important on your site.3
- From URL submissions.Google lets site owners request crawling of individual URLs in Google Search Console.
Crawling
Processing and rendering
Processing is where Google works to understand and extract key information from crawled pages. To do this, it has to render the page, which is where it runs the page’s code to understand how it looks for users.
Nobody outside of Google knows every detail about this process. But it doesn’t matter. All we really need to know is that it involves extracting links and storing content for indexing
Indexing
Indexing is where processed information from crawled pages gets added to the search index.
The search index is what you search when you use a search engine. AI assistants like ChatGPT, Claude, and Gemini also use search indexes to find webpages. That’s why getting indexed in major search engines like Google and Bing is so important. Users can’t find you unless you’re in the index.
Key takeaways
- A search engine consists of two main parts: index and algorithms.
- To build its index, it crawls known pages and follows links to find new ones.
- The aim of search algorithms is to return the best, most relevant results.
- Search result quality is important for building market share.
- Nobody knows all of Google’s ranking factors for organic results.
- Key ranking factors include backlinks, relevance, and freshness.
- Google personalizes its results based on your location, language, and search history.
References
- “Organizing Information — How Google Search Works”. Google. Retrieved 16th August 2022.
- “Learn about sitemaps”. Google. Retrieved 16th August 2022.
- “Googlebot”. Google. Retrieved 16th August 2022.
- “Search Engine Market Share Worldwide”. Statcounter. Retrieved 16th August 2022.
- “Google Q&A+ #March”. YouTube. Retrieved 16th August 2022.
- “96.55% of Content Gets No Traffic From Google. Here’s How to Be in the Other 3.45%”. Ahrefs. Retrieved 1st December 2023.
- “CloudFlare Radar”. CloudFlare. Retrieved 16th August 2022.
- “Ranking Search Results — How Google Search Works”. Google. Retrieved 16th August 2022.
- “Using site speed in web search ranking”. Google. Retrieved 16th August 2022.
- “Using page speed in mobile search ranking”. Google. Retrieved 16th August 2022.
- “Mobile-first indexing best practices”. Google. Retrieved 16th August 2022.
- “Find & control your Web & App Activity”. Google. Retrieved 16th August 2022.
Guide by
Former Head of Content at Ahrefs (or, in plain English, I’m the guy responsible for ensuring that every blog post we publish is EPIC).
