At the very simplest form, a search engine is a mechanism to provide an answer (search results) to a question (search query). For example, Google is a search engine, where you submit a search query (e.g. “chocolate chip cookie recipe”) and answers are displayed on a results page.
Before a search engine can deliver search results for your search query, though, it first must make a master list of every possible answer (to questions that it doesn’t yet know). The process of making this master list of answers is called crawling.
You may have heard that search engines may “crawl” your site. A crawler (also called a spider or bot) is an automated script that visits a page on your website, scans everything, including the content, images, videos, or other media, and creates a cache of that site. A modern search engine crawler can also see your site just as you, as an internet user, will see it in your web browser (with a few minor exceptions). Additionally, it will typically follow links throughout your site to visit other pages on your site and even other websites that you’ve linked to. In fact, it’s possible that the crawler discovered your website by a link from another website.
While it’s possible to block search engines from crawling your site, it’s typically not advised, except in cases where you may still be working on your brand new website, or want certain pages to no longer be indexed and searchable. This can have a negative effect in that crawlers may give up and stop attempting to crawl your site, or may try so infrequently that the content in the master list, or index, is not current.
Once a crawler visits a site and downloads all of the relevant information that it can, it stores that site’s data into a search index. An index is a specialized database (or, more accurately, multiple linked databases for different parts and types of data from your site). These databases contain information from every site that a search engine crawler could conceivably access.
According to the most recent claims, Google alone is indexing around “hundreds of billions of webpages” with an index size of over 100 billion gigabytes, but has knowledge of around 130 trillion websites.
When you type something into a search engine, in a few seconds, you’ll be forwarded to a page called a search engine results page (SERP, for short). In those few seconds between clicking “Search”, and when you see the results, is where the real magic happens.
A search engine will take what you’ve typed in, called a query, and dissect it into a number of arrangements. For example, the search query “best vegan tacos” contains a number of potential phrases that could be used to provide search results. To the best of its ability, the search engine will determine your intent with your query, rank a set of pages (out of the hundreds of billions indexed), and provide a list of the top results.
Based on your query, and the determined intent of your search query (along with over 200 other factors for the sites’ pages themselves), the results delivered to your browser are ranked. Ranking is the process in which a certain website’s page shown in the search results is ordered by the search algorithm’s perceived “importance” of that result to the query.
In our “best vegan tacos” search, we’ll pretend for a moment that you’ve made this search from your mobile device while on your mobile provider’s network. A search engine may think that since you’re searching from a mobile device, and not on wifi, that you may be looking for a restaurant. It determines your intent is for a search result to show you nearby restaurants. Your query asked for vegan tacos, so the search algorithm will attempt to limit its results to restaurants that have vegan tacos listed in their online menu, and have been referenced with the word “best” on other sites (like ones that may have large amounts of online restaurant reviews).
If you live in a larger city, even these limitations may have a large number of possible results. Since you’re on a mobile device, even if you don’t have GPS activated, your rough location can be determined by the cell tower you’re currently connected to, so the search algorithm will further limit those results to places that are within a certain radius of you, while typically highlighting those locations on a map-based search result.
The end product that you see is the SERP, or search engine results page. Keeping in tune with our example, you’re likely to see a map result showing several restaurants that hopefully serve the style of tacos you’ve asked for. You’re also likely to see links to many of those restaurants’ profiles on other sites that are review-based, as well as possibly links to the restaurants themselves.
On a rare occasion, where the intent may not be as clear, it’s entirely possible to see a wildcard result. For this example, a recipe for a vegan taco may be included. Should you have immediately clicked on that wildcard result, the search engine would use that knowledge to perhaps alter its delivery for an identical or very similar query in the future.
Consider the topic of the page or post that you selected from the previous exercise. Perform a search using terms or phrases that you would use to find information about the topic discussed in your content. Did your search results display webpages that are similar to the content on your page/post? The goal of this exercise is to find the search words that display links on the SERP for webpages that have content similar to your page/post. If necessary, repeat your search using different terms until the content found on the SERP better aligns with the topic you write about in your content.
Back to top