Crawling

The process where search engine bots (crawlers) discover and scan web pages by following links. Crawlers read page content, code, and structure to understand what the page is about. Crawling precedes indexing—pages must be crawled before they can be indexed.