> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/algolia/docsearch/llms.txt
> Use this file to discover all available pages before exploring further.

# Getting Started with the DocSearch Crawler

> Learn how the DocSearch crawler works and how it indexes your documentation

## Overview

The DocSearch crawler is the backbone of the DocSearch service. It automatically extracts content from your documentation website, transforms it into searchable records, and pushes it to an Algolia index. This enables fast, relevant search results for your users.

<Note>
  DocSearch now leverages the powerful [Algolia Crawler](https://www.algolia.com/products/search-and-discovery/crawler/), which offers a web interface to create, monitor, edit, and start your crawlers.
</Note>

## How It Works

The DocSearch crawler follows a systematic process to index your documentation:

<Steps>
  <Step title="Discover Pages">
    The crawler starts from your configured `start_urls` and recursively follows internal links to discover all documentation pages on your site.
  </Step>

  <Step title="Extract Content">
    Using CSS selectors (or XPath), the crawler extracts structured content from each page's HTML markup. It identifies headings (h1, h2, h3, etc.) to build a hierarchy and extracts text content from paragraphs and lists.
  </Step>

  <Step title="Build Records">
    The extracted content is transformed into JSON records with a hierarchical structure:

    * `lvl0` - Typically the page title or h1
    * `lvl1` - Usually h2 headings
    * `lvl2` - Usually h3 headings
    * `lvl3`, `lvl4`, `lvl5` - Deeper heading levels
    * `text` - Paragraph and list content
  </Step>

  <Step title="Index to Algolia">
    The records are pushed to your Algolia index, replacing the previous version. This ensures your search is always up-to-date with your latest documentation.
  </Step>
</Steps>

## Crawler Architecture

### Legacy Python Crawler

The original DocSearch scraper is a Python-based tool inspired by the [Scrapy](https://scrapy.org/) framework. It's open source and available in the [algolia/docsearch-scraper](https://github.com/algolia/docsearch-scraper) repository.

<Card title="Docker Image" icon="docker" href="https://hub.docker.com/r/algolia/docsearch-scraper">
  The crawler is packaged as a Docker image for easy deployment
</Card>

### Modern Algolia Crawler

For the free DocSearch program, Algolia now uses its modern Crawler infrastructure, which provides:

* **Web Interface**: Manage crawlers from the [Crawler Dashboard](https://dashboard.algolia.com/crawler)
* **Live Editor**: Edit and test your configuration in real-time
* **Monitoring**: Track crawl statistics, errors, and performance
* **Scheduling**: Automatic weekly crawls by default
* **Manual Triggers**: Start crawls on-demand when you update your docs

## Content Extraction

### HTML Structure Requirements

The crawler works best with well-structured HTML markup:

```html theme={null}
<article>
  <h1>Getting Started</h1>
  <p>Welcome to our documentation.</p>
  
  <h2>Installation</h2>
  <p>Install the package using npm:</p>
  <code>npm install example</code>
  
  <h2>Configuration</h2>
  <h3>Basic Setup</h3>
  <p>Configure your application...</p>
</article>
```

### Selector-Based Extraction

The crawler uses CSS selectors to target specific elements:

```json theme={null}
{
  "selectors": {
    "lvl0": "article h1",
    "lvl1": "article h2",
    "lvl2": "article h3",
    "lvl3": "article h4",
    "text": "article p, article li"
  }
}
```

<Note>
  The `text` selector is mandatory. We highly recommend setting at least `lvl0`, `lvl1`, and `lvl2` for optimal search relevance.
</Note>

## Crawl Frequency

For sites enrolled in the free DocSearch program:

* **Default Schedule**: Crawls run once per week automatically
* **Manual Crawls**: Trigger on-demand from the Crawler Dashboard
* **Updates**: Changes to your documentation are reflected after the next crawl

<Warning>
  If you need real-time indexing or more frequent crawls, consider running your own crawler or upgrading to a paid Algolia plan.
</Warning>

## JavaScript Rendering

By default, the crawler expects server-side rendered content. If your site uses client-side rendering:

```json theme={null}
{
  "js_render": true,
  "js_wait": 2
}
```

<Warning>
  Client-side crawling is significantly slower than server-side crawling. We strongly recommend implementing server-side rendering for your documentation.
</Warning>

## Crawl Scope

### Following Links

The crawler automatically:

* ✅ Follows all internal links within your domain
* ✅ Respects `start_urls` as entry points
* ❌ Does not follow external links to other domains
* ❌ Stops at URLs matching `stop_urls` patterns

### Example Configuration

```json theme={null}
{
  "index_name": "my-docs",
  "start_urls": [
    "https://example.com/docs"
  ],
  "stop_urls": [
    "https://example.com/docs/archive",
    "https://example.com/blog"
  ]
}
```

## Data Privacy

### What Gets Indexed

The crawler extracts:

* Text content from your documentation pages
* Heading structure and hierarchy
* URL paths and anchor links
* Custom metadata (if configured)

The crawler does NOT extract:

* Images (only alt text if specified)
* CSS or JavaScript code
* Forms or interactive elements
* Content behind authentication (unless credentials provided)

### Data Storage

All indexed data is:

* Stored on Algolia's global infrastructure
* Replicated across regions for performance
* Subject to [Algolia's privacy policy](https://www.algolia.com/policies/privacy)

## Next Steps

<CardGroup cols={2}>
  <Card title="Configuration" icon="gear" href="/crawler/configuration">
    Learn how to configure the crawler for your documentation
  </Card>

  <Card title="Apply to DocSearch" icon="paper-plane" href="/crawler/apply">
    Get free crawler hosting for your open source project
  </Card>
</CardGroup>

## Troubleshooting

### No Results After Crawl

1. Check that your selectors match your HTML structure
2. Verify that pages are accessible (not behind authentication)
3. Ensure server-side rendering is enabled
4. Review crawl logs in the Crawler Dashboard

### Incomplete Indexing

1. Check `stop_urls` configuration
2. Verify all pages are linked from `start_urls`
3. Look for broken internal links
4. Confirm sitemap URLs (if using sitemap-based crawling)

### Poor Search Results

1. Review selector hierarchy (lvl0 through lvl5)
2. Exclude irrelevant content using `selectors_exclude`
3. Configure synonyms for common terms
4. Adjust `page_rank` for important pages

<Note>
  For technical support with the Algolia Crawler, reach out via the [Algolia support page](https://algolia.com/support).
</Note>