> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/algolia/docsearch/llms.txt
> Use this file to discover all available pages before exploring further.

# Indexing

> How the DocSearch crawler indexes your documentation

DocSearch uses the Algolia Crawler to automatically extract content from your documentation and build a searchable index. Understanding how indexing works helps you optimize your documentation for better search results.

## Crawler Overview

The Algolia Crawler is a web scraper that:

* Visits your documentation pages
* Extracts content using CSS selectors
* Creates structured records for Algolia
* Runs on a schedule (default: weekly)
* Respects robots.txt and crawl limits

<Note>
  If you're using the DocSearch program, Algolia manages the crawler for you. For custom implementations, you configure it yourself.
</Note>

## How Content is Indexed

### Record Structure

The crawler creates hierarchical records for each page section:

```json theme={null}
{
  "objectID": "page-url#heading-id",
  "hierarchy": {
    "lvl0": "Documentation",
    "lvl1": "Getting Started",
    "lvl2": "Installation",
    "lvl3": "Using npm",
    "lvl4": null,
    "lvl5": null,
    "lvl6": null
  },
  "content": "Install DocSearch using npm or yarn package manager",
  "url": "https://example.com/docs/installation#npm",
  "anchor": "npm"
}
```

### Hierarchy Levels

DocSearch organizes content into levels:

* **lvl0**: Top-level category (e.g., product name, section)
* **lvl1**: Page title or main heading (`h1`)
* **lvl2-6**: Subheadings (`h2-h6`)
* **content**: Paragraph text, list items, code snippets

## Record Extractor Configuration

The `recordExtractor` defines how content is extracted from your pages:

```javascript theme={null}
recordExtractor: ({ helpers }) => {
  return helpers.docsearch({
    recordProps: {
      lvl0: {
        selectors: "header h1",
      },
      lvl1: "article h2",
      lvl2: "article h3",
      lvl3: "article h4",
      lvl4: "article h5",
      lvl5: "article h6",
      content: "main p, main li",
    },
  });
}
```

<Steps>
  <Step title="Define Selectors">
    Specify CSS selectors for each hierarchy level and content elements.
  </Step>

  <Step title="Extract Records">
    The helper function processes the page and creates Algolia records.
  </Step>

  <Step title="Index Records">
    Records are sent to your Algolia index for searching.
  </Step>
</Steps>

## Advanced Extraction Patterns

### Fallback Selectors

Use multiple selectors as fallbacks:

```javascript theme={null}
recordProps: {
  lvl0: {
    selectors: [".page-title h1", ".header h1", "h1"],
  },
  lvl1: "article h2",
  content: [
    ".main-content p, .main-content li",
    ".content p, .content li",
    "p, li"
  ],
}
```

The crawler uses the first selector that returns results.

### Default Values

Provide fallback text when selectors don't match:

```javascript theme={null}
recordProps: {
  lvl0: {
    selectors: "header h1",
    defaultValue: "Documentation",
  },
  language: {
    selectors: "html[lang]",
    defaultValue: ["en"],
  },
}
```

### DOM Manipulation with Cheerio

Remove unwanted elements before extraction:

```javascript theme={null}
recordExtractor: ({ $, helpers }) => {
  // Remove elements you don't want indexed
  $(".edit-link").remove();
  $(".footer").remove();
  $(".sidebar").remove();
  $(".code-copy-button").remove();

  return helpers.docsearch({
    recordProps: {
      lvl0: { selectors: "header h1" },
      lvl1: "article h2",
      lvl2: "article h3",
      content: "main p, main li",
    },
  });
}
```

## Faceting and Filtering

Index custom attributes for filtering:

```javascript theme={null}
recordProps: {
  lvl0: { selectors: "header h1" },
  lvl1: "article h2",
  content: "main p, main li",
  // Custom attributes for filtering
  version: {
    selectors: ".version-badge",
    defaultValue: ["latest"],
  },
  language: {
    selectors: "html[lang]",
    defaultValue: ["en"],
  },
  tags: ".article-tags .tag",
}
```

Then filter in your frontend:

```tsx theme={null}
<DocSearch
  appId="YOUR_APP_ID"
  apiKey="YOUR_SEARCH_API_KEY"
  indexName="YOUR_INDEX_NAME"
  searchParameters={{
    facetFilters: ['version:v4', 'language:en'],
  }}
/>
```

<Info>
  Custom attributes are available as facets in your Algolia index for filtering search results.
</Info>

## Boosting Records with pageRank

Increase the ranking of important pages:

```javascript theme={null}
recordExtractor: ({ $, helpers, url }) => {
  const isGettingStarted = url.pathname.includes('/getting-started');
  const isReference = url.pathname.includes('/api-reference');
  
  return helpers.docsearch({
    recordProps: {
      lvl0: { selectors: "header h1" },
      lvl1: "article h2",
      content: "main p, main li",
      // Boost getting started pages
      pageRank: isGettingStarted ? "100" : isReference ? "-50" : "0",
    },
  });
}
```

<Accordion title="Understanding pageRank">
  The `pageRank` value is added to Algolia's computed weight for each result. Higher values appear first. Use string values including negative numbers to de-boost less important content.
</Accordion>

## Reducing Record Size

### Aggregate Content

Combine content records to reduce total record count:

```javascript theme={null}
return helpers.docsearch({
  recordProps: {
    lvl0: { selectors: "header h1" },
    lvl1: "article h2",
    lvl2: "article h3",
    content: "main p, main li",
  },
  aggregateContent: true, // Groups content under headings
});
```

<Warning>
  Pages generating more than 750 records will fail to index. Use `aggregateContent: true` to stay under the limit.
</Warning>

### Record Version

Use v3 format to reduce record size:

```javascript theme={null}
return helpers.docsearch({
  recordProps: {
    lvl0: { selectors: "header h1" },
    lvl1: "article h2",
    content: "main p, main li",
  },
  recordVersion: "v3", // Removes legacy DocSearch v2 fields
});
```

## Crawler Configuration

### Scheduling Crawls

Configure when the crawler runs:

```json theme={null}
{
  "schedule": "every 1 week",
  "startUrls": [
    "https://docs.example.com/"
  ],
  "actions": [
    {
      "indexName": "docs",
      "pathsToMatch": ["https://docs.example.com/**"],
      "recordExtractor": "/* ... */"
    }
  ]
}
```

### URL Patterns

Control which pages are crawled:

<CodeGroup>
  ```json Include Patterns theme={null}
  {
    "startUrls": [
      "https://docs.example.com/v4/"
    ],
    "pathsToMatch": [
      "https://docs.example.com/v4/**"
    ]
  }
  ```

  ```json Exclude Patterns theme={null}
  {
    "exclusionPatterns": [
      "**/drafts/**",
      "**/internal/**",
      "**/_*.html"
    ]
  }
  ```
</CodeGroup>

### Authentication

Crawl password-protected documentation:

```json theme={null}
{
  "auth": {
    "username": "docs",
    "password": "${env:DOCS_PASSWORD}"
  }
}
```

<Note>
  Authentication is available on paid Algolia plans, not the free DocSearch program.
</Note>

## Monitoring Crawls

Access the [Algolia Crawler dashboard](https://crawler.algolia.com/) to:

* Trigger manual crawls
* View crawl logs and errors
* Test URL extraction
* Monitor index size
* Preview search results

### URL Tester

Test your record extractor on specific URLs:

1. Navigate to the URL Tester in the crawler dashboard
2. Enter a documentation URL
3. View extracted records
4. Refine your selectors

### Search Preview

Test search queries against your index:

1. Open Search Preview in the dashboard
2. Enter search terms
3. View ranked results
4. Adjust relevance settings

## Common Issues

<AccordionGroup>
  <Accordion title="Pages aren't being crawled">
    Check that:

    * URLs match `pathsToMatch` patterns
    * Pages aren't blocked by robots.txt
    * Authentication is configured correctly
    * Pages return 200 status codes
  </Accordion>

  <Accordion title="Content is missing from results">
    Verify that:

    * CSS selectors match your page structure
    * Content isn't inside removed elements
    * Records aren't exceeding size limits
    * The crawler can access dynamic content
  </Accordion>

  <Accordion title="Duplicate results appearing">
    Solutions:

    * Configure canonical URLs
    * Add exclusion patterns for duplicates
    * Use `distinct` parameter in search
    * Check for trailing slash variations
  </Accordion>

  <Accordion title="Too many records error">
    Reduce records by:

    * Setting `aggregateContent: true`
    * Removing verbose content selectors
    * Excluding code blocks or examples
    * Splitting large pages into smaller sections
  </Accordion>
</AccordionGroup>

## Best Practices

<CardGroup cols={2}>
  <Card title="Use Semantic HTML" icon="code">
    Structure content with proper heading hierarchy (h1 → h2 → h3) for better indexing.
  </Card>

  <Card title="Keep Selectors Simple" icon="bullseye">
    Use stable CSS classes rather than complex selectors that may break.
  </Card>

  <Card title="Test Regularly" icon="flask">
    Use the URL tester to verify extraction after content changes.
  </Card>

  <Card title="Monitor Index Size" icon="chart-line">
    Track record count and optimize extraction to stay within limits.
  </Card>
</CardGroup>

## Next Steps

<Card title="Record Extractor Reference" icon="book" href="https://www.algolia.com/doc/tools/crawler/apis/configuration/actions/">
  Complete reference for record extractor configuration
</Card>