When I build a small retrieval index from a product's documentation, the slow part is rarely the embedding step. It is getting clean text out of dozens of HTML pages that are full of menus, footers and scripts. This note shows the two-step approach I use with XCrawl, a web data API that returns pages as Markdown or JSON.
Step 1: list the URLs
Before fetching anything, ask the Map endpoint for every URL the docs domain exposes. That gives you a list you can filter (for example, keep only paths under /guides/) so you don't waste requests on changelogs or login pages.
Step 2: scrape the pages you kept
Send each remaining URL to the Scrape endpoint and request Markdown output. JavaScript-rendered pages are rendered before extraction, so single-page-app docs come back as readable text instead of an empty shell.
# pip install xcrawl
# Pseudocode outline; see the official docs for exact parameters.
urls = map_site("https://docs.example.com")
urls = [u for u in urls if "/guides/" in u]
for u in urls:
md = scrape(u, format="markdown")
save(u, md)
Step 3: chunk and embed
Markdown keeps headings, lists and code blocks, so splitting on ## headings gives chunks that line up with real sections of the docs. Those go straight into whatever embedding model and vector store you already use.
Notes
- For a whole site in one job, the Crawl endpoint takes depth and scope rules and lets you poll for results.
- There are SDKs for Python and Node.js, a CLI, and an MCP server if you want an agent such as Claude Code to fetch pages itself.
- New accounts get 1,000 free credits, which is plenty for trying this on one docs site.
Docs: https://docs.xcrawl.com/