Spider Simulator
See your website exactly how search engines crawl it. Enter a URL to simulate a search engine spider and view raw text, links, headings, and meta elements. Perfect for SEO audits, indexing checks, and content troubleshooting—fast and free.
Spider Simulator
The Spider Simulator fetches a URL and displays its content exactly as a search engine crawler reads it — stripped of CSS styling, images, JavaScript execution, and visual layout. What remains is the raw HTML content that crawlers process: plain text, headings, links, meta tags, and structural elements. Enter any URL and click Simulate URL to see the crawler's view of your page.
This view is significant because what a search engine can read and what a user sees in a browser are often different things. Modern websites rely heavily on JavaScript to render content dynamically — navigation menus, product descriptions, article text, and internal links are frequently injected into the page after it loads, using scripts rather than static HTML. A spider simulator helps you identify whether your most important content is visible to crawlers in the initial HTML response, or whether it depends on JavaScript execution that some crawlers may not fully process.
How to use the Spider Simulator
- Enter the full URL of the page you want to simulate — include the protocol (https://) and the complete path for specific pages beyond the homepage.
- Click Simulate URL. The tool fetches the page's raw HTML without executing JavaScript or applying CSS, and displays the crawler's view of the content.
- Review the output systematically: check that the title tag, meta description, H1, body text, and internal links are all present and correctly formed. Use the 'What to check' reference table below as a checklist.
- If any important element is missing from the simulated view, identify whether it is caused by JavaScript rendering, a robots.txt block, a noindex tag, or a structural error — and fix accordingly. Re-run the simulation to confirm the fix.
How search engine crawlers work — and why this matters
A search engine crawler — also called a spider, bot, or robot — is an automated program that systematically visits web pages to collect content for the search engine's index. Googlebot, the crawler used by Google, starts with a list of known URLs (from sitemaps, previous crawls, and link discovery) and follows links from page to page in a process called web crawling.
For each URL it visits, the crawler downloads the HTML source of the page. This initial download is the raw HTML response — the document that exists before any JavaScript has executed. The crawler extracts text, headings, links, and meta information from this document. Google also runs a separate rendering process that executes JavaScript and processes the fully rendered DOM, but this second pass may happen hours, days, or weeks after the initial crawl — and not all content discovered during rendering is indexed with the same speed or reliability as content in the initial HTML.
The Spider Simulator replicates the initial HTML crawl — the first thing a crawler sees when it visits your page. Any content, links, or tags that appear in the simulation will be available to crawlers immediately. Any content that is absent from the simulation but visible in a browser is likely JavaScript-rendered and subject to the delays and uncertainties of the rendering queue.
The difference between what a browser shows and what a crawler reads is the core insight this tool provides. Open your browser, visit a page, and it looks complete and professional. Open the Spider Simulator on the same page, and you may find that the product descriptions are missing, the navigation links are absent, or the H1 heading does not appear. This discrepancy is the JavaScript rendering gap — and it is one of the most common causes of underperforming pages on modern websites built with React, Vue, Angular, or heavy WordPress page builders.
What the simulation extracts
The spider simulation displays the following elements from the page's static HTML — the same information a crawler uses to understand, categorize, and index the page:
- Title tag — the page title as it will appear in search result snippets and browser tabs. Displayed in the <title> element in the HTML head.
- Meta description — the page summary that appears beneath the title in search results. Displayed in the <meta name='description'> tag.
- Canonical tag — the URL declared as the authoritative version of this page. Displayed in the <link rel='canonical'> tag.
- Robots meta tag — any crawl or index instructions (noindex, nofollow, noarchive, noimageindex). Displayed in the <meta name='robots'> tag.
- H1 through H6 headings — the heading hierarchy of the page. Extracted from heading tags in the order they appear in the HTML source.
- Body text — the visible text content of the page as it exists in the HTML, excluding content rendered by JavaScript.
- Internal links — all links pointing to pages on the same domain. Displayed with their anchor text and href attributes.
- External links — all links pointing to other domains. Displayed with their anchor text and href attributes.
- Image alt attributes — the alt text associated with images, which functions as text content for crawler relevance signals.
What to check in the simulation output — element by element
Once the simulation runs, reviewing it systematically is more effective than scanning it at random. The table below covers each key element, what a correctly configured version looks like, and what to do if it looks wrong:
| Element | What to look for | If it looks wrong |
| Title tag | One title tag per page. Should contain the primary keyword and be 50–60 characters long. | Missing title: add one to the <head>. Duplicate title across pages: make each page's title unique. Title not matching page content: rewrite to accurately describe the page. |
| Meta description | One meta description. Should summarise the page in 150–160 characters. Should not duplicate the title. | Missing: add one. It does not directly affect rankings but influences click-through rate in search results. Duplicate: make each page's description unique. |
| H1 heading | Exactly one H1 per page. Should contain the primary keyword or topic of the page. Should appear near the top of the crawled content. | Missing H1: add one. Multiple H1s: reduce to one and use H2s for subheadings. H1 not visible (JavaScript-only): ensure H1 is in the static HTML, not injected by script. |
| Heading hierarchy | Headings follow a logical order: H1 > H2 > H3. No skipped levels (e.g., H1 directly to H3). | Disordered or skipped headings: restructure for logical sequence. Headings used for styling instead of structure: use CSS for visual styling and reserve heading tags for document structure. |
| Body text content | The core page text should be visible in the simulation. Check that key paragraphs, descriptions, and factual content appear as plain text. | Body text missing or sparse: the page likely relies on JavaScript to render its main content. Move critical text to static HTML or implement server-side rendering. |
| Internal links | Navigation links, contextual body links, and footer links should all be visible with descriptive anchor text. Crawlers follow these links to discover other pages. | Navigation links missing: they may be rendered by JavaScript. Ensure primary navigation is in static HTML. Non-descriptive anchor text (e.g., 'click here', 'read more'): rewrite with keyword-relevant descriptions. |
| Image alt attributes | Images should have descriptive alt text. The crawler sees alt attributes as text content — they contribute to keyword relevance for image-heavy pages. | Missing alt text: add descriptive alt attributes to all informational images. Decorative images can use empty alt attributes (alt=""). |
| Canonical tag | If present, the canonical tag tells crawlers which version of the URL is the definitive one. Check it points to the correct URL. | Canonical pointing to wrong URL: fix to point to the correct, intended page. Missing canonical on pages with duplicate or similar content: add one to prevent duplicate content issues. |
| robots meta tag | Check for <meta name='robots' content='noindex'> or similar directives. A noindex tag tells search engines not to include this page in their index. | Unintended noindex: remove the tag if the page should be indexed. Intended noindex: confirm it is correct. Noindex on a page that should rank is a critical error. |
JavaScript rendering and crawler visibility
The most consequential finding from a spider simulation is often the JavaScript rendering gap — the difference between what appears in the static HTML and what a user sees after the page's JavaScript has executed. This is particularly relevant for:
Single-Page Applications (SPAs) built with React, Vue, or Angular
These frameworks typically render the entire page content client-side using JavaScript. The raw HTML delivered to the server may contain very little text — just a root element and a large script tag. The actual page content (headings, body text, product descriptions, links) is generated by the JavaScript after the browser loads. The Spider Simulator will show this skeleton HTML, revealing that crawlers may see almost none of the page's actual content on their first visit.
The solution is server-side rendering (SSR) or static site generation (SSG), which produces pre-rendered HTML that contains the full page content before any JavaScript executes. Next.js, Nuxt, SvelteKit, and similar frameworks provide this capability for JavaScript-heavy applications.
WordPress pages built with visual page builders
Page builders such as Elementor, Divi, Beaver Builder, and WPBakery generate HTML that is often more complex than necessary, and may include content structures that vary in their crawlability. Run the Spider Simulator on your most important WordPress pages — particularly homepage, service pages, and landing pages — and confirm that all text, headings, and links appear in the raw HTML output. Also check that unnecessary JavaScript is not blocking resources that Googlebot needs to render the page.
Navigation and menus loaded by JavaScript
Some themes and templates load the primary navigation menu using JavaScript after the initial page load. This means the navigation links — which are some of the most important internal links on any page, providing crawl pathways to all major sections of the site — may be invisible to crawlers in the initial HTML crawl. If navigation links are absent from the Spider Simulator output, ensure they exist in the static HTML source. A simple test: view the page source in your browser (Ctrl+U or Cmd+U) and search for the navigation link text — if it is absent from the source, it is JavaScript-rendered.
Crawl budget and crawl accessibility
Search engines allocate a crawl budget to each domain — the number of pages they will crawl within a given time window. Larger, more authoritative domains receive larger budgets; smaller sites receive smaller ones. Pages that are slow to respond, blocked by robots.txt, returning errors (404, 500), or buried deep in a site's link structure may not be crawled as frequently.
The Spider Simulator helps identify crawl accessibility issues at the page level: whether links are in the static HTML for crawlers to follow, whether important pages link to other important pages, and whether any technical barriers exist that would impede a crawler's ability to access and process the page. For site-wide crawl budget analysis, a dedicated crawler tool is more appropriate — but page-level accessibility checks are a good starting point.
Robots.txt and noindex are different things and block different stages of the crawl-index pipeline. Robots.txt blocks the crawler from accessing the URL at all — the page will not be crawled or indexed. A noindex meta tag allows the crawler to visit the page but instructs it not to add the page to the index. Blocking a page with robots.txt while also using a noindex tag is a common mistake: if the page is blocked by robots.txt, the noindex tag cannot be read, and Google may still index the URL based on external links pointing to it. The correct approach for pages you do not want indexed is to allow crawling and use a noindex meta tag.
Common crawl and visibility problems — causes and fixes
| Problem | Why it matters | How to fix it |
| Content only visible after JavaScript executes | If main content (headings, body text, product descriptions) is injected by JavaScript, it may not be indexed at the first crawl. Google uses a two-wave indexing process: initial HTML crawl, then a later rendering pass — content in the second wave may be indexed weeks later or missed entirely. | Move critical content to static HTML. Use server-side rendering (SSR) or static site generation (SSG) for React, Vue, and Angular apps. For WordPress with page builders, verify content appears in HTML source view (Ctrl+U). |
| Navigation links not in static HTML | If site navigation is built entirely with JavaScript and rendered client-side, Googlebot may not discover all pages during the initial crawl. This limits the crawl depth and the number of pages indexed. | Ensure primary navigation links appear in the static HTML source. Use progressive enhancement: navigation should work without JavaScript, with JavaScript enhancing the experience. |
| Important pages blocked by robots.txt | A robots.txt Disallow rule prevents crawlers from accessing those URLs entirely. If important pages are accidentally blocked, they will not be indexed regardless of their content quality. | Review robots.txt at yourdomain.com/robots.txt. Remove Disallow rules for pages that should be indexed. Use Google Search Console's robots.txt tester to verify. Never block CSS or JS files Googlebot needs to render pages. |
| Accidental noindex meta tag | A <meta name='robots' content='noindex'> tag instructs crawlers not to include the page in the index. This is useful for staging environments, thank-you pages, and admin pages — but catastrophic if applied accidentally to content pages. | Check the spider simulator output for noindex tags on every important page. A common source of this error: staging environments with noindex enabled are accidentally migrated to production. |
| Non-descriptive or missing anchor text | Anchor text is how crawlers and search engines understand what the linked page is about. Generic anchor text like 'click here', 'read more', or 'learn more' provides no topical signal. Missing anchor text (image links without alt text) provides even less. | Rewrite internal link anchor text to describe the destination page's content. Image links should have descriptive alt attributes that act as the anchor text signal. |
| Duplicate title tags or H1s across multiple pages | If multiple pages share the same title tag or H1, search engines have difficulty distinguishing between them for different search queries. This can cause keyword cannibalization — multiple pages competing for the same query — and reduce the ability to rank either page effectively. | Run the spider simulator on each key page and compare title and H1 content. Each page should have a unique title tag and H1 that accurately reflects its specific topic and target keyword. |
When to run the Spider Simulator
- After launching a new page or section of your website — confirm all key elements are crawlable before the page is discovered and indexed.
- After a major site redesign, CMS migration, or framework change — the new site structure may have introduced JavaScript rendering gaps or accidental noindex tags.
- After a page builder or theme update on WordPress — updates can affect how content is delivered in the HTML source.
- When a page that previously ranked has seen an unexplained drop — compare the current simulation against what you expected the crawler to see.
- As part of a routine technical SEO audit — run the simulator on your 10 to 20 highest-priority pages quarterly to catch regressions before they affect rankings.
- When investigating why a specific page is not appearing in Google Search Console or has not been indexed despite being submitted in the sitemap.
Usage limits
| Guest users | 25 simulations per day. No account required. |
| Registered users | 100 simulations per day. Free to register. |
Related tools
- SSL Checker — verify your SSL certificate is valid. Crawlers reject pages with invalid certificates, which prevents indexing.
- Check GZIP Compression — confirm your server compresses text resources. Fast-loading pages are crawled more frequently and more completely.
- Websites Broken Link Checker — identify broken links that create dead ends for crawlers and waste crawl budget.
- MozRank Checker — evaluate the link authority of individual pages alongside their crawlability.
Frequently asked questions
What is a spider simulator?
A spider simulator is a tool that fetches a web page and displays its content exactly as a search engine crawler sees it — without executing JavaScript, applying CSS, or rendering visual layout. It shows the raw HTML content: title tag, meta description, headings, body text, internal and external links, alt attributes, and crawl directive tags (canonical, robots meta). The output represents what Google, Bing, and other search engines extract from your page during their initial crawl, before any rendering of JavaScript-dependent content.
Why is the spider simulation different from what I see in my browser?
Your browser executes JavaScript, applies CSS, and renders a complete visual page. A search engine crawler's first pass retrieves only the raw HTML — the document delivered by the server before any client-side JavaScript runs. Modern websites frequently use JavaScript to dynamically inject content: navigation links, article text, product descriptions, prices, and headings may all be generated by scripts rather than present in the initial HTML. The spider simulation shows the crawler's view — if something is missing from this view but visible in your browser, it is likely JavaScript-rendered and may not be indexed on the crawler's first visit.
Does the Spider Simulator execute JavaScript?
No. The Spider Simulator shows the raw HTML response — the content that exists before any JavaScript executes. This is intentional: it replicates the initial crawl phase that search engines perform, which does not execute JavaScript. Google does run a later rendering pass that processes JavaScript, but this second-wave indexing can lag behind the initial crawl by hours, days, or even weeks. Content that depends entirely on JavaScript to appear is subject to this delay and uncertainty — moving critical content into static HTML eliminates this risk.
What should I look for when reviewing a spider simulation?
Check these elements systematically: title tag (present, contains primary keyword, 50–60 characters), meta description (present, 150–160 characters, unique to this page), exactly one H1 heading (containing the page's primary topic), logical heading hierarchy (H1 > H2 > H3, no skipped levels), body text content (all important paragraphs and descriptions visible, not relying on JavaScript), internal links with descriptive anchor text, image alt attributes, canonical tag pointing to the correct URL, and no unintended noindex meta tags. The reference table on this page covers each element in detail.
How can I fix content that is missing from the simulation?
If important content is absent from the spider simulation but visible in your browser, it is being rendered by JavaScript after the page loads. The most robust fix is to move that content into static HTML — either by having the server deliver pre-rendered HTML (server-side rendering), generating static HTML files at build time (static site generation), or restructuring the page to put critical content in the initial HTML payload. For WordPress sites, check whether the page builder is delivering content in the HTML source by pressing Ctrl+U to view page source and searching for the missing text.
What is the difference between robots.txt and a noindex tag?
Robots.txt is a file at the root of your domain that tells crawlers which URLs they are not allowed to access. If a URL is blocked by robots.txt, the crawler will not visit it at all. A noindex meta tag is placed in the HTML of a specific page and instructs crawlers that they may visit the page but should not add it to the search index. The critical distinction: if a page is blocked by robots.txt, the noindex tag in its HTML cannot be read — the crawler never sees it. Pages you do not want indexed should use a noindex meta tag while remaining accessible to crawlers, not be blocked by robots.txt.
How does this relate to Google Search Console's URL Inspection tool?
Google Search Console's URL Inspection tool shows how Google specifically sees a URL, including the rendered version after JavaScript execution, and whether the page is currently indexed. The Spider Simulator shows the raw HTML crawl view — what any crawler sees in the initial fetch, without JavaScript rendering or Google-specific data. They complement each other: use the Spider Simulator for a quick raw HTML check on any URL (including competitor URLs or pages on sites you do not own), and use Google Search Console for authoritative, Google-specific indexing data on pages you own.
Is the Spider Simulator free?
Yes. The tool is free within the daily usage limits shown above. Guest users can run 25 simulations per day without creating an account. Registering a free ToolsPiNG account increases the daily limit to 100 simulations and gives access to usage history and saved favorites.