Why Google isn't indexing your Next.js site: a technical SEO checklist

You launched the site weeks ago, the sitemap is live, and a site: search still shows a handful of pages, or none. Before rewriting content or buying backlinks, rule out the technical causes. In Next.js projects the usual culprits are mundane: a preview-environment noindex that leaked into production, canonicals pointing at the wrong host, error pages that return 200, or content that only exists after client-side JavaScript runs.

This checklist goes from the hard blockers that guarantee a page will not be indexed to the softer signals that make Google choose not to. Examples use the App Router's metadata API and file conventions. Work through it in order: there is no point improving content quality on a page that Google is being told not to index, and no point requesting indexing until the blockers are gone. Most fixes are a few lines of code; the hard part is finding which one applies.

1. Check what Google actually receives

Start in Google Search Console. The URL Inspection tool shows whether a URL is indexed, which canonical Google selected, when it was last crawled and, with Test live URL, the rendered HTML and HTTP response. The Page indexing report groups every known URL by reason, such as Excluded by 'noindex' tag, Blocked by robots.txt, Duplicate, Google chose different canonical than user or Soft 404. That reason is usually your answer.

Then reproduce it yourself. Headers matter as much as HTML, because an X-Robots-Tag header is invisible in the page source.

URL="https://www.example.com/blog/my-post"

# Status code, redirects and any X-Robots-Tag header
curl -sIL "$URL" | grep -iE "^HTTP/|^location:|^x-robots-tag:"

# Robots meta and canonical in the server-rendered HTML
curl -sL "$URL" | grep -ioE '<meta name="robots"[^>]*>|<link rel="canonical"[^>]*>'

# Is the main content in the initial HTML (before JavaScript)?
curl -sL "$URL" | grep -c "a phrase from your article body"

# What does robots.txt say?
curl -s https://www.example.com/robots.txt

2. Remove accidental blockers: robots.txt and noindex

A Disallow: / copied from a staging setup blocks crawling of the whole site. Note the subtlety: robots.txt blocks crawling, not indexing, and a blocked page cannot show Google its noindex tag. To remove a page from the index, allow crawling and use noindex; to save crawl budget on pages that should never be fetched, use Disallow.

The other classic is a global robots: { index: false } in the root layout, intended for preview deployments and shipped to production. Drive it from the environment so it cannot leak. Hosting platforms often add their own protections, too: Vercel, for example, sends X-Robots-Tag: noindex on preview deployment URLs, which is correct, but means you must test indexability on the production domain.

import type { MetadataRoute } from "next";

const isProduction = process.env.VERCEL_ENV === "production" || process.env.APP_ENV === "production";

export default function robots(): MetadataRoute.Robots {
  if (!isProduction) {
    return { rules: { userAgent: "*", disallow: "/" } };
  }
  return {
    rules: { userAgent: "*", allow: "/", disallow: ["/api/", "/dashboard/"] },
    sitemap: "https://www.example.com/sitemap.xml",
  };
}

Also check middleware. Auth or geo-redirect logic that runs on every path can send Googlebot to a login page or a different locale, and Google indexes what it is redirected to, not what you intended.

3. Get canonicals and redirects consistent

Every indexable page should declare one canonical URL, and that URL should be the one that returns 200, is linked internally and appears in the sitemap. Mismatches between www and the apex domain, trailing slashes, http versus https, or tracking parameters make Google pick its own canonical, often not the one you want.

import type { Metadata } from "next";
import { notFound } from "next/navigation";
import { getPost } from "@/lib/posts";

export async function generateMetadata({ params }: { params: Promise<{ slug: string }> }): Promise<Metadata> {
  const { slug } = await params;
  const post = await getPost(slug);
  if (!post) return {};
  return {
    title: post.title,
    description: post.excerpt,
    // Resolved against metadataBase set in the root layout
    alternates: { canonical: `/blog/${post.slug}` },
  };
}

export default async function Page({ params }: { params: Promise<{ slug: string }> }) {
  const { slug } = await params;
  const post = await getPost(slug);
  if (!post) notFound(); // real 404 instead of an empty 200 page
  return <article>{/* ... */}</article>;
}

Watch for pages that canonicalise to the homepage or to page one of a paginated list. Both tell Google the page is a duplicate and should be dropped. Each paginated page, each localised version and each distinct article should point to itself. If you serve multiple languages, add alternates.languages so Google understands the variants are translations rather than duplicates.

Set metadataBase once in the root layout to your production origin so relative canonicals resolve correctly, and redirect every non-canonical host and protocol to it with a single permanent redirect, not a chain.

4. Return honest status codes

If a missing product or deleted post renders an empty template with status 200, Google reports it as a soft 404 and may start distrusting similar URLs. Call notFound() when data is missing, as above, and use permanentRedirect() for moved content. Be aware that once a streamed response has started, the status code is already sent; Next.js then injects a noindex robots meta tag for not-found pages, but it is cleaner to resolve the data before streaming begins so crawlers get a real 404. Also make sure server errors return 5xx rather than a friendly page with 200, so Google retries instead of indexing the error.

5. Help discovery: sitemaps and internal links

A sitemap tells Google which URLs you consider canonical and when they changed. Generate it from the same data source as your pages with app/sitemap.ts, include only indexable 200 URLs, set lastModified to the real content date rather than the build time, and submit it in Search Console. Then link to important pages from navigation and related content. Pages that are only reachable through the sitemap, with no internal links, are often discovered but never indexed.

After fixing a technical issue, use Validate fix on the relevant reason in the Page indexing report and request indexing for a few key URLs. Recrawling takes time, so judge progress over weeks rather than hours, and keep an eye on the server logs or your host's analytics to confirm Googlebot is fetching the pages and getting 200 responses.

Finally, accept that indexing is selective. If Search Console says Crawled - currently not indexed and every technical check passes, the issue is content: thin pages, near-duplicates such as filtered listings, or pages that do not answer anything better than existing results. Consolidate, improve or noindex them.

Key takeaways

• Use URL Inspection and the Page indexing report first; the exclusion reason usually names the problem.
• Check headers as well as HTML; X-Robots-Tag and middleware redirects are invisible in the page source.
• Gate noindex and restrictive robots rules on the environment so they never reach production.
• Keep canonicals, redirects, internal links and the sitemap pointing at the same URLs.
• Return real 404 and 5xx status codes and server-render the content you want indexed.

Need a technical SEO audit of your Next.js app? Our frontend team builds fast, crawlable sites and fixes indexing problems at the source. Get in touch.