User-agent: * Allow: / # guide.html / checkout.html / the staff tool / /jobs/signin are NOT disallowed here on # purpose: each carries its own meta noindex, and blocking them in robots.txt would stop # Google fetching that noindex (→ URL-only "indexed though blocked" stubs) AND would # publicly advertise the staff path. Let Google crawl them and honour the noindex. # These board areas ARE blocked: session-gated, zero public value, saves crawl budget. Disallow: /jobs/api/ Disallow: /jobs/account Disallow: /jobs/candidate Disallow: /jobs/admin Sitemap: https://www.ukcareersfair.com/sitemap.xml Sitemap: https://www.ukcareersfair.com/jobs/sitemap.xml