Did You Know? Disallow Doesn't Remove an Indexed Page
Adding a URL to robots.txt does not automatically remove it from Google’s index.
💡 Did You Know?
Adding a URL to robots.txt does not automatically remove it from Google’s index.
Disallow controls future crawling—it is not an instruction to remove an already indexed URL from search results. If a URL has already been indexed, simply blocking it in robots.txt may prevent crawlers from accessing the page while leaving the URL in the index.
SEO teams need to treat crawling and indexing as separate concerns when cleaning up obsolete storefront URLs.
Savan Koradia
Architect & FounderCEO & Founder of AakarDev Softwares • Salesforce B2C Commerce Architect
Hands-on engineer sharing practical, experience-driven insights on Salesforce Commerce Cloud architecture, technical debt, cartridge health, and platform decisions from real production work.
Related Articles
Did You Know? robots.txt is Not a Security Mechanism
robots.txt can tell bots what not to crawl—but it cannot stop them from directly requesting a page.
Did You Know? Dynamic Mappings Exclusively for Legacy
Dynamic Mappings are engineered exclusively for legacy non-SFCC links arriving from external platforms during a migration.
Did You Know? Pushing URL Rules
Pushing URL Rule settings from Staging to Production requires selecting BOTH the Catalogs and Libraries data replication groups.