{"product_id":"robots-txt-customization-retina","title":"robots.txt Customization","description":"\u003cp class=\"task-lead\"\u003eA retailer with leftover code from multiple old themes had search engines indexing pages they shouldn't, and wanted the crawl rules cleaned up.\u003c\/p\u003e\u003ch2\u003eThe request\u003c\/h2\u003e\u003cblockquote\u003e\n\u003cp\u003eWe've had several templates in use and have leftover old code. Search engines are indexing pages that shouldn't be indexed. We need the robots rules and the old code cleaned up.\u003c\/p\u003e\n\u003ccite\u003eOriginal customer request, anonymized\u003c\/cite\u003e\n\u003c\/blockquote\u003e\u003cp\u003eCrawl clutter from years of theme changes.\u003c\/p\u003e\u003ch2\u003eThe approach\u003c\/h2\u003e\u003cp\u003eWe removed the leftover code, set robots rules to exclude the pages that shouldn't be indexed, and confirmed the sitemap only contained the pages that should.\u003c\/p\u003e\u003ch2\u003eWhat we shipped\u003c\/h2\u003e\u003cul\u003e\n\u003cli\u003eLeftover theme code removed\u003c\/li\u003e\n\u003cli\u003eRobots rules updated\u003c\/li\u003e\n\u003cli\u003eSitemap cleaned\u003c\/li\u003e\n\u003cli\u003eVerified in Search Console\u003c\/li\u003e\n\u003c\/ul\u003e","brand":"TaskHusky Portfolio","offers":[{"title":"Default Title","offer_id":44155686846538,"sku":"TASK-0214","price":0.0,"currency_code":"USD","in_stock":false}],"url":"https:\/\/www.taskhusky.com\/products\/robots-txt-customization-retina","provider":"TaskHusky","version":"1.0","type":"link"}