# Search engines are welcome on the public pages. User-agent: Googlebot User-agent: Googlebot-Image User-agent: Bingbot User-agent: DuckDuckBot User-agent: Slurp User-agent: Applebot User-agent: Qwantify User-agent: YandexBot User-agent: Baiduspider User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Claude-SearchBot User-agent: Claude-User Disallow: /account Disallow: /admin Disallow: /listings/ Disallow: /login Disallow: /register Disallow: /forgot Disallow: /reset/ Disallow: /saved Disallow: /pay/ Disallow: /directory/all-listings Disallow: /pt/account Disallow: /pt/admin Disallow: /pt/listings/ Disallow: /pt/login Disallow: /pt/register Disallow: /pt/forgot Disallow: /pt/reset/ Disallow: /pt/saved Disallow: /pt/pay/ Disallow: /pt/directory/all-listings Disallow: /fr/account Disallow: /fr/admin Disallow: /fr/listings/ Disallow: /fr/login Disallow: /fr/register Disallow: /fr/forgot Disallow: /fr/reset/ Disallow: /fr/saved Disallow: /fr/pay/ Disallow: /fr/directory/all-listings Disallow: /de/account Disallow: /de/admin Disallow: /de/listings/ Disallow: /de/login Disallow: /de/register Disallow: /de/forgot Disallow: /de/reset/ Disallow: /de/saved Disallow: /de/pay/ Disallow: /de/directory/all-listings # Crawlers that copy content for AI training, data brokers and SEO tools are not allowed (see /terms). User-agent: GPTBot User-agent: CCBot User-agent: ClaudeBot User-agent: Claude-Web User-agent: anthropic-ai User-agent: Google-Extended User-agent: Bytespider User-agent: Amazonbot User-agent: Applebot-Extended User-agent: meta-externalagent User-agent: FacebookBot User-agent: Diffbot User-agent: ImagesiftBot User-agent: omgili User-agent: omgilibot User-agent: DataForSeoBot User-agent: SemrushBot User-agent: AhrefsBot User-agent: MJ12bot User-agent: PetalBot User-agent: DotBot User-agent: BLEXBot User-agent: Timpibot User-agent: cohere-ai User-agent: YouBot User-agent: ICC-Crawler User-agent: Scrapy User-agent: AwarioBot User-agent: Kangaroo Bot User-agent: SeekportBot Disallow: / # Everyone else: slow down, and do not walk through filtered or paged search results. User-agent: * Disallow: /account Disallow: /admin Disallow: /listings/ Disallow: /login Disallow: /register Disallow: /forgot Disallow: /reset/ Disallow: /saved Disallow: /pay/ Disallow: /directory/all-listings Disallow: /pt/account Disallow: /pt/admin Disallow: /pt/listings/ Disallow: /pt/login Disallow: /pt/register Disallow: /pt/forgot Disallow: /pt/reset/ Disallow: /pt/saved Disallow: /pt/pay/ Disallow: /pt/directory/all-listings Disallow: /fr/account Disallow: /fr/admin Disallow: /fr/listings/ Disallow: /fr/login Disallow: /fr/register Disallow: /fr/forgot Disallow: /fr/reset/ Disallow: /fr/saved Disallow: /fr/pay/ Disallow: /fr/directory/all-listings Disallow: /de/account Disallow: /de/admin Disallow: /de/listings/ Disallow: /de/login Disallow: /de/register Disallow: /de/forgot Disallow: /de/reset/ Disallow: /de/saved Disallow: /de/pay/ Disallow: /de/directory/all-listings Disallow: /*?page= Disallow: /*&page= Disallow: /*?sort= Disallow: /*&sort= Disallow: /*?q= Crawl-delay: 10 Sitemap: https://taxiwaymarket.com/sitemap.xml