TXTs in Webdirectories

In the root of the webdirectories of many websites, you will find various .txt files, which all have their own usecases.

robots.txt

Purpose: Tells search engine crawlers which parts of the site they can or cannot access. Use case: SEO, crawl control, and protecting server resources.

Example:

User-agent: *
Disallow: /admin/
Allow: /

security.txt

Purpose: A standardized file for security researchers to report vulnerabilities.

Example:

Contact: mailto:[email protected]
Encryption: https://example.com/pgp-key.txt
Policy: https://example.com/security-policy

humans.txt

Purpose: A human-readable credits file containing information about the people who built or maintain the website.

Example:

Team: WebDev Department
Developer: Alice & Bob
Contact: [email protected]

ads.txt

Purpose: Used by publishers to declare authorized digital advertising sellers to prevent fraud in online ads, especially for Google AdSense / programmatic ads.

Example:

google.com, pub-1234567890, DIRECT, f08c47fec0942fa0

app-ads.txt

A variant of ads.txt for mobile apps, often found on a website associated with an app developer.

manifest.txt / manifest.webmanifest

Sometimes found at root. Purpose: Provides metadata for Progressive Web Apps (PWAs).

sitemap.txt or sitemap.xml

Sitemaps are normally XML, but sometimes a plaintext list is provided. Tells search engines which pages on your website should be indexed.

Example:

https://example.com/
https://example.com/about
https://example.com/blog/post-1