docs: 🔒 robots.txt asset

Closes #10640, #10608

README.md

@@ -209,6 +209,7 @@ Please don't use ElasticSearch unless you already have an ElasticSearch setup an
 - To add custom error images, place them in the `./custom/assets/badges` directory, replacing the files already there.
 - To add custom sounds, place only mp3 files in the `./custom/assets/sounds` directory.
 - To update custom assets without rebuilding, just run `pnpm run gulp`.
+- To block ChatGPT, CommonCrawl, or other crawlers from indexing your instance, uncomment the respective rules in `./custom/robots.txt`.
 
 ## 🧑‍🔬 Configuring a new server
 

custom/assets/robots.txt

@@ -0,0 +1,14 @@
+User-agent: *
+Allow: /
+
+# Uncomment the following to block CommonCrawl
+#
+# User-agent: CCBot
+# User-agent: CCBot/2.0
+# User-agent: CCBot/3.1
+# Disallow: /
+
+# Uncomment the following to block ChatGPT
+#
+# User-agent: GPTBot
+# Disallow: /

packages/backend/src/server/web/views/clip.pug

@@ -24,9 +24,11 @@ block meta
 	unless privateMode
 		if profile.noCrawle
 			meta(name='robots' content='noindex')
+
 		if profile.preventAiLearning
 			meta(name='robots' content='noai')
 			meta(name='robots' content='noimageai')
+			meta(name='GPTBot' content='noindex')
 
 		meta(name='misskey:user-username' content=user.username)
 		meta(name='misskey:user-id' content=user.id)