9 ms·
The article contains some good observations, but I'm struggling to understand this one: "Some sites try to communicate with Google through comments in robots.t
by dredge 12y ago
The article contains some good observations, but I'm struggling to understand this one:
"Some sites try to communicate with Google through comments in robots.txt"
In the examples given, none appear to be trying to "communicate with Google through comments" - how is including...
# What's all this then?
# \
#
# -----
# | . . |
# -----
# \--|-|--/
# | |
# |-------|
...a "mistake" to avoid? There's no harm in it at all.
- SoftwareMaven 12y agoI don't think those examples were of people trying to communicate with a crawler. I think they were examples of comments that the owners knew would be thrown away by crawlers.
- Istof 12y ago"Some sites try to communicate with Google through comments in robots.txt" I thought that was the whole point of robots.txt
- lmm 12y agoNo, the point is to communicate with Google through non-comments in robots.txt.