6 ms·
Out of curiosity, I see my website not being well indexed and I wondered whether it is because I include ``` User-agent: * Disallow: / ``` in my robots.txt. Wha
by ggirelli 10mo ago
Out of curiosity, I see my website not being well indexed and I wondered whether it is because I include
```
User-agent: *
Disallow: /
```
in my robots.txt. What should one add to allow Marginalia to crawl their website?
- marginalia_nu 10mo agoThis should work User-agent: search.marginalia.nu Allow: /
- ggirelli 10mo agoThank you! I just added it :)