4 ms·
> I wonder if one solution here could be to turn up Anubis difficulty if the first page hit is not one of the obvious entry-points to cgit. Anubis has a fairly
by colinsane 18d ago
> I wonder if one solution here could be to turn up Anubis difficulty if the first page hit is not one of the obvious entry-points to cgit.
Anubis has a fairly capable "policy" system. you can place something like this in your policy.json:
```
{
"bots": [
{
"action": "WEIGH",
"expression": "path.startsWith(\"/expensive/endpoint\")",
"name": "scrutinize-expensive-endpoints",
"weight": {
"adjust": 20
}
}
]
}
```
another thing smaller sites benefit from -- where the load induced by crawlers tends to be bursty (e.g. as they discover new expensive endpoints to crawl) -- is to adjust the difficulty up/down to maintain a steady system load.
```
{
"bots": [
{
"action": "WEIGH",
"expression": "load_15m <= 16.0",
"name": "sustained-low-load",
"weight": {
"adjust": -10
}
},
{
"action": "WEIGH",
"expression": "load_5m >= 24.0",
"name": "intermittent-high-load",
"weight": {
"adjust": 10
}
},
]
}
```