7 ms·
Shower thought: But how often per week do you run the pelican these days? And do you have it automated at this point or would the automation take out the meani
by persedes 13d ago
Shower thought: But how often per week do you run the pelican these days?
And do you have it automated at this point or would the automation take out the meaning of the benchmark?
- simonw 13d agoMy automation is pretty simple. I use my https://llm.datasette.io https://llm.datasette.io tool where I have a template saved: llm "Generate an SVG of a pelican riding a bicycle" --save pelican When a model comes out I first make sure LLM can talk to it - usually by updating the relevant plugin, but if it's on OpenRouter I can use it directly with https://github.com/simonw/llm-openrouter https://github.com/simonw/llm-openrouter - sometimes I use this mechanism instead, for OpenAI-compliant API models: https://llm.datasette.io/en/stable/other-models.html#configure-an-openai-compatible-model https://llm.datasette.io/en/stable/other-models.html#configu... Then I run something like this: llm -m gpt-6-astra -m pelican Then I grab the most recent log export as markdown: llm logs -cu | pbcopy -c means most recent conversation, -u includes token usage I paste that into https://gist.github.com https://gist.github.com and then paste the resulting Gist URL into the URL tab on https://tools.simonwillison.net/markdown-svg-renderer https://tools.simonwillison.net/markdown-svg-renderer If the model supports multiple reasoning levels I run it once per level and put those in the same file. I really should automate this a bit more.