typpo

5 ms·

1.
by typpo 2mo ago
No need to hack it, we'll add proper support for this.
2.
by typpo 3mo ago
>> Terra has competitive performance to GPT‑5.5 [while being 2x cheaper]… > To me that means “it’s an inferior product but marketing dictates we try and hide that.” I interpret this to mean you're about to get today's mai
3.
OpenAI frontier models and Codex are now available on AWS(openai.com)
370 points by typpo 4mo ago|131 comments
4.
by typpo 7mo ago
Lately my company has been doing a lot of complex accounting and reporting in spreadsheets. Overall was surprised by how well both GPT and Claude handled some of these extremely tedious tasks. Not uncommon to have an hours-long task compr
5.
How to replicate the Claude Code attack with Promptfoo(promptfoo.dev)
6 points by typpo 10mo ago|0 comments
6.
by typpo 1y ago
Nice work! This is like a much better version of Ancient Earth[0], which I made ~10 years ago using GPlates[1]. I like your approach of rendering the map itself from data, which makes it continuous, rather than just wrapping map textures
7.
by typpo 2y ago
This is so fun and creative. Congrats on launching!
8.
Questions censored by DeepSeek(promptfoo.dev)
384 points by typpo 2y ago|227 comments
9.
Llama 3.2(huggingface.co)
21 points by typpo 2y ago|0 comments
10.
by typpo 2y ago
Thanks to Meta for their work on safety, particularly Llama Guard. Llama Guard 3 adds defamation, elections, and code interpreter abuse as detection categories. Having run many red teams recently as I build out promptfoo's red teaming
11.
Automated jailbreaking techniques with DALL-E(promptfoo.dev)
2 points by typpo 2y ago|0 comments
12.
by typpo 2y ago
If anyone is interested in evaling Gemma locally, this can be done pretty easily using ollama[0] and promptfoo[1] with the following config: prompts: - 'Answer this coding problem in Python: {{ask}}' providers: - ol
13.
Show HN: Automated red teaming for your LLM app(promptfoo.dev)
23 points by typpo 2y ago|2 comments
14.
by typpo 2y ago
Care to explain why you think so?
15.
by typpo 2y ago
The problem in this case is not that it was trained on bad data. The AI summaries are just that - summaries - and there are bad results that it faithfully summarizes. This is an attempt to reduce hallucinations coming full circle. A simple
16.
by typpo 2y ago
The amount of negativity in these comments is astounding. Congrats to the teams at Google on what they have built, and hoping for more competition and progress in this space.
17.
by typpo 2y ago
Paul's benchmarks are excellent and they're the first thing I look for to get a sense of a new model performance :) For those looking to create their own benchmarks, promptfoo[0] is one way to do this locally: prompts: - &
18.
by typpo 2y ago
Great idea and congrats on shipping the project! I'm curious if you noticed certain models worked better for summarizing and converting to steps. For example, in my projects I've found that Gemini outperforms "better" mo
19.
by typpo 2y ago
Public benchmarks are broadly indicative, but devs really should run custom benchmarks on their own use cases. Replicate created a Llama 3 API [0] very quickly. This can be used to run simple benchmarks with promptfoo [1] comparing Llama 3
20.
by typpo 2y ago
If anyone wants to eval this locally versus codellama, it's pretty easy with Ollama[0] and Promptfoo[1]: prompts: - "Solve in Python: {{ask}}" providers: - ollama:chat:codellama:7b - ollama:chat:codegemma:i
21.
Benchmark Command R vs. GPT/Claude on your own data(promptfoo.dev)
2 points by typpo 2y ago|0 comments
22.
DBRX vs. Mixtral vs. GPT: create your own benchmark(promptfoo.dev)
1 points by typpo 2y ago|0 comments
23.
by typpo 2y ago
I'm working on https://quickchart.io/ , a web API for generating chart images. I've expanded it to a WYSIWYG chart editor at https://quickchart.io/chart-maker/ , which lets you create an endpo
24.
by typpo 3y ago
Congrats on the launch! I've been interested in automatic testset generation because I find that the chore of writing tests is one of the reasons people shy away from evals. Recently landed eval testset generation for promptfoo ( http
25.
by typpo 3y ago
I posted this visualization of mine in a recent thread on the Quadrantids, but sharing again because people seemed to enjoy it: https://www.meteorshowers.org/view/Quadrantids It uses meteor data from NASA CAMS [1] to r
26.
by typpo 3y ago
Using recorded meteor data from NASA CAMS, I built this visualization of the meteor cloud that creates the Quadrantids: https://www.meteorshowers.org/view/Quadrantids The Quadrantids are interesting because their sourc
27.
How to benchmark Gemini vs. GPT with your own data(promptfoo.dev)
1 points by typpo 3y ago|0 comments
28.
by typpo 3y ago
Evals are important for LLM app development. I've noticed dozens of tools in this space, including 11 (!) YC companies, so I put them together on a page.
29.
A collection of LLM evaluation tools(ianww.com)
2 points by typpo 3y ago|1 comments
30.
by typpo 3y ago
Years ago, I built Asterank, an open-source database of asteroids. It landed me a job at Planetary Resources, an "asteroid mining" company: https://www.asterank.com/

More ›