okhat

7 ms·

1.
by okhat 1y ago
This is a DSPy optimizer, built by the DSPy core team. Just wait for open sourcing.
2.
by okhat 3y ago
I'll admit even I sometimes wish ColBERT was more user-friendly. I'll probably start using ColBERT throught RAGatouille now.
3.
by okhat 3y ago
Writing right now. This month :-)
4.
by okhat 3y ago
Thanks! Lots to discuss from your excellent response, but I'll address the easy part first: DSPy is v2 of DSP (demonstrate-search-predict). The DSPy paper hasn't been released yet. DSPy is a completely different thing from DSP. It
5.
by okhat 3y ago
@wokwokwok Okay now we disagree. This task is not easy, it's just easy to follow in one notebook. (If it were easy, the RAG score wouldn't be 26%.) As for "carefully crafted string templates", I'm not sure what your
6.
by okhat 3y ago
Ah okay makes sense, yeah we'll release more examples. This is just an intro to the key concepts/modules.
7.
by okhat 3y ago
What did you find underwhelming if I may ask? It shows you how it takes some ~25 Pythonic lines of code to make GPT-3.5 retrieval accuracy go from the 26-36% range to 60%. Not a bad deal when you apply it to your own problem?
8.
by okhat 3y ago
Posted an answer here: https://news.ycombinator.com/item?id=37420175
9.
by okhat 3y ago
Here's the key idea. You give DSPy (1) your free-form code with declarative calls to LMs, (2) a few inputs [labels optional], and (3) some validation metric [e.g., sanity checks]. It simulates your code on the inputs. When there's
10.
by okhat 3y ago
These are answers to specific questions below.
11.
by okhat 3y ago
just posted a top-level answer, copied from the FAQs of DSPy
12.
by okhat 3y ago
See the discussion of teleprompters here: https://colab.research.google.com/github/stanfordnlp/dspy/bl...
13.
by okhat 3y ago
"print out every prompt that is generated to a log" --- yes of course This Colab is full of prompts and examples of improving the quality of gpt-3.5-turbo: https://t.co/Oa1RDp3XbZ Paper incoming, but basically we&
14.
by okhat 3y ago
@simonw it sounds like we'd agree that: 1] when prototyping, it's useful to not have to tweak each prompt by hand as long as you can inspect them easily 2] when the system design is "final", it's important to be abl
15.
by okhat 3y ago
btw read a more official answer here: https://github.com/stanfordnlp/dspy#5a-dspy-vs-thin-wrappers...
16.
by okhat 3y ago
"A neural network layer is just a matrix. Why abstract that matrix and learn it?" Well, because it's not your job to figure out how to hardcode delicate string or floats that work well for a given architecture & backend.
17.
DSPy: Framework for programming with foundation models(github.com)
141 points by okhat 3y ago|52 comments
18.
by okhat 3y ago
DSPy provides composable and declarative modules for instructing LMs in a familiar Pythonic syntax and an automatic compiler that teaches LMs how to conduct the declarative steps in your program. Specifically, the DSPy compiler will interna
19.
by okhat 3y ago
There’s always DSP for those who need a lightweight but powerful programming model — not a library of predefined prompts and integrations. It’s a very different experience from the hand-holding of LangChain, but it packs reusable magic in g
20.
by okhat 4y ago
Very cool! But for hard enough problems, prompt engineering is kind of like hyperparameter tuning. It's only a final (and relatively minor) step after building up an effective architecture and getting its modules to work together. DSP