sftombu

7 ms·

1.
Rate limiter for LLMs outperforms exponential backoff(github.com)
2 points by sftombu 2y ago|1 comments
2.
by sftombu 2y ago
rateLLMiter is a Python rate limiter that smoothes out requests to LLM APIs to get faster, more consistent performance. It uses a ticket bucket algorithm rather than the usual exponential backoff.
3.
Gemini Flash outperforms more expensive models(nian.llmonpy.ai)
1 points by sftombu 2y ago|0 comments
4.
Fewer if statements with Nothing instead of None(github.com)
3 points by sftombu 2y ago|0 comments
5.
by sftombu 2y ago
The models benchmarked by RULER do worse in needle in a needlestack. It will be interested to see how 4o does with RULER.
6.
by sftombu 2y ago
Previous answer to this question: https://news.ycombinator.com/item?id=40361419s
7.
by sftombu 2y ago
It can't answer the questions without the limericks in the prompt. The benchmark is to establish how well it uses the context window. For example, I just asked it "What is sought by the English top brass?". The answer from t
8.
by sftombu 2y ago
Previous answer to this question: https://news.ycombinator.com/item?id=40361419
9.
by sftombu 2y ago
Previous answer to this question: https://news.ycombinator.com/item?id=40361419
10.
by sftombu 2y ago
Interesting!
11.
by sftombu 2y ago
I tested the LLMs to make sure they could not answer the questions unless the limerick was given to them. Other than 4o, they do very badly on this benchmark, so I don't think the test is invalidated by their training.
12.
by sftombu 2y ago
They come from a database of 98k limericks -- https://zenodo.org/records/5722527
13.
by sftombu 2y ago
That is an interesting idea
14.
by sftombu 2y ago
The reason I made Needle in a needlestack is the LLMs are getting to good at needle in a haystack. Until GPT-4o, no model was good at the NIAN benchmark.
15.
by sftombu 2y ago
If you ask the questions without providing the limerick first, it never gets the right answer. When the LLM gets the wrong answer, it is usually because it reverts to its training data and gives a generic answer that doesn't apply to
16.
by sftombu 2y ago
I was thinking about something similar -- to make part of the question be sufficient information that the LLM can find the limerick. Then the 2nd part would ask something that would require a deeper understanding of the limerick (or other
17.
by sftombu 2y ago
I had the same experience. With a 16k prompt, Turbo was nearly flawless. But it wasn't very good at 32k and not usable at 100+. You have to repeat information to get good results with longer prompts
18.
by sftombu 2y ago
If I had access to Gemini with a reasonable token rate limit, I would be happy to test Gemini. I have had good results with it in other situations.
19.
by sftombu 2y ago
GPT-4o's breakthrough memory -- https://nian.llmonpy.ai/
20.
GPT-4o's Memory Breakthrough(github.com)
2 points by sftombu 2y ago|2 comments