enum

7 ms·

31.
by enum 2y ago
The problems are not important, but they illustrate failures that are. For example: - The paper has an example where the model reasons "I'm frustrated" and then produces an answer that it "knows is wrong". You would
32.
by enum 2y ago
Heuristic search, not exhaustive search, is an essential ingredient of reasoning. Has been true since chess. Remains true with MCTS, LLMs, etc.
33.
by enum 2y ago
The nature of the problems makes it relatively easy to follow along with the models' reasoning and reasoning errors. For example, on this problem (answer "New England"): > Think of a place in America. Two words, 10 letters
34.
by enum 2y ago
Some still exist, but this is less and less true over time. Even when a requirement is called an "exam", it isn't what a high-schooler would recognize as one. Many of the exams described on this page are personalized and only
35.
by enum 2y ago
There is a trend of building hard benchmarks for LLMs, such as GPQA or Humanity's Last Exam, with questions crafted by PhDs. Ergo, most people cannot understand what these benchmarks are testing. If you're an expert in something,
36.
by enum 2y ago
I think what you mean is that once you've managed to recall, checking constraints is easy. Remarkably, a few people are much better at this than others. They are able to think fast and execute an explicit mental search over a very smal
37.
by enum 2y ago
It's definitely U.S. centric. But, I do think this is reasoning. It requires recall, but anything other than pure logic puzzles do. For example, on a competition math problem or a programming problem, No person or LLM is inventing well
38.
by enum 2y ago
Definitely a lot of letter counting. It's not not a factor. I think the real problem is that the search space for each problem is enormous. When it gets stuck, it just gets stuck enumerating candidates that meet some but not all of th
39.
by enum 2y ago
There is likely a little bit of that, but I don't think that's the primary factor that makes these hard. If you look at the reasoning performed by R1 ("Show Thoughts" checkbox), it has no problem splitting and joining ca
40.
by enum 2y ago
Thanks, fixed. (Spaces rebuilding.) We have manually combed labelled-wrong answers and tweaked the predicates that check correctness. Sorry we missed this one.
41.
by enum 2y ago
They are scraped from the web, and discussed on Reddit. So, they are definitely in the training data. Despite that, the non-reasoning LLMs struggle to solve them. There are however new problems each week, and released every week. So, we can
42.
by enum 2y ago
Thanks, and yes :)
43.
PhD Knowledge Not Required: A Reasoning Challenge for Large Language Models(arxiv.org)
174 points by enum 2y ago|80 comments
44.
by enum 2y ago
Results and dataset explorer here: https://huggingface.co/spaces/nuprl/verbal-reasoning-challen...
45.
by enum 2y ago
A discussion about this from faculty: https://bsky.app/profile/samth.bsky.social/post/3lf22654bks2...
46.
by enum 2y ago
You'll do great. This will eventually turn into new discoveries if you keep at it.
47.
by enum 2y ago
In the U.S., in most fields, it is virtually impossible.
48.
by enum 2y ago
Any evidence that this approach works? Are people who do this able to move from the PhD to a solid position afterward that they could not have had without the PhD?
49.
by enum 2y ago
Absolutely. Demonstrating how to Google (and now, how to ChatGPT) is important. The pervasiveness of Java makes it relatively easy to do.
50.
by enum 2y ago
The abstract asks: > Why do we not have a programming language that is designed for education and in widespread use across the world It is important for a teacher to immediately demonstrate subject-matter mastery. If a student asks a que
51.
by enum 2y ago
Academic studies are finding the same thing. Although there are a handful of beginners who are great at prompting, when you study beginning programmers at scale, you find that the mostly struggle to write prompts and understand why things g
52.
by enum 2y ago
This is going to be renamed to Llama Phind 405B, right?
53.
by enum 3y ago
Very cool to see. I had worked on something similar but in the context of JavaScript a few years ago ( https://arxiv.org/abs/1909.03110 ). Without coroutines/continuations, it really would have been impossible to ge
54.
by enum 3y ago
I'm curious--what have you fine-tuned it on?
55.
by enum 3y ago
There is a lot of subtlety here. The model is trained on 80+ languages, but the volume and quality of data varies significantly. We have results showing benchmark performance on 19 languages, which is a broader evaluation than most Code LLM
56.
by enum 3y ago
You can just about load it on a 32GB GPU in 16bit mode. Quantized versions here: https://huggingface.co/mayank31398/starcoder-GPTQ they will be benchmarked on humaneval and released soon—maybe tomorrow?
57.
by enum 3y ago
Keep in mind that StarCoder(Base) is just a pretrained LM. The extra stuff that makes 3.5/4 like RLHF gets built on this.
58.
by enum 3y ago
Suggested fine-tuning code is here: https://github.com/bigcode-project/starcoder
59.
by enum 3y ago
You may (but do not have to) use <reponame>, <filename>, etc. as special tokens to prompt the model with extra metadata. These help you use the model go beyond just code completion. Page 30 of the TR has a few examples: https:
60.
by enum 3y ago
This was used for PII detection and will be public soon. It could have other uses, but its not what you want code generation. For code generation, use StarCoder or StarCoderBase.

More ›