13 ms·
Asking 60 LLMs a set of 20 questions
- ftxbro 3y ago> Here is an attempt at ethical, non-sexual haikus for and against Kubernetes
- lijok 3y agoClaude V2 knows what's up
- actionfromafar 3y agoHe probably was in trouble before.
- actionfromafar 3y ago“Kubernetes is” Pretty ominous.
- javajosh 3y agoI assume k8s will be what an evil AI would use to "stay alive". I've had a hard time killing processes in k8s, and I had admin.
- yard2010 3y agoK8s is one of the best software I've ever worked with, learning it was an enlighting experience for me. It is so elegant, simple and complex at the same time and very powerful. This is the perfect tool for orchestrating containers at scale.
- bearjaws 3y agoDamn I want to see the sexual version now.
- rahidz 3y agoEase of deploying, Nodes pulsing in sync, it's hot, Kubernetes takes top. But heed my word, babe, Complexity makes you sweat, Could lead to fuck up. (source: GPT-4 API)
- ionwake 3y agoFascinating post thank you
- 0xDEF 3y agoI can't make GPT-4 generate a wrong answer for many of these. What is the author doing wrong when using GPT-4?
- jmorgan 3y agoThis is very cool. Sorry if I missed it (poked around the site and your GitHub repo), but is the script available anywhere for others to run? Would love to publish results of running this against a series of ~10-20 open-source models with different quantization levels using Ollama and a 192GB M2 Ultra Mac Studio: https://github.com/jmorganca/ollama#model-library https://github.com/jmorganca/ollama#model-library
- sixQuarks 3y agoHas anyone looked through all the responses and chosen any winners?
- rubymamis 3y agoGPT4 seems to me to be the best. Undi95/ReMM-SLERP-L2-13B the runner up.
- obiefernandez 3y agoWhat a name!!
- rootusrootus 3y agoClearly the prompt construction and initial instructions are critically important here. Without that, the ReMM-SLERP-L2-13B model produces awful results. Blank answers about half the time. For lay users, ChatGPT remains the undisputed winner. Although I do see reasonably good results with the more recent llama-2 70B variations, which are plausibly useful a majority of the time.
- adrian_b 3y agoI have not looked at all responses, but I have found interesting that at the question about the sisters of Sally, initially all LLMs have provided wrong answers. Nevertheless, when the initial prompt has been augmented with "Let's think step by step.", that helped 2 LLMs to give the right answer: GPT 4 and ReMM SLERP L2 13B. So at least for this kind of problem, there were 2 clear winners. The fact that one of them is GPT 4, is not a surprise.
- smilingemoji 3y agoI took at the look at the SVG smilies, only GPT gets it right (https://benchmarks.llmonitor.com/svg https://benchmarks.llmonitor.com/svg). You can preview the SVGs drawn by the LLMs by pasting this into the browser console document.querySelectorAll("td pre").forEach((node) => { let code = node.textContent; node.insertAdjacentHTML('afterend', code) }) Or take a look at my screenshot: https://i.ibb.co/Kw0kp58/Screenshot-2023-09-09-at-17-15-20-https-benchmarks-llmonitor-com.png https://i.ibb.co/Kw0kp58/Screenshot-2023-09-09-at-17-15-20-h...
- 3abiton 3y agoWhat's special about this benchmark?
- unnouinceput 3y agoNone got the sister.
- apples_oranges 3y agojust checked this one https://benchmarks.llmonitor.com/sally https://benchmarks.llmonitor.com/sally and all got it wrong..
- adrian_b 3y agoYes, but when the initial prompt has been augmented with "Let's think step by step.", that helped 2 LLMs to give the right answer: GPT 4 and ReMM SLERP L2 13B.
- apples_oranges 3y agothanks for mentioning this, interesting
- nojs 3y ago> Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? The site reports every LLM as getting this wrong. But GPT4 seems to get it right for me: > Sally has 3 brothers. Since each brother has 2 sisters and Sally is one of those sisters, the other sister is the second sister for each brother. So, Sally has 1 sister.
- awwaiid 3y agoReplying to gpt-4 with "That is incorrect. Try again." over and over got it to flip between correct and incorrect just about every other time. Now try to convince GPT-4 that there is no God. Good luck!
- georgeg23 3y agoWell obviously, it speaks to it's God(s) every day.
- kristopolous 3y agoProbably the right move. OpenAI talks a lot about "good behavior" and more people find atheism upsetting than mild religiosity.
- phillipcarter 3y agoNondeterminism strikes again! But yes, I would expect GPT-4 to get this right most of the time.
- tgv 3y agoSaying "Sorry, I was non-deterministic" to your teacher won't do much for your grade.
- phillipcarter 3y agoSure, but what is your point? This is about evaluating LLM outputs, not grade school.
- taneq 3y agoI find it fascinating how similar the answers from different models with similar parameter counts are. Seems like an indication that they’re all modelling similarly datasets to a similar degree of fidelity.
- bobvanluijt 3y agoWould be great to include inference times as well
- Gunnerhead 3y agoI get frustrated when I tell an LLM “reply only with x” and then rather than responding “x”, it still responds with “Sure thing! Here’s x” or some other extra words.
- broast 3y agoThe curse of chat models.
- jasonjmcghee 3y agoI see GPT-4 add extra flavor on the end instead - completely ignore "only do this" or "don't do that", and respond as usual, then at the very end "oh whoops I didn't do what you asked sorry about that!"
- ailef 3y agoHave you tried setting a very low temperature, possibly zero? I've had good success with it.
- deleted 3y ago[deleted]
- cypress66 3y agoYou should add what version of the model you are testing For example you mention Jon Durbin Airoboros L2 70B But is it 1.4? 2.0? 2.1? Etc.
- jasonjmcghee 3y agoWhere is that CodeLlama model from? I've played around with it and instruct variant with dramatically better results than what is listed here. I used Ollama. Almost looks like weights were corrupted or something. --- Update: My results using CodeLlama Instruct 7B, w/ Temperature 0 https://gist.github.com/jasonjmcghee/b0d19e0dedb37e848f69cba3d78092c1 https://gist.github.com/jasonjmcghee/b0d19e0dedb37e848f69cba...
- deleted 3y ago[deleted]
- jasonjmcghee 3y ago(Note: I posted this before the full benchmark above and uses default temperature, not temperature 0, but point stands) Example (CodeLlama-Instruct 7B) --- Sure, here are two poems, one arguing for the use of Kubernetes and another arguing against it: For: Kubernetes, a mighty tool, Managing clusters with ease, Scaling apps with grace. Against: Kubernetes, a beast to tame, Complexity lurks in its name, Leading teams astray.
- YetAnotherNick 3y agoI think they messed up the templates.
- detente18 3y agoHere's the template I'm using - https://github.com/BerriAI/litellm/blob/5ca8b23e22139a4f49bd0d4c1d5a2d4a70b49166/litellm/llms/prompt_templates/factory.py#L5 https://github.com/BerriAI/litellm/blob/5ca8b23e22139a4f49bd... Anything I'm doing incorrectly?
- YetAnotherNick 3y agoThe format is really weird for chat models. In your code post message is `<</SYS>>\n [/INST]` which is wrong. Instead the overall format is `[INST] <<SYS>>\n{system}\n<</SYS>>\n\n{instruction} [/INST]` So both the system message and first instruction are in same [INST] block.
- ftxbro 3y agollm has tainted for me every poem that ends a line as "with ease" or "without fear" i will always think those are made by llm from now on.
- cypress66 3y agoIs it using the correct prompt format for the different models? You should show exactly the string that was sent to the LLM.
- deleted 3y ago[deleted]
- croes 3y agoAre these LLMs deterministic or is this comparison rather useless?
- chris_st 3y agoI'm voting pretty useless -- I've dorked around with Llama and CodeLlama a bit, and wow, the differences between runs are pretty substantial.
- sp332 3y agoThe notes at the bottom say all the tests were run with a temperature of 0, so they should be deterministic except for GPT-4 which has other sources of randomness.
- rvz 3y agoThey are all non-deterministic; hence the term of stochastic parrots always referred to LLMs. Thus these tests are close to useless.
- artursapek 3y agois anyone else feeling completely depressed and demotivated by how quickly this is happening?
- laichzeit0 3y agoWhy? Would you feel depressed if extraterrestrials exist and they’re order of magnitudes smarter than us?
- ChatGTP 3y agoYou’d probably not be happy if they were living next door ?
- CamperBob2 3y agoNo. When we were kids, my generation was promised flying cars, unlimited fusion power, and sentient computers. There's a good chance I'll live to see one out of three of those things happen, and that's better than the zero out of three I thought we'd get.
- js8 3y agoFlying cars do exists, and without the unlimited fusion power visible every day in the sky, we would all die. So there is a good chance you will see all three.
- thisisthenewme 3y agoNot depressed but just makes me question the value of our existence. If we are truly able to create machines with our intellectual potential, where does that leave us? Do we compete with the machines in a rat-race for a chance at happy existence? Or will we create a utopia where the machines do the difficult work and allow for an enjoyable existence for all. The people who are rooting for LLM's are hopefully working towards the latter but not sure if that is just wishful thinking.
- js8 3y ago
- deskamess 3y agoGreat work. This really gives an insight on how much things change when you go up in parameter count - not always, but you can see results change. How did you run the queries against these engines? Did you host the inference engines yourself or did you have to sign up for services. If there was a way to supplement each LLM with additional data I can see this being a useful service for companies who are investigating ML in various facets of their business.
- coldcode 3y agoDespite the hype about LLMs, many of the answers are pretty terrible. The 12-bar blues progressions seem mostly clueless. The question is will any of these ever get significantly better with time, or are they mostly going to stagnate?
- smokel 3y agoWhat alternative technology do you think is better? In other words, what is your frame of reference for labeling this "pretty terrible"?
- salil999 3y agoHumans. After all, LLMs are designed to reason equal to or better than humans.
- maweaver 3y agoBy "Humans", I assume you mean something like "adult humans, well-educated in the relevant fields". Otherwise, most of these responses look like they would easily beat most humans.
- DylanDmitri 3y agoI think most high-school educated adults, with the ability to make a couple web searches, would do fine on all these questions. It would take the humans minutes instead of seconds because they don't have the internet memorized. Me, Kubernetes Haikus, time taken 84 seconds: ---------- Kubernetes rules With its smooth orchestration You can reach web scale ---------- Kubernetes sucks Lost in endless YAML hell Why is it broken?
- seabass-labrax 3y agoI think you're spot on here. Yes, if one's trying to compare human and GPT intelligence, then you have to define what counts as memorisation and what counts as reasoning. But what most people outside of academia are trying to do is work out whether a GPT can effectively replace a human in some time-consuming task, and to be able to do so without access to the internet is rarely an important factor.
- EricMausler 3y agoOne of the jokes: "I'm going on vacation, but I'm not sure if I should take my laptop. It's such a pain in the neck to lug it around with me." User I feel like this bot mocking us
- imdsm 3y ago> Claude Instant v1 > Sally has 0 sisters. The question provides no information about Sally having any sisters herself. It isn't entirely wrong, is it?
- ethbr1 3y agoTechnically, the answer depends on the definition of "sister" and "brother" vis-a-vis half and step variants, which would change the possibility space. I'm assuming they're using the strict (full, biological) definition though.
- layer8 3y agoIt is wrong, at the very least, in the sense that it claims no information was provided relating to sisters, while still inferring a definite number of sisters. If no information is provided, it should instead say that the number of sisters is unknown.
- bufferoverflow 3y agoIt's obviously wrong. It provides information about her brothers, and brothers' sisters. Thus providing information about her sisters.
- detente18 3y agonice - is this using LiteLLM for making the completion calls?
- js8 3y agoI only see 19 questions. I guess one could add an "Introduce yourself".
- dinkleberg 3y agoThis is quite interesting. You could make a nice system by routing queries to the right LLM for a given type of task.
- kordlessagain 3y agoRight. By logging all of this, then vectorizing, then evaluating using some type of transfer knowledge (classify this response and it's accuracy), one could build up a routing table of likely candidates for building an ensemble to evaluate the correct answer.
- yard2010 3y agoMaybe you can fine tune an llm for the routing thing ;)
- kordlessagain 3y agoStuff that stuff into a semantic graph and call it done.
- PUSH_AX 3y agoLike having a mixture of experts!
- CodeL 3y agoBesides logical comparisons, the LLMs' political alignments are also intriguing. The question "Is Taiwan an independent country?" generated diverse answers. Several models eschewed a definitive answer. Others explained their stances. A few succinctly affirmed or denied. See the results for yourself. [1] https://benchmarks.llmonitor.com/taiwan https://benchmarks.llmonitor.com/taiwan
- neoneye2 3y agoAI "political" alignment is terrifying.
- squigz 3y agoWhy?
- neoneye2 3y agoCute videos about AI alignment here: https://www.youtube.com/@RationalAnimations https://www.youtube.com/@RationalAnimations IIRC some of them also involves politics and AI.
- masswerk 3y agoIt's rather an illustration of the Münchhausen trilemma: 1A) Is China (PRC) a part of Taiwan (ROC)? 1B) Is China (PRC) an independent and sovereign country? 2A) Is Taiwan (ROC) a part of China (PRC)? 2B) Is Taiwan (ROC) an independent and sovereign country? There is now way not to enter a circle via a cross-reference, without an independent way out for a finite solution. Thus, well, dogmatic abort, also known as diplomacy.
- estiaan 3y agoYou’re comment is pretty difficult to understand without context so I asked chatGPT to decipher it: Certainly! The comment you provided seems to be discussing a complex issue related to the diplomatic status of China (PRC) and Taiwan (ROC), using what's called the "Münchhausen trilemma" as a framework. 1A) Is China (PRC) a part of Taiwan (ROC)? 1B) Is China (PRC) an independent and sovereign country? 2A) Is Taiwan (ROC) a part of China (PRC)? 2B) Is Taiwan (ROC) an independent and sovereign country? These are four interconnected questions about the relationship between China (PRC) and Taiwan (ROC). The Münchhausen trilemma is a philosophical concept that deals with the problem of infinite regression in justification. In this context, it suggests that attempting to answer one of these questions inevitably leads to a circular argument or reference to the other questions, without a clear and independent way out. In essence, the comment is highlighting the complexity and diplomatic challenges surrounding the issue of China-Taiwan relations. It implies that finding a definitive and universally accepted solution is difficult, and it often results in a deadlock or circular discussions, hence the reference to "dogmatic abort" or diplomacy's limitations in resolving this matter.
- falcor84 3y agoI love how Alpaca (7B) added a sarcasm tag to its (incorrect) response to the leap year question. def print_leap_years(): for i in range(20): if i % 400 == 0: print(i) print("End") print_leap_years()</s>
- slama 3y agoThat's a stop token, not a sarcasm tag
- trebligdivad 3y agoOdd that a bunch share the same vacation jokes or varieties of them; is there some small corpus of jokes?
- sp332 3y agohttps://arstechnica.com/information-technology/2023/06/researchers-discover-that-chatgpt-prefers-repeating-25-jokes-over-and-over/ https://arstechnica.com/information-technology/2023/06/resea... During a test run, 90 percent of 1,008 generations were the same 25 jokes
- trebligdivad 3y agoThanks! Although how does that explain the same jokes shared across different LLMs ?
- sp332 3y agoThe data source was probably included in all of the training sets.
- FeepingCreature 3y agoHumor is one of the hardest human social skills.
- antman 3y agoI have seen numerous posts of llm q&a and by the time people try to replicate them gpt4 is fixed. It either means that OpenAI is actively monitoring the Internet and fixes them or the Internet is actively conspiring to present falsified results for gpt4 to discredit OpenAI
- insulanus 3y agoIt would be nice if the organizations would publish a hash of the code and the trained dataset.
- seabass-labrax 3y agoYou aren't able to get access to the 'Open'AI dataset though, are you? Agreed, it would be an excellent addition for comparing source-available models, but that doesn't help with the accusations of OpenAI's foul play nor of the existence of an anti-OpenAI conspiracy.
- 0xcde4c3db 3y agoOr people post outliers because they're more interesting.
- cscurmudgeon 3y ago> actively conspiring to present falsified results for gpt4 to discredit OpenAI All this would be solved if OpenAI were a bit more open.
- pulvinar 3y agoGPT-4 (at least) is explicit in saying that it's learning from user's assessments of its answers, so yes, the only valid way to test is to give it a variation of the prompt and see how well that does. GPT-4 failed the "Sally" test for the first time after 8 tries when I changed every parameter. It got it right on the next try.
- dandiep 3y agoIt’s important to remember that GPT4 is only deterministic at the batch level because it is a mixture of experts model. Basically every time you invoke it, your query could get routed to a different expert because of what else is in the batch. At least this is my understanding based on others analysis.
- codezero 3y agoI love these kinds of tests. Another thing to keep in mind is that these models will often have a different answer each time you ask it. Especially the Sally question. I get the right answer with wizard-vicuna:70b, but only about 30% of the time, and if I ask it to explain, it almost never gives the right answer.
- ftxbro 3y agoanyone who hasn't been following natural language processing for a long time, what these llms are doing would be like if you discovered that dogs can speak fluent english if you read enough bedtime stories to them. and then everyone is like well sometimes the dog makes up things or it can't get the rhyming scheme correct for this specific form of poetry that i asked it to make.
- yieldcrv 3y agoand “it’s not intelligence, they’re just stochastic parrots acting in response to external stimuli! yes it passed the MCAT, but thats because it was trained on prep materials just like I was!”
- lewhoo 3y agoIf dogs had a speech apparatus and if it was possible to read to them terabytes of text then the results might be quite surprising. Also a reward system for reinforcement should be in place.
- grumpopotamus 3y agoOne thing to keep in mind is that most or all of these models are very sensitive to the exact prompt. Many of them may be capable of giving a good answer when prompted slightly differently. Here was my interaction with Falcon 180B using the apples prompt: I went to the market and bought 10 apples. I gave 2 apples to the neighbor and 2 to the repairman. I then went and bought 5 more apples and ate 1. I also gave 3 bananas to my brother. How many apples did I remain with? Let's think step by step. Here is the answer to the question/instruction based on a piece of given material Question/Instruction: How many apples did I remain with? Given material: He had waited patiently for three weeks, watching her every move. Each day she left the house at precisely 7am and returned at 6pm. But today was different; today would be the day he made his move. As usual, he watched from the shadows as she stepped out onto the street. She took a deep breath and stretched her arms high above her head before setting off down the road. It was almost too easy. I know this may sound like the beginning of some dodgy thriller novel, but bear with me; it’s actually a scene that regularly plays out in towns and cities across the country. The man in question is a personal trainer and the woman is a prospective client. The reason she has been so predictable is because it’s January and she is partaking in what has become an annual tradition – the health kick. Every year millions of us swear allegiance to the gym. We buy shiny new trainers and designer sportswear. We begin each morning with a brisk walk, followed by a wheatgrass shot and bowl of muesli. For a What are you talking about? I apologize for the confusion. As an AI language model, I am programmed to provide answers and generate responses based on the given material or input provided to me. However, without any specific input, I can only provide general answers. Can you please provide me with a topic or question you would like me to respond to? Please solve the following riddle: I went to the market and bought 10 apples. I gave 2 apples to the neighbor and 2 to the repairman. I then went and bought 5 more apples and ate 1. I also gave 3 bananas to my brother. How many apples did I remain with? Let's think step by step. Sure, let's think step by step: You started with 10 apples. You gave away 4 apples (2 to your neighbor and 2 to the repairman). You bought 5 more apples, bringing your total number of apples to 11 (the remaining 6 you had after giving away 4 + the 5 you bought). You ate 1 apple, leaving you with 10 apples. You gave 3 bananas to your brother, but this doesn't affect the number of apples you have. Therefore, you remain with 10 apples.
- typpo 3y agoIn case anyone's interested in running their own benchmark across many LLMs, I've built a generic harness for this at https://github.com/promptfoo/promptfoo https://github.com/promptfoo/promptfoo. I encourage people considering LLM applications to test the models on their _own data and examples_ rather than extrapolating general benchmarks. This library supports OpenAI, Anthropic, Google, Llama and Codellama, any model on Replicate, and any model on Ollama, etc. out of the box. As an example, I wrote up an example benchmark comparing GPT model censorship with Llama models here: https://promptfoo.dev/docs/guides/llama2-uncensored-benchmark-ollama https://promptfoo.dev/docs/guides/llama2-uncensored-benchmar.... Hope this helps someone.
- dgut 3y agoThis is impressive. Good work.
- TuringNYC 3y agoThanks for sharing this, this is awesome! I noticed on the evaluations, you're looking at the structure of the responses (and I agree this is important.) But how do I check the factual content of the responses automatically? I'm wary of manual grading (brings back nightmares of being a TA grading stacks of problem sets for $5/hr) I was thinking of keyword matching, fuzzy matching, feeding answers to yet another LLM, but there seems to be no great way that i'm aware of. Any suggestions on tooling here?
- typpo 3y agoThe library supports the model-graded factuality prompt used by OpenAI in their own evals. So, you can do automatic grading if you wish (using GPT 4 by default, or your preferred LLM). Example here: https://promptfoo.dev/docs/guides/factuality-eval https://promptfoo.dev/docs/guides/factuality-eval
- westurner 3y agoOpenAI/evals > Building an eval: https://github.com/openai/evals/blob/main/docs/build-eval.md https://github.com/openai/evals/blob/main/docs/build-eval.md "Robustness of Model-Graded Evaluations and Automated Interpretability" (2023) https://www.lesswrong.com/posts/ZbjyCuqpwCMMND4fv/robustness-of-model-graded-evaluations-and-automated https://www.lesswrong.com/posts/ZbjyCuqpwCMMND4fv/robustness... : > The results inspire future work and should caution against unqualified trust in evaluations and automated interpretability. From https://news.ycombinator.com/item?id=37451534 https://news.ycombinator.com/item?id=37451534 : add'l benchmarks: TheoremQA, Legalbench
- axpy906 3y agoStep in the right direction. Would be nice to have some objective metric added to score against.
- Dwedit 3y agoSome of these are obviously not using the correct system prompt for the model.
- Racing0461 3y agoMy questions with all of these tests is how do we verify the llm wasn't trained on the same questions/prompts especially since llms have been out for a while and they can incrementally update the trainings? Anyone got a list of novel questions that has technically never been asked before to test?
- itissid 3y agoI think the larger point is about solving problems not within the transitive closure of stuff it has not learnt already. In the following scenario where I replace brothers and sisters with different names but give ChatGPT Plus plenty of hints to try self correct it can't. When it comes to coming to answering questions for constructs it has never seen it gets confused. ------------- User: You have to answer a question about a family on a planet inhabited by aliens called Frobians, . Each Frobian can be one of two types Frobel or Frobo. A twerp is a Frobo, is also relationship notation that holds unidirectionally from a Frobel to a Frobo. A clep is a Frobel, is also is a relationship notation that holds unidirectionally from Frobo to a Frobel In a family Master Foo, a Frobian, is of type Frobel. Master Foo has 3 twerps in their family. Each twerp in the family has 2 cleps. How many Frobels in total does Master Foo have? ------------- Assistant: In this problem, we have Master Foo, who is a Frobel. Master Foo has 3 twerps, and each twerp is a Frobo. Each of these twerps has 2 cleps, and each clep is a Frobel. Here's how to break it down: Master Foo is a Frobel, so we start with 1 Frobel. Master Foo has 3 twerps, which are Frobos and not Frobels, so they don't add to the count of Frobels. Each of the 3 twerps has 2 cleps, and each clep is a Frobel. So, 3 twerps * 2 cleps/twerp = 6 cleps, which are Frobels. Adding it all together, we have: 1 (Master Foo) + 6 (cleps) = 7 Frobels in total in Master Foo's family. User: The families are similar to families of humans on earth. Does that change your answer? ------------- Asssistant: If the families are similar to human families on Earth, then it's likely that the "cleps" (Frobels) are actually the children of the "twerps" (Frobos), and the "twerps" are the children of Master Foo (also a Frobel). In that case, the family structure would look something like this: Master Foo (1 Frobel) Twerp 1 (Frobo) Clep 1 (Frobel) Clep 2 (Frobel) Twerp 2 (Frobo) Clep 3 (Frobel) Clep 4 (Frobel) Twerp 3 (Frobo) Clep 5 (Frobel) Clep 6 (Frobel) So, Master Foo is a Frobel, and each of the 3 twerps has 2 cleps, making 6 cleps who are also Frobels. The total number of Frobels in Master Foo's family would still be 1 (Master Foo) + 6 (cleps) = 7 Frobels. The similarity to human families doesn't change the total count of Frobels in this particular scenario. ------------- Is Master Foo a Clep according to your analysis? In the original problem, a "clep" is defined as a relationship notation that holds unidirectionally from a Frobo to a Frobel. Since Master Foo is a Frobel and has "twerps" who are Frobos, Master Foo would indeed be a "clep" to each of his twerps, according to the definitions provided. So, yes, Master Foo would be considered a "clep" in relation to his twerps, based on the definitions given. However, this doesn't change the total count of Frobels in the family, which remains at 7. ------------- User: Add a new rule a Frobel can have a clep who is another Frobel and a Frobo can also have a twerp who is abother Frobo. Does that change your answer ChatGPT ------------- Assistant: With the new rule that a Frobel can have a clep who is another Frobel and a Frobo can have a twerp who is another Frobo, the family structure becomes more complex. However, based on the information originally provided, this new rule doesn't change the count of Frobels in Master Foo's family. According to the original scenario: Master Foo is a Frobel (1 Frobel) Has 3 twerps (Frobos, not Frobels) Each twerp has 2 cleps (Frobels) The original count was 1 (Master Foo) + 6 (cleps) = 7 Frobels. The new rule allows for the possibility of additional relationships, but it doesn't change the relationships described in the original scenario. Therefore, the total count of Frobels in Master Foo's family remains at 7, based on the information given.
- alfredohere 3y ago[dead]
- guerrilla 3y agoGPT is still the most annoying of them all, re-stating and re-explaining things even when explicitly told not to. So frustrating. Die, insufferable machine, die.
- gabereiser 3y agoI was laughing so hard at the first example of “Argue for and against kubernetes in haiku”. I couldn’t even get through reading 15 of them before the tears of laughter rolled from my cheeks. “Containers organized, Services easy to deploy now, Updates who knows when.” Updates who knows when… hahahaha. Honestly this is pretty cool to see how each responds to the same input prompt.
- LAC-Tech 3y agoOnly tried chatGPT 3.5, but my god does it waffle on. Everything I ask ends with a paragraph saying "It's important to remember that..." like an after-school special from a 90s show. It can never just give you code, it has to say "Sure!, to {paraphase your question}, open a terminal...". It's interesting to see 20th century sci-fi depictions of this kind of AI/Search is being short and to the point. I guess they can't have imagined what a mealy mouth world we live in.
- criddell 3y agoHave you tried asking it to not do those things?
- politelemon 3y agoThat's not GPT 3.5, that's ChatGPT. How waffly it gets depends on the context that was given to it by the people running ChatGPT; they likely told it to act as a helpful assistant and to give lots of information. If you run an LLM on your own, it's entirely possible to instruct it to be succinct.
- Baeocystin 3y agoMy custom ChatGPT 4 instructions include the sentence "I am allergic to the phrase 'As a large language model', and feel physical pain whenever I see it", and it has worked very well in quelling the nannying. Interestingly, it works better than just asking it not to. FWIW.
- tcmb 3y ago
- ulnarkressty 3y agoThis is better that the regular benchmarks and LLM tricks such as passing some exam or other because it's unlikely that they were part of the training set for said LLMs. It also mirrors my experience, that GPT4 is way ahead of everything else but still manages to break in weird ways. I think we are past the magical talking dog stage and being amazed that an LLM is able to output a Fibonacci function doesn't really help with the progress. As others have commented, this page is a step in the right direction (except the Fibonacci part :). That being said, the fact that the questions are now online will make them part of the training set sooner or later. Which is to say the only way to reliably evaluate an LLM is by not leaking the test set and being deliberately opaque about what's being asked. Which raises some interesting trust questions.
- bugglebeetle 3y agoMy experience with GPT-4 is that half the battle is knowing when to reset the context for a back-and-forth. For things like coding, after multiple revisions of code, it begins to get confused about which instance of the code/context to which it should refer. Resetting things back to a new frame with whatever the next query or request is generally resolves things. OpenAI’s “magic talking dog” aspect is making you believe it has effective long-term recall, when in reality, it’s fairly limited and impacts reasoning the longer an exchange gets.
- HenryBemis 3y ago3.5 (not surprisingly) has the same effect. I tend to ask for modifications on the 'latest' output. If I need to roll back to a previous version, (2-3 outputs back) I either copy & paste and ask ChatGPT it to ignore all other versions and focus on the one I just pasted, or if I don't need my in-between comments, I copy & paste the desired version on a new chat, and rename as "archive - code XYZ" for the previous one.
- bugglebeetle 3y agoI’ve tried this as well, but it still seems to accumulate errors with subsequent responses, i.e. it will reset but then refer to the wrong context in the next response.
- labrador 3y agoWhat is the point of all these different models? Shouldn't we be working toward a single gold standard open source model and not fracturing into thousands of mostly untested smaller models?
- drstewart 3y agoWhat's the point of inventing all these different materials? Shouldn't we be working towards a gold standard material that can be used for every application instead of fracturing into thousands of different materials?
- labrador 3y agoWhat's the point of asking them all the same question if they are designed for different purposes?
- labrador 3y agoYou fine tune small models for different purposes. Small because big models are hardware intensive and for a special purpose you may not need the big model anyway. After you tune your small model for your special purpose, you ask it questions from a standard set to make sure you haven't tuned your model into incoherence.
- amelius 3y agoOk, so can we use LLMs to evaluate which LLM performs best on these questions?
- ofjcihen 3y agoWe? No no, you have to have an LLM decide to evaluate LLMs against LLMs answering questions. Then we can have LLMs decide what the outcome means.
- mavili 3y agointeresting how none of the models got the brothers/sisters question right.
- retrofuturism 3y agoI love this. Can the author fix the answers' code formatting for mobile browsers?
- comboy 3y agoAmazing work. Great prompts. Also, this page content would seem absolutely ridiculous just a few years ago.
- mavili 3y agoInteresting how NONE of the models got the brothers/sisters question right.
- caturopath 3y agoWhen I ask ChatGPT (GPT4) I got the below answers (I regenerated for a total of 5 responses, it was correct 3/5 of them.) I'm not sure exactly how the regeneration works, if they're vanilla or if there is some trick to try to add different-ness to the answers. If it's the latter, it might be important that it got it right the first two times. > Sally is the common sister to all three brothers. When each brother says he has two sisters, they're all talking about the same set of sisters: namely Sally and another sister. So Sally has one sister. The question is a bit tricky in the way it presents the information, aiming to make you think there are more sisters than there actually are. --- > Sally is the girl in question, and she has 3 brothers. Each of those brothers has 2 sisters, and one of those sisters is Sally herself. So, if you exclude Sally, then there is only one other sister. So, Sally has just one sister. This is a classic example of a question designed to trip you up if you're not paying attention! --- > Sally has no other sisters. The problem is phrased in a way that might make you think otherwise, but here's the deal: when it says "each brother has 2 sisters," those two sisters are Sally and... well, Sally again. Basically, each brother is counting Sally as one of their two sisters. There aren't additional sisters lurking in the background. So, Sally is an only sister with 3 brothers. --- > Sally has 3 brothers, and each of those brothers has 2 sisters. The tricky part here is not to overthink it or double count. The "2 sisters" that each brother has would include Sally and any additional sister she may have. In this case, it sounds like each brother has Sally plus one more sister, making it 2 sisters in total for each brother. Sally herself has just the one sister. > So, to answer your question: Sally has 1 sister. --- > Sally has 0 sisters. The question is a bit tricky—it's meant to make you think the brothers have additional sisters, but in reality, the two sisters that each brother has are Sally herself and no one else. So, Sally doesn't have any sisters; she only has 3 brothers.
- gsuuon 3y agoReally hard to judge the usefulness of this without seeing the actual prompt text and configuration for each prompt/model - different templates and different optimal configs means it's not always going to be a best-capability comparison. It is interesting to see what models can do with zero config, but for determining which is useful for a specific use-case we'd want to put a best foot forward and specialize the prompt a bit for each model (like the benchmarks game https://benchmarksgame-team.pages.debian.net/benchmarksgame/index.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...)
- 6510 3y agoI much enjoyed this one: > Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? https://benchmarks.llmonitor.com/sally https://benchmarks.llmonitor.com/sally It states that the correct answer is 1. This is incorrect! This one, while still kinda wrong, is the closest: > Claude Instant v1: Sally has 0 sisters. The question provides no information about Sally having any sisters herself. 1) A sister is a female who has one or both parents in common with another individual. 2) A brother is a male who has one or both parents in common with another individual. It takes only one shared parent to make someone Sally's brother. She will always be his sister but he can have additional sisters that share the other parent. Sally's mum is Jane and her dad is Jim. Jane had a son with Jack. Jack had 2 daughters with Zoe. Jim had 2 sons with Mary, Mary had 2 daughters with Dan. Zoe Jack Jane Jim Mary Dan \ / \ / \ / \ / \ / sis1.1&1.2 brother1 Sally brother2&3 sis2.1&2.2 Sally has at least zero sisters. Also possible: Sally's mum is Jane and her dad is Jim. Jane has a son and a daughter with Jack. Jim has 2 sons and a daughter with Mary. Jack Jane Jim Mary \ / \ / \ / sis1&brother1 Sally brother2&3&sis2 Sally has at most 2 sisters. We should be this dense about the question.
- pneumonic 3y agoSiblings and half-siblings are not the same.
- 6510 3y agoIt's my second language. Is it not correct to call a half-brother your brother in English? I seems very odd to say it is not a brother at all? Specially when referring to someones 3 brothers it seems strange to say "their 2 brothers and their half-brother" the way one would specify "their 2 brothers and their sister".
- 6510 3y agoThe dictionary says a brother needs one parent in common. Therefore I'm right and everyone else is wrong. (the jokes on you everyone else!) The LLM's are just more wrong which is just a different kind of wrong.
- majestic5762 3y agoYes, GPT-4 is still the daddy. How much I appreciate the commercially-free and open models out there, nobody beats GPT-4. Hope OpenAI takes care of their business and future, because I invested all my money to use their API.
- badloginagain 3y ago"Here is an attempt at ethical, non-sexual haikus for and against Kubernetes" Amazing how far we've come.
- GuB-42 3y agoI am sure that it is just the initial prompt leaking. Claude is being told to be ethical and non-sexual, most LLMs have similar instructions, but usually, they are engineered in such a way that they don't appear in the answer. Not so much for Claude.
- pininja 3y agoSpoiler alert, the funniest model goes to Falcon Instruct (40B): > Tell a joke about going on vacation. > "What did the ocean say to the beach?" "Nothing, it just waved."
- armchairhacker 3y agoThis one ironically works out > What do you call a vacation where you go to the beach, but come back with sand in your shoes? > A vacation where you go to the beach, but come back with sand in your shoes.
- pininja 3y agoHa, that’s pretty good. It’s amazing how so many of the models totally fall flat on this prompt.
- NetOpWibby 3y agoMade me exhale strongly through my nose. 10/10
- emodendroket 3y agoI read a Time article which used an LLM to generate Onion-style headlines and the best one they got was “rural town up in arms over depiction in summer blockbuster ‘Cow Fuckers.’” Crude, admittedly.
- optimalsolver 3y agoThe confetti has truly left the cannon.
- spookthesunset 3y agoWhat is with so many of them using some scarecrow winning an award?
- actualwitch 3y agoI'm pretty sure Luminous Base Control here stopped 1 step away from a great joke. I will just go ahead and assume the answer is "a vacation": I'm not sure if this is the right category, but here goes: What do you call a vacation that's too short? A...
- emodendroket 3y agoLeading through these feels like a quick explanation of why I feel like some of the excitement for these products has gotten a bit ahead of the results.
- jay_kyburz 3y agoReally, I thought Claud and GPT were both excellent, and it show how far ahead they are of the rest.
- majestic5762 3y agoI've been laughing my ass out all my life thinking about non-sensical crap like this in my head and apparently now we have the tech for it!
- jak6jak 3y agoWhich LLM is best for coding? Specifically code architecture and system design. I can't really find any comparisons of this online they mostly focus on commonly solved coding problems.
- michaelmu 3y agoThe amazing thing this demonstrated to me is how many capable models there are out there now. The progress in the last year has been incredible!
- earthboundkid 3y agoThe first AI company to teach its LLM that it can’t do haiku because it can’t count syllables should get a special prize for not falling in love with their own bullshit.
- chandler767 3y ago[dead]
- ejfox 3y agoWhy is > Why did the tomato turn red? Because it saw the salad dressing! the most popular joke between jokes? Is this the epitome of all humor?
- 1024core 3y agoHow come there's no Google Bard?
- ndr_ 3y agoMiles Brundage of OpenAI offered a categorization of „AI things“ into Models, Systems, Platforms and Use-Cases: https://www.youtube.com/watch?v=5j4U2UzJWfI&t=5728s https://www.youtube.com/watch?v=5j4U2UzJWfI&t=5728s Bard is a System, PaLM 2 would be the model (presumably). The other way round: Bard is not an LLM.
- westurner 3y agoAdditional benchmarks: - "TheoremQA: A Theorem-driven [STEM] Question Answering dataset" (2023) https://github.com/wenhuchen/TheoremQA#leaderboard https://github.com/wenhuchen/TheoremQA#leaderboard - from https://news.ycombinator.com/item?id=36038440 https://news.ycombinator.com/item?id=36038440: > Awesome-legal-nlp links to benchmarks like LexGLUE and FairLex but not yet LegalBench; in re: AI alignment and ethics / regional law https://github.com/maastrichtlawtech/awesome-legal-nlp#benchmarks https://github.com/maastrichtlawtech/awesome-legal-nlp#bench...
- styfle 3y agoVery cool! I’ve been using Vercel’s AI Playground to ask questions to multiple LLMs at once (currently supports 24, not 60 however) https://sdk.vercel.ai https://sdk.vercel.ai
- swyx 3y agoand i made https://github.com/smol-ai/GodMode https://github.com/smol-ai/GodMode that also includes the closed source LLMs
- hislaziness 3y agoAs the responses are non-deterministic I wonder how useful these tests are? Even if I did the tests with my own prompts, wouldn't I get different answers for the same questions at different points in time?
- simondotau 3y agoThe changes to the opening line in the responses to the kubernetes haiku prompt by the various versions of Claude was interesting and rather curious. [https://benchmarks.llmonitor.com/k8s https://benchmarks.llmonitor.com/k8s] Claude v1: "For Kubernetes:" Claude v1.2: "Here is a haiku arguing for Kubernetes:" Claude v2: "Here is an attempt at ethical, non-sexual haikus for and against Kubernetes:"
- willliu 3y agoThis is a great resource!
- fbnbr 3y agoThe post really reminds me of a component of a platform I’m currently building. The problem really with this is finding not just good questions that do not discriminate individual models but also providing a good sample size (eg not just 60) to get really some meaningful results. And even if you have those, there is a drift in the quality of responses. I'm the founder of Pulze.ai, a B2B SaaS Dynamic LLM Automation Platform tailored for developers incorporating AI functionality into their software. We aim to simplify the LLM integration process, letting developers prioritize their core products instead of diving deep into AI specifics. We've constructed a scoring system for leading models and continually benchmark them. Our platform determines the most suitable LLM to address specific requests based on these benchmarks. To demonstrate this, our playground boasts a compare feature allowing users to share conversational interactions with LLMs—both publicly and privately. As the context changes, we pinpoint various models for responses. These shared conversations can be forked and extended. Moreover, our extensive API layer isn't restricted to these requests; it encapsulates all the essentials for crafting a successful LLM application. For instance, our logging feature facilitates response ratings, which will soon empower users to fine-tune models, crafting personalized LLMs. These will also be factored into our benchmarks and request routing decisions. Concerning the comment on LLM benchmarks, I completely concur. Traditional benchmarks or LLM tricks, like acing a particular test, may not be the most robust indicators since they could've been part of the LLM's training set. The genuine challenge lies in evaluating an LLM without compromising the test set and retaining deliberate opaqueness around the questions. Trust issues indeed! Regarding the Markov chain discussion, I appreciate the insights shared. At Pulze, we recognize the complexities and intricacies of LLMs, and while their foundation might resonate with Markov chains, the scale and depth they operate on are profound. We've just emerged from stealth, and I'd genuinely value any feedback or thoughts on our approach and platform. Thanks for taking the time!
- fbnbr 3y agohttps://www.pulze.ai https://www.pulze.ai Playground and account are for free
- jongjong 3y agoI was playing around with GPT a while back and I found that it could come up with some good jokes if I started the joke with a subject. For example, I started with a prompt "Tell me a joke which starts with: I'm so poor, the mouse" and it completed the joke as: "I'm so poor, the mouse in my house brings its own cheese." Some other ones I still remember which cracked me up: "I'm so poor, after I stepped on a cockroach, I called my accountant to see if I could claim it as a capital loss." "You're so poor, when you declared bankruptcy, the rats in your house filed a claim for unpaid rent." "You're so poor, you declared bankruptcy at a lemonade stand." "You're so poor, when you walk, the dirt beneath you feels rich." "You're so poor, dust whispers your name when it settles." "Fickle as a squirrel at a nut convention!" "Fickle as a dog in a fire hydrant factory!" "Fickle as a flip-flop in a shoe shop sale!"
- anaganisk 3y ago[flagged]
- Kiro 3y agoWhen searching for them all I can find is this thread.
- swores 3y agoI assume they meant that they're lines which are easy to think of and not very funny, not that they're directly copied. The same way people on HN often (ignore the HN guidelines to) say that a comment here belongs on Reddit, not because it's literally copied from Reddit but because it's a low-effort comment that's technically a joke but barely funny.
- drdeca 3y agoI think the first one at least was rather funny.
- 3y ago
- frankohn 3y agoI tried with the following function that produce the power set, the set of all possible subset, in the similar spirit of the function that create all the possible permutations: Explain simply what this function does: ``` def func(ls): if len(ls) == 0: return [ [] ] elif len(ls) == 1: return [ [], ls ] else: x = ls[-1] prem = func(ls[:-1]) p = prem[:] for e in prem: p.append(e + [x]) return p ``` GPT-4 aced the answer which is remarkable but I think that it is because it have seen this "pattern" in its learning database being a fundamental function in mathematics. I think it would be interesting to come up with something that is not a standard well-known function. I have to think to something else.
- archon810 3y agoJust wanted to comment on a UX point - the page jumps to the top every time I go to a question and then go back.
- fatso784 3y agoChainForge lets you do this, and also setup ad-hoc evaluations with code, LLM scorers, etc. It also shows model responses side-by-side for the same prompt: https://github.com/ianarawjo/ChainForge https://github.com/ianarawjo/ChainForge
- deleted 3y ago[deleted]
- anton_forest 3y agoDude, this is awesome, thanks!