9 ms·
GPT-4o's Memory Breakthrough – Needle in a Needlestack
- parrt 2y agoThe article shows how much better GPT-4o is at paying attention across its input window compared to GPT-4 Turbo and Claude-3 Sonnet. We've needed an upgrade to needle in a haystack for a while and this "Needle In A Needlestack" is a good next step! NIAN creates a prompt that includes thousands of limericks and the prompt asks a question about one limerick at a specific location.
- mianos 2y agoI agree, I paid for Claude for a while. Even though they swear the context is huge and having a huge context uses up tokens like crack, it's near useless when source code in context just a few pages back. It was so frustrating as everything else was as good as anything and I liked the 'vibe'. I used 4o last night and it was still perfectly aware of a C++ class I pasted 20 questions ago. I don't care about smart, I care about useful and this really contributes to the utility.
- dmose2 2y agoIt's interesting (though perhaps not surprising) to see the variance in curve shape across models.
- 19h 2y agoI'd like to see this for Gemini Pro 1.5 -- I threw the entirety of Moby Dick at it last week, and at one point all books Byung Chul-Han has ever published, and it both cases it was able to return the single part of a sentence that mentioned or answered my question verbatim, every single time, without any hallucinations.
- parrt 2y agoWow. Cool. I have access to that model and have also seen some impressive context extraction. It also gave a really good summary of a large code base that I dumped in. I saw somebody analyze a huge log file, but we really need something like this needle in a needlestack to help identify when models might be missing something. At the very least, this could give model developers something to analyze their proposed models.
- 19h 2y agoFunnily enough I ran a 980k token log dump against Gemini Pro 1.5 yesterday to investigate an error scenario and it found a single incident of a 429 error being returned by a third-party API provider while reasoning that "based on the file provided and the information that this log file is aggregated of all instances of the service in question, it seems unlikely that a rate limit would be triggered, and additional investigation may be appropriate", and it turned out the service had implemented a block against AWS IPs, breaking a system that loads press data from said API provider, leaving the customer who was affected by it without press data -- we didn't even notice or investigate that, and Gemini just randomly mentioned it without being prompted for that.
- parrt 2y agoThat definitely makes it seem like it's noticing a great deal of its context window. impressive.
- sftombu 2y agoIf I had access to Gemini with a reasonable token rate limit, I would be happy to test Gemini. I have had good results with it in other situations.
- cj 2y agoWhat version of Gemini is built into Google Workspace? (I just got the ability today to ask Gemini anything about emails in my work Gmail account, which seems like something that would require a large context window)
- underlines 2y agoSuch tasks don't need a large context window. Just good RAG.
- Fernicia 2y agoBut this content is presumably in its training set, no? I'd be interested if you did the same task for a collection of books published more recently than the model's last release.
- ben_w 2y agoI would hope that Byung-Chul Han would not be in the training set (at least not without his permission), given he's still alive and not only is the legal question still open but it's also definitely rude. This doesn't mean you're wrong, though.
- sebzim4500 2y agoIt's pretty easy to confirm that copywritten material is in the training data. See the NYT lawsuit against OpenAI for example.
- ben_w 2y agoPart of that back-and-forth is the claim "this specific text was copied a lot all over the internet making it show up more in the output", and that means it's not a useful guide to things where one copy was added to The Pile and not removed when training the model. (Or worse, that Google already had a copy because of Google Books and didn't think "might training on this explode in our face like that thing with the Street View WiFi scanning?")
- 19h 2y agoTo test this hypothesis, I just took the complete book "Advances in Green and Sustainable Nanomaterials" [0] and pasted it into the prompt, asking Gemini: "What absorbs thermal radiations and converts it into electrical signals?". It replied: "The text indicates that graphene sheets present high optical transparency and are able to absorb thermal radiations with high efficacy. They can then convert these radiations into electrical signals efficiently.". Screenshot of the PDF with the relevant sentence highlighted: https://i.imgur.com/G3FnYEn.png https://i.imgur.com/G3FnYEn.png [0] https://www.routledge.com/Advances-in-Green-and-Sustainable-Nanomaterials-Applications-in-Energy-Biomedicine-Agriculture-and-Environmental-Science/Goyal-Kulkarni/p/book/9781774911662 https://www.routledge.com/Advances-in-Green-and-Sustainable-...
- deleted 2y ago[deleted]
- DominikPeters 2y agoJust put the 2500 example linked on the article through Gemini 1.5 Flash and it answered correctly ("The tree has diseased leaves and its bark is peeling.") https://aistudio.google.com/ https://aistudio.google.com/
- sftombu 2y agoInteresting!
- nsagent 2y agoA number of people in my lab do research into long context evaluation of LLMs for works of fiction. The likelihood is very high that Moby Dick is in the training data. Instead the people in my lab have explored recently published books to avoid these issues. See BooookScore (https://openreview.net/forum?id=7Ttk3RzDeu https://openreview.net/forum?id=7Ttk3RzDeu) which was just presented at ICLR last week and FABLES (https://arxiv.org/abs/2404.01261 https://arxiv.org/abs/2404.01261) a recent preprint.
- robbiep 2y agoI’m not involved in the space, but it seems to me that having a model, in particular a massive model, exposed to a corpus of text like a book in the training data would have very minimal impact. I’m aware that people have been able to return data ‘out of the shadows’ pf the training data but to my mind a model being mildly influenced by the weights between different words in this text hardly constitute hard recall, if anything it now ‘knows’ a little of the linguistic style of the authour. How far off am I?
- int_19h 2y agoIt depends on how many times it had seen that text during training. For example, GPT-4 can reproduce ayats from the Quran word for word in both Arabic and English. It can also reproduce the Navy SEAL copypasta complete with all the typos.
- Salgat 2y agoRemember, it's also trained on countless internet discussions and papers on the book.
- causality0 2y agoMan, we are like 2-5 years away from being able to feed in an ePub and get an accurate graphic novel version in minutes. I am so ready to look at four thousand paintings of Tolkien trees.
- nickca 2y agoWould love to see Gemini there too!
- personjerry 2y agoThat's great to hear. My biggest issue with GPT-4.0 was that as the conversation got longer, the quality diminished (especially relevant for coding projects) I wonder if it'll be better now. Will test today.
- sftombu 2y agoI had the same experience. With a 16k prompt, Turbo was nearly flawless. But it wasn't very good at 32k and not usable at 100+. You have to repeat information to get good results with longer prompts
- throwthrowuknow 2y agoThat’s been my experience so far. My current conversations are crazy long compared to any of my gpt4 convos which I had to frequently copy context from and start over in a new chat
- youssefabdelm 2y agoSomeone needs to come up with a "synthesis from haystack" test that tests not just retrieval but depth of understanding, connections, abstractions across diverse information. When a person reads a book, they have an "overall intuition" about it. We need some way to quantify this. Needle in haystack tests feel like a simple test that doesn't go far enough.
- sftombu 2y agoI was thinking about something similar -- to make part of the question be sufficient information that the LLM can find the limerick. Then the 2nd part would ask something that would require a deeper understanding of the limerick (or other text).
- adamgordonbell 2y agoI've been thinking about that as well. It's hard, but if you have a piece of fiction or non-fiction it hasn't seen before, then a deep reading comprehension question can be a good indicator. But you need to be able to separate a true answer from BS. "What does this work says about our culture? Support your answer with direct quotes." I found both gpt-4 and haiku to do alright at this, but sometimes give answers that imply fixating on certain sections of a 20,000 k context. You could compare it against chunking the text, getting the answer for each chunk and combining them. I suspect if you do that then the chunking would win for things that are found in many chunks, like the work is heavy handed on a theme, but the large context would be better for a sublter message, except sometimes it would miss it altogether and think a Fight Club screenplay was a dark comedy. Interpretation is hard I guess.
- deleted 2y ago[deleted]
- Eisenstein 2y agoMy idea is to buy to a unpublished novel or screenplay with a detailed, internally consistent world built in to it and a cast of characters that have well crafted motivations and then ask it to continue writing from an arbitrary post-mid-point by creating a new plot line that combines two characters that haven't yet met in the story. If it understands the context it should be able to write a new part of the story and will be able to use a reader's intuitive sense of the character's motivations to move through their arc. This whole thing would have to be kept under lock-and-key in order to be useful, so it would only serve as a kind of personal benchmark. Or it could possibly be a prestige award that is valued for its conclusions and not for its ability to use the methodology to create improvements in the field.
- petulla 2y agoYou need to know that this test set data wasn't included in the training data for this to be meaningful.
- sftombu 2y agoIf you ask the questions without providing the limerick first, it never gets the right answer. When the LLM gets the wrong answer, it is usually because it reverts to its training data and gives a generic answer that doesn't apply to the limerick.
- trifurcate 2y agoWhy are you ruling out the possibility that training on the material may confer an advantage when the data is presented, even if the advantage may not be strong enough to pass the test without the data present in the context window?
- lmeyerov 2y agoI thought the test limericks were autogenerated?
- sftombu 2y agoThey come from a database of 98k limericks -- https://zenodo.org/records/5722527 https://zenodo.org/records/5722527
- a_wild_dandan 2y agoNo you don't. Compare the model's performance before and after uploading the material.
- sftombu 2y agoPrevious answer to this question: https://news.ycombinator.com/item?id=40361419s https://news.ycombinator.com/item?id=40361419s
- asadm 2y agoI have had good experience with Gemini 1M context model with this kind of tasks.
- rguptill 2y agoWe also need a way to determine where a given response fits in the universe of responses - is it an “average” answer or a really good one
- edmara 2y agoIf you have an evaluation function which does this accurately and generalizes, you pretty much already have have AGI.
- m3kw9 2y agoI thought google Gemini had almost perfect needle in haystack performance inside 1 million tokens?
- sftombu 2y agoThe reason I made Needle in a needlestack is the LLMs are getting to good at needle in a haystack. Until GPT-4o, no model was good at the NIAN benchmark.
- m3kw9 2y agoOne could have LLM to route it to a text search function and have the function report back to the LLM for secondary processing.
- cararemixed 2y agoWhat's the chance that these limericks are now in the training set? As others mention, it'd be interesting to come up with a way to synthesize something sufficiently interesting so it always evades training fit.
- sftombu 2y agoPrevious answer to this question: https://news.ycombinator.com/item?id=40361419 https://news.ycombinator.com/item?id=40361419
- deleted 2y ago[deleted]
- causal 2y agoYour test is a good one but the point still stands that a novel dataset is the next step to being sure.
- dontupvoteme 2y agoOne could also programmatically (e.g. with nltk or spacy, replace nouns, named entities, etc) modify the dataset, even up to the point that every test run is unique. You could also throw in vector similarity if you wanted to keep words as more synonyms or antonyms.
- thorum 2y agoThe needle in the haystack test gives a very limited view of the model’s actual long context capabilities. It’s mostly used because early models were terrible at it and it’s easy to test. In fact, most recent models now do pretty good at this one task, but in practice, their ability to do anything complex drops off hugely after 32K tokens. RULER is a much better test: https://github.com/hsiehjackson/RULER https://github.com/hsiehjackson/RULER > Despite achieving nearly perfect performance on the vanilla needle-in-a-haystack (NIAH) test, all models (except for Gemini-1.5-pro) exhibit large degradation on tasks in RULER as sequence length increases. > While all models claim context size of 32k tokens or greater (except for Llama3), only half of them can effectively handle sequence length of 32K by exceeding a qualitative threshold, Llama2-7b performance at 4K (85.6%). The performance exceeding the threshold is underlined.
- WhitneyLand 2y agoMaybe, but 1. The article is not about NIHS it’s their own variation so it could be more relevant. 2. The whole claim of the article is that Gpt4o does better, but the test your pointing to hasn’t benchmarked it.
- sftombu 2y agoThe models benchmarked by RULER do worse in needle in a needlestack. It will be interested to see how 4o does with RULER.
- throwthrowuknow 2y agoThis is a very promising development. It would be wise for everyone to go back and revise old experiments that failed now that this capability is unlocked. It should also make RAG even more powerful now that you can load a lot more information into the context and have it be useful.
- demilich 2y agoAgreed
- deleted 2y ago[deleted]
- throw7381 2y agoAnyone has done any benchmarks for RAG yet?
- itissid 2y agoHow Do we know that gpt-4o.has not been trained on this dataset?
- sftombu 2y agoPrevious answer to this question: https://news.ycombinator.com/item?id=40361419 https://news.ycombinator.com/item?id=40361419
- irthomasthomas 2y agoThis is based on a limericks dataset published in 2021. https://zenodo.org/records/5722527 https://zenodo.org/records/5722527 I think it very likely that gpt-4o was trained on this. I mean, why would you not? Innnput, innnput, Johnny five need more tokens. I wonder why the NIAN team don't generate their limericks using different models, and check to make sure they're not in the dataset? Then you'd know the models couldn't possibly be trained on them.
- sftombu 2y agoI tested the LLMs to make sure they could not answer the questions unless the limerick was given to them. Other than 4o, they do very badly on this benchmark, so I don't think the test is invalidated by their training.
- cma 2y agoWhy wouldn't it still be invalidated by it if it was indeed trained on it? The others may do worse and may or may not have been trained on it, but them failing on ititself doesn't imply 4o can do this well without the task being present in the corpus.
- sftombu 2y agoIt can't answer the questions without the limericks in the prompt. The benchmark is to establish how well it uses the context window. For example, I just asked it "What is sought by the English top brass?". The answer from the limerick is "Cranberry glass" and 4o answers correctly when given the associated limerick once out of 2500+ limericks. However, without the limerick, 4o responded with: "The term "English top brass" typically refers to high-ranking officials or leaders within the British government, military, or other institutions. What they seek can vary widely depending on the context and the specific goals of their roles. Here are some general pursuits that might be sought by such individuals: National Security: Ensuring the safety and security of the United Kingdom from internal and external threats is a primary concern. This involves defense strategies, intelligence operations, and counter-terrorism efforts. Economic Stability: High-ranking officials often focus on policies and initiatives aimed at maintaining and improving the country’s economic health. This includes managing inflation, unemployment, trade relations, and economic growth. Political Influence: Top brass often seek to maintain or expand their influence both domestically and internationally. This can involve diplomacy, forming alliances, and participating in international organizations like the United Nations or NATO. Social Cohesion: Ensuring social stability and addressing issues such as inequality, healthcare, education, and social services are critical. This can involve implementing policies that promote social welfare and cohesion. Public Policy Implementation: Leaders are responsible for developing and implementing policies that reflect the government’s priorities. This includes legislation, regulatory frameworks, and public administration. Technological Advancement: Keeping the nation at the forefront of technological innovation is often a priority. This includes investments in research and development, supporting tech industries, and ensuring cybersecurity. Environmental Sustainability: Addressing climate change and promoting sustainable practices are increasingly important. This includes policies aimed at reducing carbon emissions, protecting natural resources, and transitioning to renewable energy sources. Cultural and Heritage Preservation: Protecting and promoting the country’s cultural heritage and national identity can also be a focus. This includes supporting the arts, preserving historical sites, and promoting cultural initiatives. These pursuits are shaped by the current political climate, global trends, and the specific priorities of the leaders in question. Would you like more detailed information on any of these areas?"
- ionwake 2y agoI am in England, do US users have access to memory features? ( Also do you ahve access to voice customisation yet? Thanks
- whimsicalism 2y agoIncreasingly convinced that nobody on the public internet knows how to do actual LLM evaluations.
- tedeh 2y agoI'm just glad that we are finally past the "Who was the 29th president of the United States" and "Draw something in the style of Van Gogh" LLM evaluation test everyone did in 2022-2023.
- yatz 2y agoWell, I can now use GPT to transform raw dynamic data into beautiful HTML layouts on the fly for low-traffic pages, such as change/audit logs, saving a ton of development time and keeping my HTML updated even when the data structure has changed. My last attempt did not consistently work because GPT4-Turbo sometimes ignored the context and instructions almost entirely.
- ijidak 2y agoDo you have an example of this? I would love to learn more.
- balder1991 2y agoI guess you just need to offer a template in the prompt? Then maybe some validation after.
- yatz 2y agoNo templates, just some rules and the model does the rest. It worked like a charm, even gave me ideas on how to layout and format the page to make it easy to read.
- yatz 2y agoHere is the entire prompt. I used rules to ensure the formatting is consistent as otherwise sometimes it might format date one way and other times in an entirely different way. Imagine, a truly dynamic and super personal site, where layout, navigation, styling and everything else gets generated on the fly using user's usage behavior and other preferences, etc. Man! --------------------------------------------- {JSON} ------ You are an auditing assistant. Your job is to convert the ENTIRE JSON containing "Order Change History" into a human-readable Markdown format. Make sure to follow the rules given below by letter and spirit. PLEASE CONVERT THE ENTIRE JSON, regardless of how long it is. --------------------------------------------- RULES: - Provide markdown for the entire JSON. - Present changes in a table, grouped by date and time and the user, i.e., 2023/12/11 12:40 pm - User Name. - Hide seconds from the date and time and format using the 12-hour clock. - Do not use any currency symbols. - Format numbers using 1000 separator. - Do not provide any explanation, either before or after the content. - Do not show any currency amount if it is zero. - Do not show IDs. - Order by date and time, from newest to oldest. - Separate each change with a horizontal line.
- causality0 2y agoI don't understand OpenAI's pricing strategy. For free I can talk to GPT 3.5 on an unlimited basis, and a little to GPT 4o. If I pay $20 a month, I can talk to GPT 4o eighty times every three hours, or once every two and a half minutes. That's both way more than I need, and way less than I would expect for twenty dollars a month. I wish they had a $5 per month tier that included, say, eighty messages per 24-hours.
- hackerlight 2y agoIt'll make more sense when they deploy audio and image capability to paying users only, which they say they're going to do in a few weeks
- causality0 2y agoYeah, but I want a tier where I have access to it in a pinch, but won't feel guilty for spending the money and then going a whole month without using it.
- olddustytrail 2y agoGuilty? Over $20 a month? I spend more than that in an hour down the pub.
- causality0 2y agoYou get something for your money at the pub. If you pay twenty dollars for a service and then don't use it you have wasted your money.
- olddustytrail 2y agoWell, yes, I get more liver damage. I wouldn't worry too much about $20. Give an equal amount extra to charity or bet on something. There was a time that amount mattered to me but if you're working any kind of dev/ops job it just doesn't any more.
- croes 2y ago>Needle in a Needlestack is a new benchmark to measure how well LLMs pay attention to the information in their context window I asked GPT-4o for JavaScript code and got Python, so much for attention.
- kolinko 2y agoWhat was your query?
- 8thcross 2y agoThese benchmarks are becoming like the top 10 lists you find on the internet. I agree that everything has a space, but frankly how many of us need a test that tells you that this is great at limericks?
- bearjaws 2y agoI just used it to compare two smaller legal documents and it completely hallucinated that items were present in one and not the other. It did this on three discrete sections of the agreements. Using ctrl-f I was able to see that they were identical in one another. Obviously this is a single sample but saying 90% seems unlikely. They were around ~80k tokens total.
- kylebenzle 2y agoWhat you are asking an llm to do here makes no sense.
- potatoman22 2y agoWhy not? It seems like a natural language understanding task
- marshray 2y agoYou haven't seen the promotion of the use of LM AI for handling legal documents? It's purported to be a major use case.
- cmrdporcupine 2y agoYou might be right but I've lost count of the number of startups I've heard of trying to do this for legal documents.
- bckr 2y agoYeah I asked for an estimate of the percentage of the US population that lives in the DMV area (DC, Maryland, Virginia) and it was off by 50% of the actual answer, which I only realized when I realized I shouldn’t trust its estimate for anything important
- KeplerBoy 2y agoThose models still can't reliably do arithmetic, so how could it possibly know that number unless it's a commonly repeated fact? Also: would you expect random people to fare any better?
- EGreg 2y agoI think large language models can be used to classify people, lying, or saying, rehearsed, things or being disingenuous. Simply train them on a lot of audio of people talking, and they would become better than most polygraph machines. There’s something about how a person says something that quickly reveals that it was rehearsed earlier, or premeditated, and I’m sure when they’re lying there can be things like that too. the LLM can instantly pick up with some probability and classify it I’ve seen claims during open AI demo that is there software can now pick up on extremely subtle emotional clues, how people speak. Then, it shouldn’t take much more to make it read between the lines and understand what people are intending to say, for example, by enumerating all possible interpretations and scoring them based on, many factors, including the current time, location, etc. In fact, by taking into account so much context in factors, the LLM‘s will be better than people the vast majority of the time understanding what a person meant, assuming they were genuinely trying to communicate something. it will become very hard to lie because everyone’s personal LLM will pick up on it fairly quickly, and find tons of inconsistencies, which it will flag for you later. You will no longer be fooled so easily, and if it has the context of everything the person has said publicly, plus if the person gives permission for your LLM to scan everything they’ve said privately because you’re their Business partner or sexual partner, it can easily catch you in many lies and so on. I predict that in the next 5 to 10 years, human society will completely change as people start to prefer machines to other people, because they understand them so well, and taken into account, the context of everything they’ve ever said. They will be thoughtful, remembering details about the person in many different dimensions, and use them to personalize everything. By contrast, the most thoughtful husband or boyfriend will seem like, a jerk seems now. Or a cat. Humor and seductive conversation, will also be at a superhuman standards. People will obviously up their game too, just like when they do when playing the game go after Lee Sedol was totally destroyed by Alpha go, or when people start using Alpha Zarro to train for Chess. However, once the computers understand what triggers people to laugh or have sexual response, they will be able to trigger them a lot more predictively, they simply need more training data. And bullshitting will be done on a completely different level. Just like people no longer walk to destinations but use cars to go thousands of miles a year, similarly people won’t interact with other people so much anymore. The LLM’s, trained to bullshit 1000 times better than any human, Will be undetectable and gradually shift public opinion as open source models will power swarms of accounts.
- sschueller 2y agoWe are all so majorly f*d. The general public does not know nor understand this limitation. At the same time OpenAI is selling this a a tutor for your kids. Next it will be used to test those same kids. Who is going to prevent this from being used to pick military targets (EU law has an exemption for military of course) or make surgery decisions?
- fragmede 2y agoOpenAI is. Their TOS says don't use it for that kind of shit. https://openai.com/policies/usage-policies/ https://openai.com/policies/usage-policies/
- tsimionescu 2y agoThat's the license for the public service. Nothing prevents them from selling it as a separate package deal to an army.
- Arn_Thor 2y agoIf any regulator acts it will be the EU. The action, if it comes, will of course be very late, possibly years from now, when the horse has long left the stable.
- sschueller 2y agoMy only hope for the EU government is that they put and AI in charge and it accidentally becomes sentient...
- rolandog 2y ago> or make surgery decisions? Analyzing surgical field... Identified: open chest cavity, exposed internal organs Organs appear gooey, gelatinous, translucent pink Comparing to database of aquatic lifeforms... 93% visual match found: Psychrolutes marcidus, common name "blobfish" Conclusion: Blobfish discovered inhabiting patient's thoracic cavity Recommended action: Attempt to safely extract blobfish without damaging organs
- tartrate 2y agoAre there any prompts/tests about recalling multiple needles (spread out) at once? For example, each needle could be a piece to a logic puzzle.
- feverzsj 2y agoLLMs are still toys, no one should treat them seriously. Apparently, the bubble is too massive now.
- cdelsolar 2y agoMust be a pretty cool toy; it constantly 10X’s my productivity.
- nopromisessir 2y agoYou said it mate. I feel bad for folks who turn away from this technology. If they persist... They will be so confused why they get repeatedly lapped. I wrote a working machine vision project in 2 days with these toys. Key word: working... Not hallucinated. Actually working. Very useful.
- SubiculumCode 2y agoMy daughter berated me for using AI (the sentiment among youth is pretty negative, and it is easy to understand why), but I simply responded, "if I don't my peers still will, then we'll be living on the street." And it's true, I've 10x'd my real productivity as a scientist (for example, using llms to help me code one off scripts for data munging, automating our new preprocessing pipelines, etc, quickly generating bullet points for slides). The trick though is learning how to prompt, and developing the sense that the LLM is stuck with the current prompt and needs another perspective. Funnily enough, the least amount of luck I've had is getting the LLM to write precisely enough for science (yay I still have a job), even without the confabulation, the nuance is lacking...that it's almost always faster for me to write it myself.
- Aerbil313 2y ago> My daughter berated me for using AI (the sentiment among youth is pretty negative, and it is easy to understand why) I can’t relate. Currently in university. Everyone is thankful to God ChatGPT exists. I’d think it must be a joke, or your daughter somehow managed to live in a social circle which doesn’t yet adopted chatbots for school purposes.
- ammar_x 2y agoThe article compares GPT-4o to Sonnet from Anthropic. I'm wondering how Opus would perform at this test?
- DeathArrow 2y agoI wonder how llama3 is doing.
- whereismyacc 2y agoI always thought it seemed likely that most needle in a haystack tests might run into the issue of the model just encoding some idea of 'out of place-ness' or 'significance' and querying on that, rather than actually saying something meaningful about generalized retrieval capabilities. Does that seem right? Is that the motivation for this test?
- pojzon 2y agoMeh still for a lot of stuff it simply lies. Just today it lied to me about VRL language syntax, tryin to sell me some python stuff in there. Senior ppl will often be able call out the bullshit, but I believe for junior ppl it will be very detrimental. Nether the less amazing tool for d2d work if you can call out BS replies.