shahules

9 ms·

1.
Cloning Bench: Evaluating AI Agents on Visual Website Cloning(github.com)
2 points by shahules 6mo ago|1 comments
2.
by shahules 6mo ago
My team works on automatic environment generation for RL post-training. One of our projects is using coding agents to build web clones for BUAs/CUAs. We tested Gemini, Claude Code, GLM, and Codex using our harness on their abilities to
3.
by shahules 7mo ago
Nice, their training recipe seems unique.
4.
by shahules 7mo ago
After doing few experiments, I think that having Agents work on browser for all tasks wouldn't be best due to many factors like token cost, safety, etc. But browser/computer can be a tool that the agent can be alongside MCPs to co
5.
by shahules 7mo ago
There are few agents like browser-use, skyvern etc that may provide this capability.
6.
PA bench: Evaluating web agents on real world personal assistant workflows(vibrantlabs.com)
38 points by shahules 7mo ago|9 comments
7.
PA Bench: Evaluating Frontier Models on Multi-Tab Pa Tasks(vibrantlabs.com)
7 points by shahules 7mo ago|1 comments
8.
by shahules 7mo ago
Most current web agent benchmarks focus on single-tab tasks (e.g., 'go to Gmail and star this email'). We found that frontier models that score highly on those tasks (like in WebArena) often fall apart when they have to coordinate
9.
by shahules 2y ago
It's an interesting article and I agree with some points you brought up here. But here are some of them to which I don't agree to 1. Evals are used throughout the article in the sense of LLM benchmarking, but this is not the point
10.
by shahules 2y ago
Can't agree with you more my friend. Another point on a philosophical level is efficiency or optimization in life, which always focuses on tangible aspects and ignores the greater intangible aspects of life.
11.
by shahules 2y ago
Deepeval also uses Ragas underneath. They initially took a different approach by allowing uses to formulate test cases but we were focusing on RAGs only and creating metrics and features like synthetic test data generation for it. Now that
12.
by shahules 2y ago
Hey, I would recommend checking out our PRs. There would be PRs that have modified some of the prompts to better suit Mixtral.
13.
by shahules 2y ago
I think it's true for any early-stage library/framework. The tradeoff is then you will have to keep maintaining it, add support to other LLMs if you change LLMs, etc. Then in the end OSS will be far ahead because by that time it w
14.
by shahules 2y ago
Hey, DeepEval is interesting. What do you mean by "evaluating any LLMs"?
15.
by shahules 2y ago
1) Horizondal expansion and support are core to every framework/library. This won't make us a competitor to LC, we actually use langchain-core to support many of these like supporting different LLMs. 2) We operate in a layer under
16.
by shahules 2y ago
Yes, this is an interesting challenge we are also excited about.
17.
by shahules 2y ago
Hey, this is a highly requested feature. We will be implementing it soon. Something like a rough estimate is what we are planning to do.
18.
by shahules 2y ago
Thanks for your feedback. We have tested Ragas on alternatives like Claude, Mixtral, Gemini, etc. Although we support all LLMs supported by Langchain, sadly many of the OSS models out of the box aren't capable of generating JSON-type o
19.
by shahules 2y ago
Thank you.
20.
by shahules 2y ago
Thank you for asking this question. To answer this question, I will explain two directions of Ragas. The first one is the horizontal expansion of the library which involves features like - Giving you the ability to use any LLMs instantly wi
21.
by shahules 2y ago
Hey, thanks for trying out Ragas. As an open-source library, we are continuously improving from the feedback from the community which I see as our primary strength. I am sure that Ragas is not perfect yet, but I can assure you that it is 10
22.
Show HN: Ragas – Open-source library for evaluating RAG pipelines(github.com)
121 points by shahules 2y ago|26 comments
23.
by shahules 2y ago
Hey, The cost depends on how many samples you would want to generate. We are finetuning smaller OSS models to tackle the cost factor of synthetic data generation which we will open source this month.
24.
by shahules 2y ago
Thank you. Yes. We support almost all opensource models. You can customise it easily by referring to https://docs.ragas.io/en/stable/howtos/customisations/bring-...
25.
by shahules 2y ago
Yes,ofc. We support all LLMs supported by Langchain. https://docs.ragas.io/en/stable/howtos/customisations/bring-...
26.
Show HN: Ragas – Open-source library for evals and testing RAG systems(github.com)
15 points by shahules 2y ago|9 comments
27.
by shahules 3y ago
Thank you. That's great. We are working on forming paradigms to evaluate Agents, I'll get in touch with you.
28.
by shahules 3y ago
Evals is not suitable for evaluating LLM applications such as RAG, etc because one has to evaluate on their own data where no golden test data exists, and techniqus used have poor correlation with human judgement. We have build RAGAS frame
29.
by shahules 3y ago
This guy doesn't have any idea what he is talking about. He consistently posts such bullshit on twitter. Mostly copy paste with added spice mix.
30.
by shahules 3y ago
Great, excellent read. This article I wrote some of the open-source finest foundation LLMs https://explodinggradients.com/the-rise-of-open-source-large...

More ›