aazo11

10 ms·

1.
The Accuracy of On-Device LLMs(medium.com)
2 points by aazo11 1y ago|2 comments
2.
by aazo11 1y ago
I tested on-device LMs (Gemma, DeepSeek) across prompt cleanup, PII redaction, math, and general knowledge on my M2 Max laptop using LM Studio + DSPy. Some observations - Gemma-3 is the best model for on-device inference - 1B models look fi
3.
AI's Version of Moore's Law(metr.org)
2 points by aazo11 1y ago|1 comments
4.
by aazo11 1y ago
The trend is that the length of tasks AI can do is doubling every 7 months. Accompanying YT video https://www.youtube.com/watch?v=evSFeqTZdqs
5.
by aazo11 1y ago
This is a huge unlock for on-device inference. The download time of larger models makes local inference unusable for non-technical users.
6.
by aazo11 1y ago
A better solution would train/finetune the smaller model from the responses of the larger model and only push to the inference to the edge if the smaller model is performant and the hardware specs can handle the workload?
7.
by aazo11 1y ago
Thanks for calling that out. It was 32GB. I updated the post as well.
8.
by aazo11 1y ago
Very interesting. I had not thought about gaming at all but that makes a lot of sense. I also agree the goal should not be to replace ChatGPT. I think ChatGPT is way overkill for a lot of the workloads it is handling. A good solution should
9.
by aazo11 1y ago
They look awesome. Will try it out.
10.
by aazo11 1y ago
Exactly. Why does this not exist yet?
11.
by aazo11 1y ago
By "too hard" I do not mean getting started with them to run inference on a prompt. Ollama especially makes that quite easy. But as an application developer, I feel these platforms are too hard to build around. The main issues bei
12.
Local LLM inference – impressive but too hard to work with(medium.com)
84 points by aazo11 1y ago|58 comments
13.
by aazo11 1y ago
I spent a couple of weeks trying out local inference solutions for a project. Wrote up my thoughts with some performance benchmarks in a blog. TLDR -- What these frameworks can do on off the shelf laptops is astounding. However, it is very
14.
by aazo11 2y ago
Great question! The purpose of github-assistant is to showcase the technologies that make it easy to build a tool/feature like this, not necessarily for it to be a stand-alone service. With dlt/Relta/LangGraph/assistant-
15.
by aazo11 2y ago
Yes in the future. We share the source code in both commercial and non-commercial engagements already. Drop me a line at amir [at] relta.dev if interested.
16.
by aazo11 2y ago
There will new data from the graphql API added over time. Would love your feedback on which data you like to see added https://docs.github.com/en/graphql
17.
by aazo11 2y ago
No this currently only answers questions from the GitHub graphql API.
18.
by aazo11 2y ago
We pull data from the GitHub API which includes data that that is not available from GitHub.com pages. Currently only PR, Issues, Commit and Star data is being loaded. You can also read more here https://medium.com/relta&#x2
19.
by aazo11 2y ago
Was able to reproduce and pushed an update. Thanks for calling this out.
20.
by aazo11 2y ago
Hi -- strange that didn't work. Overall, the semantic layer is designed to provide very tight guardrails and not hallucinate. You can see the agent suggest changes to the semantic layer if you give the produced answer a thumbs down. Th
21.
Show HN: GitHub-assistant – Natural language questions from your GitHub data(github.com)
49 points by aazo11 2y ago|16 comments
22.
by aazo11 2y ago
Right now the supported Vector stores are Chroma (which you can self-host), Pinecone and Astra. Adding a new vector store is quite easy: you just need to extend the VectorStore class ( https://github.com/Dataherald/datah
23.
by aazo11 2y ago
There are organizations using Dataherald in production right now. The latency is ~20-30s and it takes some set up, so as long as those are not blockers it can be used in prod.
24.
by aazo11 2y ago
Yes when you connect Dataherald to a DB it scans it and you can do exploratory queries.
25.
by aazo11 2y ago
ORMs generally map around entities and dimensions. Users generally ask about metrics and measures, which can be expressed in aggregations and group bys. How ould the NLP+ORM system do this?
26.
by aazo11 2y ago
While the engine response is not accurate all the time, the engine returns a confidence score. We have never encountered cases where a deployment with necessary training data indicates a .9 confidence score on an incorrectly generated SQL.
27.
by aazo11 2y ago
Tables, columns and views are scanned at configuration time (or based on an API trigger) and stored in the data store and a vector store, not on every run. They are then retrieved and injected based on relevance to the query.
28.
by aazo11 2y ago
Added the License
29.
by aazo11 2y ago
As I wrote on the original thread, we recommend using the RDBMS row-level security features. This blog discusses how to do that on Postgres https://www.2ndquadrant.com/en/blog/application-users-vs-row...
30.
by aazo11 2y ago
We recommend users leverage row-level security features built into modern RDBMS so the query results only return data for a given user. You can read more on how to do that on Postgres here https://www.2ndquadrant.com/en&#x2F

More ›