egorr

6 ms·

1.
by egorr 8mo ago
personal site, some photos and stuff i worked on: https://intest.dev/
2.
by egorr 2y ago
hey, supabase engineer here; we didn’t check that out with files that small, but thanks for the idea, i will try it out the only thing i can say related to the topic is that s3 multipart outperforms other methods for files larger than 50mb
3.
by egorr 3y ago
interesting quote: "A vital component of Foundations is its robust and ergonomic API for seccomp, a Linux kernel feature for syscall sandboxing. This feature enables the setting up of hooks for syscalls used by an application, allowing
4.
Foundations – modular Rust library, designed for prod-grade distributed systems(blog.cloudflare.com)
31 points by egorr 3y ago|1 comments
5.
by egorr 3y ago
hey hn, supabase engineer and blogpost coauthor here. we made our first experiments with HNSW index for vector search and noticed about 5 times better performance compared to IVF with high dimensional vectors, such as embedding-ada by OpenA
6.
by egorr 3y ago
even networking alone adds about 1ms or so, and these are 2 hops each side: lb - sv - db, instead of the same instance for pgb scenario another 1ms goes for a gain of a millions connections, multitenancy, and a lot of features that are poss
7.
by egorr 3y ago
supabase engineer here would be happy to answer questions about load testing setup here is the github repo[0] with an app that helps me launch terraforms and k6 and keep track of launches/results [0] https://github.com/s
8.
by egorr 3y ago
should be yes, but even in examples from openai, they usually do splitting into chunks For example in chatgpt-retrieval-plugin[0] repo default chunk size is just 200 tokens this is anyway a limitation, no doubt, but chunking is pretty often
9.
by egorr 3y ago
heya, supabase engineer here, i coauthored the part about benchmarking different models with pgvector to compare QPS when using fewer dims models. btw, you can find the dataset with embeddings generated by all 3 mentioned: text-embedding-ad
10.
by egorr 3y ago
it means the precision at K(10) or precision for first 10 documents retrieved. In other words - number of relevant documents amongst retrieved 10(k) of them So 0.98 precision would mean that 9.8 out of 10 documents are relevant on average
11.
by egorr 3y ago
i am not Paul of course, but i just want to highlight that the idea behind blogpost was to show that if you set some settings for postgres correctly, you can achieve much better performance ie you can notice that most of results are on the
12.
by egorr 3y ago
supabase engineer here, would be happy to share details regarding our research and outcomes if you have any questions
13.
by egorr 3y ago
is it the right convo branch? as this is a chatgpt plugin and uses openai embeddings api, it for certain sends some data to their servers. both when embeddings are generated and when chatgpt sends request to your plugin and receives respon
14.
by egorr 3y ago
Hey, co-author here, thanks for caring about me. This is me who added myself as an author to the blog, i feel like “engineering” is the best description for the work I do in Supabase :) You can find it in PR https://github.com&#x
15.
by egorr 3y ago
heya, yep, if you add ivfflat index for embedding <column> in your table. I have to note that I am not pgvector contributor and this is how i understood it from the repo and its code. you can find a bit more context from the paper ht
16.
by egorr 3y ago
Let me know if you want to play around with it and plug it in your ChatGPT, we can expose it if there is a need
17.
by egorr 3y ago
hey hn, supabase engineer here I just added a PR to the great retrieval plugin template repo from OpenAI to add Datastore options for Supabase and Postgres. This is an implementation of Datastore interface using Postgres database and 2 diff
18.
by egorr 3y ago
Hey, from what we’ve seen up to 50k per node should be ok for supavisor on 4 cores as well. And TPS is here mainly to demonstrate that we not only create dummy connections but that these connections are really used. And 20k TPS is just the
19.
by egorr 3y ago
Hi, i was running these million connections benchmarks, happy to answer if you have any questions regarding this:) In short, that was pretty easy to do, only things we had to tweak to make it work were some elixir and linux limits
20.
by egorr 4y ago
I like C4 for system design overview and sequence diagrams to show detailed process of interactions in one particular scenario. I found these two the most useful for me. I also like to write diagrams as code with https://plantuml