jeadie

7 ms·

1.
by jeadie 5mo ago
This is exactly what we found. Ingest rates were tough. We partitioned and ran over multiple duckdb instances too (and wrangled the complexity). We ending up building a Sqlite + vortex file alternative for our use case: https://s
2.
by jeadie 5mo ago
You might find https://github.com/apache/datafusion and https://github.com/datafusion-contrib/datafusion-federation of interest
3.
by jeadie 1y ago
We’re building vector indexes into Datafusion for search (starting with S3 vectors). Open source at https://github.com/spiceai/spiceai
4.
by jeadie 1y ago
This is one of the ideas behind using DuckDB in github.com/spiceai/spiceai
5.
by jeadie 1y ago
There’s also https://github.com/spiceai/spiceai
6.
by jeadie 2y ago
This is a common feature now. If anything, for being so early to vector databases, Pinecone was rather late to integrating embeddings. Timescale most recently added it but, yes a bunch of others: Weaviate, Spice AI, Marqo, etc.
7.
by jeadie 2y ago
Why not just federate Postgres and parquet files? That way the query planner can push down as much of the query and reduce how much data has to move about?
8.
by jeadie 2y ago
This looks functionally similar as using http://github.com/spiceai/spiceai with a postgreSQL data accelerator.
9.
by jeadie 2y ago
Spice AI | Senior Software Engineer | GMT+10 (e.g. Australia) through GMT-7 (e.g. Seatle/SF/LA) | Remote | Full Time Spice AI provides building blocks for data and AI-driven applications by composing real-time and historical time-
10.
by jeadie 2y ago
And yes, Iceberg is very high up on our list
11.
by jeadie 2y ago
Yes! It can connect to FlightSQL compatible servers (see https://docs.spiceai.org/data-connectors/flightsql ) and its also a FlightSQL compatible server
12.
by jeadie 3y ago
Have you seen github.com/marqo-ai/marqo? It does all this wrapping, and you don't even need to pay for OpenAI or pinecone
13.
by jeadie 3y ago
I'm very glad that this has some added funding. I am building a serverless API on the cloudflare edge network using GGML as the backbone --> tryinfima.com
14.
by jeadie 3y ago
"AI Native" catching on
15.
by jeadie 3y ago
I've tried both Chroma and Qdrant. I don't think Chroma lacks that much. Definitely newer, but is also a great product. I think cloud support coming Q3 2023
16.
by jeadie 3y ago
(Not affiliated with hyperDB)
17.
by jeadie 3y ago
I've been using https://github.com/jdagdelen/hyperDB and it's been really easy to use. I think Clickhouse support is on the short-term roadmap.
18.
by jeadie 3y ago
Most people, like me, who end up needing to use vector DBs, are wanting to use LLMs on a specific, often private dataset/use case. Typically one starts with something like unstructured JSON data, then need to pick and manage LLMs to cr
19.
by jeadie 3y ago
Not a dumb question at all! Essentially what can do Marqo, and this blog shows, is that there is alot of logic and work to do what you said (i.e. pass raw data into LLM, get embeddings, store in vector DB, then query both embeddings and ori
20.
by jeadie 3y ago
Its a great tool. Unlike vectorDBs alone, Marqo helps the full process that alot of people end up wanting to use vectorDBs for (e.g. have structured data, use LLMs to create embeddings, and perform search/CRUD on embeddings + original
21.
by jeadie 3y ago
Being able to handle and ask questions of audio data is a pretty big field. https://www.assemblyai.com/ , for example, is a company entirely dedicated to audio intelligence. They have some great example use cases on their pa
22.
by jeadie 3y ago
This is generally very context/use case specific. In general, if a document is a `Dict[str, Any]`, then you either have to have one (or multiple) vector(s) per field, unless you want to combine vectors across fields (it's not self
23.
by jeadie 3y ago
I'm skeptical about some vector databases these days, but your article misses a few import points when it comes to LLMs. 1. To use LLMs effectively, you often need to generate and store more than 1 vector per document. 10 million vecto
24.
After All Is Said and Indexed – Unlocking Information in Recorded Speech(github.com)
57 points by jeadie 3y ago|13 comments
25.
by jeadie 3y ago
A really interesting blog post I found using LLMs for audio search which I think is a pretty nifty/new idea. I've found it cumbersome using some of the new vector DBs (chroma, faiss, etc) to make end to end systems, but with Marqo
26.
by jeadie 3y ago
I think so, but checkout a bunch of awesome resources, and make up your mind https://github.com/Jeadie/awesome-chatgpt-plugins
27.
by jeadie 3y ago
Maybe https://github.com/hora-search/hora but I've never used it
28.
by jeadie 3y ago
I forgot about https://github.com/qdrant/qdrant . It's a DB not a library so again may not be an exact answer for what you're looking for
29.
by jeadie 3y ago
Although there is some work going on right now to add support for the type of algorithms in pgvector to alot it to scale better (and also to have better recall/speed tradeoffs).
30.
by jeadie 3y ago
I know rust has beings to FAISS (see https://github.com/Enet4/faiss-rs ), I don't know if there's anything that would be considered comparable. Alot of work has gone into FAISS

More ›