4 ms·

It's good enough and only going to get better. The postgres community only has to improve a vector index extension. These vector databases have to improve every

It's good enough and only going to get better. The postgres community only has to improve a vector index extension. These vector databases have to improve everything else (all of the really hard stuff) about a database. My money is on postgres. EDIT>> Plus pgvector is enabled by default on AWS RDS databases. Huge bonus.

dmezzetti 3y ago
There are other approaches that could work too. For example, txtai can use Postgres as a data store and combine that with a vector index like Faiss or Hnswlib. In this case, you get the robustness of Postgres (which has been around a while) and Faiss (which is one of the most mature vector indexes). https://neuml.hashnode.dev/external-database-integration https://neuml.hashnode.dev/external-database-integration
threeseed 3y ago
> These vector databases have to improve everything else No they don't. Because they aren't trying to build general purpose databases. And often trying to jam everything into one product just makes it worse not better.
- richardw 3y ago
  User management, documentation, distribution, UI’s, upgrades, high availability and clustering, backups, libraries, skills required to administer and therefore hire for. I’ve worked in companies where we grabbed any tech we want, and I’ve worked in companies with one default DB unless you had an excellent reason. The latter meant a much simpler universe where entire teams knew how to fix things that broke at 2am. You just hire one type of DBA. It also meant dev teams really knew how to squeeze the one DB platform rather than learn a little about 10 database types.
inkyoto 3y ago
> It's good enough […] «Good enough» is not a quantative metric by any conceivable measure. Furthermore, people in surrounding threads have reported inadequate performance of the pg_vector PgSQL extension (which, indubitably, can and will be fixed at some point). An example of a useful quantative metric would: «compared to alternative vector processing/indexing engines/databases A, B and C and a dataset comprising N inputs and a total size of S Mb/Gb/Tb and using the M, N and P use cases in the performance testing scenario and pg_vector being the performance baseline, the following results have been obtained: X, Y and Z» – do you have anything similar to that you could share? Not only the comparison results would be insightful, they could also feed into someone's decision making process of whether X should be favoured over Y. > […] and only going to get better. Beyond doubt, it will. In the future. But people need to solve problems in the present.

dmezzetti 3y ago

There are other approaches that could work too. For example, txtai can use Postgres as a data store and combine that with a vector index like Faiss or Hnswlib. In this case, you get the robustness of Postgres (which has been around a while) and Faiss (which is one of the most mature vector indexes). https://neuml.hashnode.dev/external-database-integration https://neuml.hashnode.dev/external-database-integration

threeseed 3y ago

> These vector databases have to improve everything else No they don't. Because they aren't trying to build general purpose databases. And often trying to jam everything into one product just makes it worse not better.

richardw 3y ago
User management, documentation, distribution, UI’s, upgrades, high availability and clustering, backups, libraries, skills required to administer and therefore hire for. I’ve worked in companies where we grabbed any tech we want, and I’ve worked in companies with one default DB unless you had an excellent reason. The latter meant a much simpler universe where entire teams knew how to fix things that broke at 2am. You just hire one type of DBA. It also meant dev teams really knew how to squeeze the one DB platform rather than learn a little about 10 database types.

inkyoto 3y ago

> It's good enough […] «Good enough» is not a quantative metric by any conceivable measure. Furthermore, people in surrounding threads have reported inadequate performance of the pg_vector PgSQL extension (which, indubitably, can and will be fixed at some point). An example of a useful quantative metric would: «compared to alternative vector processing/indexing engines/databases A, B and C and a dataset comprising N inputs and a total size of S Mb/Gb/Tb and using the M, N and P use cases in the performance testing scenario and pg_vector being the performance baseline, the following results have been obtained: X, Y and Z» – do you have anything similar to that you could share? Not only the comparison results would be insightful, they could also feed into someone's decision making process of whether X should be favoured over Y. > […] and only going to get better. Beyond doubt, it will. In the future. But people need to solve problems in the present.