retakeming

5 ms·

1.
by retakeming 1y ago
What would you say is the most neutral oil then?
2.
by retakeming 1y ago
The segments themselves being immutable doesn't mean that Tantivy is incompatible with Postgres - it just means that Tantivy needs to be made compatible with Postgres' concurrency control mechanisms (MVCC) and storage format (bloc
3.
by retakeming 1y ago
I'm one of the pg_search maintainers. Hello! A few thoughts. First, both strategies - the one outlined by the Neon/ParadeDB article, and the one used here -- are presented as viable alternatives by the Postgres docs: https:/
4.
by retakeming 2y ago
Good question. That was from a very old version of pg_bm25 (since renamed to pg_search). BM25 indexes are now strongly consistent.
5.
by retakeming 2y ago
If you're a Postgres pg_search user, partial indexes ( https://docs.paradedb.com/search/full-text/index#partial-bm2... ) can solve.
6.
by retakeming 2y ago
pg_search (full text search Postgres extension) can be used with pgvector for hybrid search over Postgres tables. It comes with a helpful hybrid search function that uses relative score fusion. Whereas rank fusion considers just the order o
7.
by retakeming 2y ago
Could you clarify which result you're referring to as the baseline and "number 1 place?" I should clarify that our published Clickbench results are from our pg_analytics extension. New results with pg_lakehouse will be releas
8.
by retakeming 2y ago
pg_house just wasn't as catchy! In all seriousness though, I see your point. While it's true that we don't provide the storage or table format, our belief is that companies actually want to own the data in their S3. We called
9.
by retakeming 2y ago
Thanks! 1. It's single node, but DataFusion parallelizes query execution across multiple cores. We do have plans for a distributed architecture, but we've found that you can get ~very~ far just by scaling up a single Postgres node
10.
by retakeming 2y ago
Whereas pg_analytics stores the data in Postgres block storage, pg_lakehouse does not use Postgres storage at all. This makes it a much simpler (and in our opinion, more elegant) extension. We learned that many of our users already stored t
11.
by retakeming 2y ago
This is anecdotal, but I feel that we (ParadeDB) have received more requests for Iceberg integration vs. Delta Lake. We were actually hesitant to launch pg_lakehouse without Iceberg support, but pulled the trigger on it because the iceberg-
12.
Pg_lakehouse: Query Any Data Lake from Postgres(github.com)
5 points by retakeming 2y ago|0 comments
13.
by retakeming 3y ago
Datafusion and Deltalake abstract away most of Arrow/Parquet. And those APIs were very nice to work with.
14.
by retakeming 3y ago
Datafusion's SQL dialect has some slight quirks that pertain to Datafusion-specific functionality. For instance, the ability to create an external table. With regards to ParadeDB, we rely on the Datafusion SQL parser, which can transfo
15.
by retakeming 3y ago
Yes and no, depending on what you mean. The custom types/indexes introduced by PostGIS won't work with deltalake tables. Even if it were possible, the benefits of using deltalake tables to execute geospatical queries are unclear,
16.
by retakeming 3y ago
Sort ordering is handled automatically by Datafusion. Collations at the column/operation level are not yet supported but we're working on it.
17.
by retakeming 3y ago
Yes. delta-rs is a Rust-based implementation of Delta Lake. The existing version of pg_analytics uses delta-rs to manage Parquet files stored within Postgres. In the future, we plan on integrating external object stores. This means that you
18.
by retakeming 3y ago
One of the authors of pg_analytics here. 1. On Clickbench, make sure you're doing an apples-to-apples comparison by comparing scores from the same instance. We used the most commonly-used c6a.4xlarge instance. While a few databases lik
19.
Pg_analytics: Column-Oriented Tables in Postgres(blog.paradedb.com)
5 points by retakeming 3y ago|0 comments
20.
by retakeming 3y ago
On Clickbench, ParadeDB load times are 1.85x faster than Postgres. Typically, you would expect slower inserts if you were using an index to accelerate queries since inserting into the index takes time. But pg_analytics is not an index. It&
21.
by retakeming 3y ago
Thanks! We released support for metrics aggregations a few days ago, including count: https://docs.paradedb.com/aggregations/metrics#count . We haven't gotten around to benchmarking aggregations - that's the f
22.
by retakeming 3y ago
Blog post author and one of the pg_bm25 contributors here. Super excited to see the interest in pg_bm25! pg_bm25 is our first step in building an Elasticsearch alternative on Postgres. We built it as a result of working on hybrid search in
23.
by retakeming 3y ago
Hey HN - lead dev on pg_bm25 here. To provide a bit more context: the goal of pg_bm25 is to make BM25-based full text search feel native to Postgres. pg_bm25 is built on Tantivy, a Rust-based alternative to Apache Lucene. We've integra
24.
by retakeming 3y ago
This looks neat. https://github.com/ShishirPatil/gorilla feels like a project with certain similarities here. How would you compare yourselves to them?
25.
by retakeming 3y ago
Thanks, appreciate it! We forked pgsync for the silly reason that they hadn't published to PyPi in months, and some of their dependencies were out of date. We haven't made any modifications to pgsync so maintaining compatibility s
26.
by retakeming 3y ago
Thanks! We debated what the right decision was in the beginning but are glad to have settled on Apache.
27.
by retakeming 3y ago
You don't need to define all keys in a JSON object - by default, new keys will automatically be added to the index mapping when a JSON document containing that key is added to the index. Details on how to query JSON objects can be foun
28.
by retakeming 3y ago
1. Correct - we don't rely on pgvector. As a result, we're compatible with more existing managed Postgres services. 2. Probably the biggest differentiator between Vespa and Retake is the core architecture - Retake is built on top
29.
by retakeming 3y ago
We don't run any batch jobs - Retake streams changes in real time via CDC (change data capture). The only batch job you would need to run is to populate an index when it's first created.