8 ms·
If it's a choice between performance and being able to "safely" run sketchy extensions, I'd rather have performance.
by fpgaminer 2mo ago
If it's a choice between performance and being able to "safely" run sketchy extensions, I'd rather have performance.
- jeltz 2mo agoThreads does not offer any major performance advantage, performance of processes vs threads is virtually the same. The reason the PostgreSQL project is moving towards threads is to make development easier.
- codesnik 2mo agounless you're spawning them for new connections.
- jeltz 2mo agoSome, but not that much. Switching PostgreSQL to a threaded model will not magically make spawning connections fast. PostgreSQL connections are quite heavyweight. The reason to use threads is almost entirely about ease of development, not about performance. If you use shrared memory like PostgreSQL does you need to write your own allocators, etc. So much you get for free if you use threads.
- malisper 2mo ago> Threads does not offer any major performance advantage This is very not true. When it comes to parallel queries, a process model adds a ton of overhead. You can't pass pointers between processes because the address space is different. This adds a ton of overhead in a bunch of different places. For example when doing a parallel hash join, Postgres will have each worker build a local hash table. Then it will take all the tuples out of the local hash table and copy them through shared memory to the leader who will then construct a new hash table. This duplicates a lot of work as you have to hash the tuples multiple times. A lot of getting to Clickhouse level performance was making better use of parallelism.
- jeltz 2mo agoOk ... you know PostgreSQL supports hash tables in shared memory, right? PostgreSQL could in theory share those if we wanted to. The issue is just that coding anything which uses shared memory is a lot of work. Additionally the reasons PostgreSQL does not offer Clickhouse performance has very little to do with parallelism. PostgreSQL plans to move to threading but the efforts around imporving OLAP performance are almost entirely unrelated.
- dupontcyborg 2mo agoi may be missing context, but shared memory across processes, without ipc?
- Giefo6ah 2mo agoThere's nothing special about threads vs processes in Linux. mmap works the same, the challenge is to map the same file. You can share a path, pass a file descriptor via fork or unix domain socket, among other techniques.
- LoganDark 2mo agoThat induces disk I/O overhead (even if it somehow doesn't impact IPC performance)
- drdexebtjl 2mo agoThe file doesn’t have to be disk-backed.
- SigmundA 2mo agoMSSQL can handle 32k open connections no need to run a pooler in front of it, can PG do 32k connections and a process for each? MSSQL shares cached query plans between connections including jitted code, PG cannot do that and the changes needed to make the plans cross process portable would be extensive while sharing between threads is just normal code sharing between threads.
- brianaker 2mo agoA mixture of threads and processes that can be used to match processors, disk I/O, and network interfaces. A very long time ago, there was once a feature called "Data Blades" which tanked a commercial database vendor. A badly behaving blade could bring down the entire database. Most anyone who has been working on databases for a few decades remembers this and makes a point of either not introducing these sorts of features or making use of processes over threads. I have not looked at the code referenced in the mentioned project, but thus far I haven't seen a model that could craft a complete SQL parser on its own. There are a number of problems, and design decisions, that a developer decides on when writing a database that I don't see any current models… just because you have the ingredients does not mean that the stew is edible.
- mulmen 2mo ago> A very long time ago, there was once a feature called "Data Blades" which tanked a commercial database vendor. I have no idea what this is and a web search turned up Harbor Freight woodworking tools.
- Izkata 2mo agoI think it's this: https://www.ibm.com/docs/en/informix-servers/15.0.x?topic=concepts-datablade-modules-database-server https://www.ibm.com/docs/en/informix-servers/15.0.x?topic=co... And what you found seems to be "Dado", not "Data".
- mulmen 2mo agoYes, Kagi was sufficiently confused as to what "Data blades" are that it actually thought I was looking for replacement woodworking blades. "DataBlade" finds the IBM Informix product.
- Verdex 2mo agoMy first web search (data blades) turned up harbor freight woodworking tools. Then I made a second web search: data blades database. That turned up some ibm database software module technology which I assume is what's being discussed.
- tremon 2mo agoDo you mean that you don't run sketchy extensions and therefore this doesn't affect you, or that you're ok with data loss due to extension failures?
- roughly 2mo agoIn the age of vibe-generated code, I promise you're gonna want the safety.
- CuriouslyC 2mo agoSo load them up in read replicas
- noworriesnate 2mo agoOr use webassembly to sandbox them
- deleted 2mo ago[deleted]
- echelon 2mo agoYou have to have a write path. You're gonna want what to be non-sketchy and performant.
- wtetzner 2mo agoWhat about just not installing sketchy extensions?
- throwitaway222 2mo agoI know right? Postgres is not firefox, we don't operate with 25 extensions on all the time. At most we have 1 and normally we have 0.
- hoppp 2mo agoWhat about extensions that are not sketchy? Lots of good ones out there.
- throwaway27448 2mo agoPresumably they'd be fine running in a threaded context.
- hoppp 2mo agoExtensions like pgvector, TimescaleDB would probably need to be ported tho, not sure how much but there are footguns.
- SoftTalker 2mo agoNot needed in many (most?) cases.
- thayne 2mo agoAn extension written for a single threaded host system might not work in a multi-threaded context. For example if has global or shared state that isn't protected with locks or similar (which is unfortunately fairly common in c code)
- throwitaway222 2mo agoPeople are assuming the extensions can't also be rewritten to be good.
- ori_b 2mo agoBut it isn't.
- socalgal2 2mo agoYou could fix probably the sketchy extension issue with WASM.
- worthless-trash 2mo agoLimit, not fix.
- rtpg 2mo agoWhile PG's behavior doesn't guarantee a lack of data corruption, "an extension crashed, all bets are off, tear everything down" is going to give you a much better fighting chance against data corruption vs the alternative.
- throwitaway222 2mo agoEspecially since all those sketchy extensions can be rewritten in rust over a weekend and have their bugs fixed as well.
- xikxp1 2mo agoThis is just not true. PGRX can't physically support expanded shared memory due to Rust being very strict with custom allocators. A lot of shady extensions rely on dynamically allocated memory and C++ is like the only possible choice there.