7 ms·

> For loads with over 20k rows/s and people pushing changes, you may need a full-time person to handle the cluster and take a look at the crazy queries people a

> For loads with over 20k rows/s and people pushing changes, you may need a full-time person to handle the cluster and take a look at the crazy queries people are going to write. I think this was a benefit of DBA culture in previous eras. Not that the DBAs were specifically necessary to write good queries (often they'd need to work with application teams to guide them towards schemas/behavior that worked well) or to maintain the database (managed DB offerings obsolete a lot of this work), but because they functioned as gatekeepers and rate-limiters of what queries and schemas could exist. In that mode, DBAs functioned a bit like a human/process version of a thin microservice wrapping database access functionality. A big benefit was that the rate of change of queries/schema changes/access patterns was controlled and had a higher probability of being reviewed and thought about by humans before it went live. This also resulted in an increased end-database-user culture of trying to make existing schemas/query patterns work before jumping straight to bespoke access patterns. That culture's not what you want as e.g. a startup or pro-rapid-big-refactors shop, but it is what you want when your DB reliability needs or query rate/dataset size are high. I don't think it's a given that a gatekeeper team is worth the overhead and cost; that's situational. I do think that the code version of that team (aforementioned microservice that wraps DB accesses/schema changes and nothing else) is usually not worth the cost. In my experience, that pretty much always reduces reliability and free performance gains that come from using direct DB clients from user code.

bushbaba 7d ago
Most startups can just scale your traditional separation of compute & storage here though. You’d be shocked how well duckdb against s3 scales for 99.9% of use cases
- zbentley 7d ago
  It's one thing to be able to scale database compute/storage; it's another thing to be able to partition it. It's extremely common for bad queries/access patterns to cause noisy-neighbor effects on other simultaneous accesses to the database, to the extreme of knocking the whole database over with timeouts/OOMs/etc. Scaling out DB compute can only help with that to a (expensive) point; eventually, you end up wanting to either prevent the bad queries from being added to the system (DBA culture) or ensure that the bad query runs on database infrastructure that doesn't affect other queries. That's why partitioning DB compute (and storage: noisy-neighbor effects from a bad query at the storage layer don't require storage to be running e.g. a BookKeeper or whatever on a server; they can manifest as hot S3 keys or cloud object/block store rate limiting) is a necessary capability if your plan for dealing with a culture of "anyone can add any access pattern they want" is to scale the DB.
  - jeremyjh 7d ago
    Iceberg & Delta Lake do partition the data. You can add as many DuckDB servers processing them as you want, each app can have as many as you need. You pay Amazon for all the I/O and let them worry about it scaling it. I'm not saying there is no point of contention - depending upon your write patterns you have practical limits, but for reading data that is already stored you pretty much just pay more to do more and the rate of that cost is close to linear. I'm sure there is a scale at which that pattern breaks, and I'm sure 99.9% of people reading this will never operate a system at that scale.
    - Melatonic 5d ago
      Sounds like a great way to give AWS infinite money
  - bushbaba 7d ago
    Iceberg addresses the hot s3 key prefixes. It places entropy in the prefix path per object so reads/writes are randomized over the bucket key prefix space. It’s much less of an issue. Also, 99%+ of Aws customers likely don’t have enough QPS to hit the per prefix limits anyway
- bigfatkitten 7d ago
  Most software developers now are absolutely ignorant of performance concerns. Just throw more compute at it until it works, and someone else will pay the AWS bill.
  - 0xbadcafebee 6d ago
    And later complain AWS was too expensive after they kept throwing hardware at it rather than writing an efficient query
    - bigfatkitten 5d ago
      And then hire a FinOps team to try and restrain the development team’s profligate spending.
tmpz22 7d ago
Instead we're... listen to this... we're going to take a software developer right. Just a normal developer right. We're going to make them be the database expert right. And the cloud expert. And we're going to put them on call. We're going to have them debug linux logs, and optimize our AWS costs. They'll be there for client escalation work. And big sales calls. From time to time we'll even have them do front end work. And get this. We pay them the exact same.
- chasd00 7d ago
  yes, that was the cloud and "devops" promise. ..or what it just another sham?
- throwaway894345 7d ago
  A decade ago we would hire them fresh from some Ruby on Rails bootcamp so we could pay them less :shrug:
- zbentley 7d ago
  I think a DBA/ops/infrastructure person as an imposed bottleneck is a useful capability in some environments. But I won't follow you as far as "expecting developers to have expertise in how and where their software runs is unreasonable". Like, yeah, it sucks that added DevOps responsibilities etc. don't come with adjusted compensation/time allocation expectations. I'm with you there. But it's simultaneously true that a ton of "just regular developer" people are significant liabilities because they don't understand anything about the environment where their software runs. That liability manifests operationally (if someone's just running integration tests on Windows for their Java business logic changes and don't have any familiarity with e.g. the Linux, container, or cloud environments where their code runs, they're going to be useless when their code breaks in production and operations staff needs context), and it also makes them less effective when writing code--this culture of "developers should just live in business logic and not have to context-switch or fill their brains with other levels of the stack" is what leads to full table scans, lack of awareness of memory use, N+1 query hell, looping microservice dependencies, misunderstanding of what HTTP fields are set on requests that are mutated by load balancers, mistaken assumptions about how many instances of code can run and what concurrency/thread/coroutine behaviors are present, and so on. Those are very common problems, and it's incumbent on developers in every specialty to gain familiarity with how and where their code runs in order to write and maintain that code effectively. If your code runs on Linux in Kubernetes, all of your developers should know how to read Linux system logs, check database sessions/queries issued by parts of the application, ls/grep/cat/strace/ps their way around, interpret k8s/application dashboards, check application logs both in log storage and as they're emitted from a process, exec into a container, restart pods, check deployment liveness, etc. Even if they don't have permission to do those things in production. That was true in 2005 when they deployed their code to IIS on Windows Server/MSSQL, too--just with different operational specifics. That's a low bar that's often unmet, and all sorts of teams suffer from that failure. Those skills can be trained, kept up to date, and hired for; I don't think there's a great excuse for not expecting them.
pstuart 6d ago
I'm going to go out on a limb here and say that having a tuned LLM would probably be able to eliminate the need for a full time DBA query analyst. Disclaimer: I've never managed a ClickHouse cluster, let alone one of this size.
winrid 6d ago
hello we still exist :)

bushbaba 7d ago

Most startups can just scale your traditional separation of compute & storage here though. You’d be shocked how well duckdb against s3 scales for 99.9% of use cases

zbentley 7d ago
It's one thing to be able to scale database compute/storage; it's another thing to be able to partition it. It's extremely common for bad queries/access patterns to cause noisy-neighbor effects on other simultaneous accesses to the database, to the extreme of knocking the whole database over with timeouts/OOMs/etc. Scaling out DB compute can only help with that to a (expensive) point; eventually, you end up wanting to either prevent the bad queries from being added to the system (DBA culture) or ensure that the bad query runs on database infrastructure that doesn't affect other queries. That's why partitioning DB compute (and storage: noisy-neighbor effects from a bad query at the storage layer don't require storage to be running e.g. a BookKeeper or whatever on a server; they can manifest as hot S3 keys or cloud object/block store rate limiting) is a necessary capability if your plan for dealing with a culture of "anyone can add any access pattern they want" is to scale the DB.
- jeremyjh 7d ago
  Iceberg & Delta Lake do partition the data. You can add as many DuckDB servers processing them as you want, each app can have as many as you need. You pay Amazon for all the I/O and let them worry about it scaling it. I'm not saying there is no point of contention - depending upon your write patterns you have practical limits, but for reading data that is already stored you pretty much just pay more to do more and the rate of that cost is close to linear. I'm sure there is a scale at which that pattern breaks, and I'm sure 99.9% of people reading this will never operate a system at that scale.
  - Melatonic 5d ago
    Sounds like a great way to give AWS infinite money
- bushbaba 7d ago
  Iceberg addresses the hot s3 key prefixes. It places entropy in the prefix path per object so reads/writes are randomized over the bucket key prefix space. It’s much less of an issue. Also, 99%+ of Aws customers likely don’t have enough QPS to hit the per prefix limits anyway
bigfatkitten 7d ago
Most software developers now are absolutely ignorant of performance concerns. Just throw more compute at it until it works, and someone else will pay the AWS bill.
- 0xbadcafebee 6d ago
  And later complain AWS was too expensive after they kept throwing hardware at it rather than writing an efficient query
  - bigfatkitten 5d ago
    And then hire a FinOps team to try and restrain the development team’s profligate spending.

tmpz22 7d ago

Instead we're... listen to this... we're going to take a software developer right. Just a normal developer right. We're going to make them be the database expert right. And the cloud expert. And we're going to put them on call. We're going to have them debug linux logs, and optimize our AWS costs. They'll be there for client escalation work. And big sales calls. From time to time we'll even have them do front end work. And get this. We pay them the exact same.

chasd00 7d ago
yes, that was the cloud and "devops" promise. ..or what it just another sham?
throwaway894345 7d ago
A decade ago we would hire them fresh from some Ruby on Rails bootcamp so we could pay them less :shrug:
zbentley 7d ago
I think a DBA/ops/infrastructure person as an imposed bottleneck is a useful capability in some environments. But I won't follow you as far as "expecting developers to have expertise in how and where their software runs is unreasonable". Like, yeah, it sucks that added DevOps responsibilities etc. don't come with adjusted compensation/time allocation expectations. I'm with you there. But it's simultaneously true that a ton of "just regular developer" people are significant liabilities because they don't understand anything about the environment where their software runs. That liability manifests operationally (if someone's just running integration tests on Windows for their Java business logic changes and don't have any familiarity with e.g. the Linux, container, or cloud environments where their code runs, they're going to be useless when their code breaks in production and operations staff needs context), and it also makes them less effective when writing code--this culture of "developers should just live in business logic and not have to context-switch or fill their brains with other levels of the stack" is what leads to full table scans, lack of awareness of memory use, N+1 query hell, looping microservice dependencies, misunderstanding of what HTTP fields are set on requests that are mutated by load balancers, mistaken assumptions about how many instances of code can run and what concurrency/thread/coroutine behaviors are present, and so on. Those are very common problems, and it's incumbent on developers in every specialty to gain familiarity with how and where their code runs in order to write and maintain that code effectively. If your code runs on Linux in Kubernetes, all of your developers should know how to read Linux system logs, check database sessions/queries issued by parts of the application, ls/grep/cat/strace/ps their way around, interpret k8s/application dashboards, check application logs both in log storage and as they're emitted from a process, exec into a container, restart pods, check deployment liveness, etc. Even if they don't have permission to do those things in production. That was true in 2005 when they deployed their code to IIS on Windows Server/MSSQL, too--just with different operational specifics. That's a low bar that's often unmet, and all sorts of teams suffer from that failure. Those skills can be trained, kept up to date, and hired for; I don't think there's a great excuse for not expecting them.

pstuart 6d ago

I'm going to go out on a limb here and say that having a tuned LLM would probably be able to eliminate the need for a full time DBA query analyst. Disclaimer: I've never managed a ClickHouse cluster, let alone one of this size.

winrid 6d ago

hello we still exist :)