6 ms·

I don't have a lot of sympathy for people using their tools wrong. Using partitioning surely would have prevented this.

morkalork 1y ago
The replies on LinkedIn are are quick to point out everything they did wrong.
minimaxir 1y ago
Querying a partitioned table only works if you also filter on the partition key, so you can still footgun with a LIMIT if not aware of it.
kittikitti 1y ago
Wouldn't people who knew their tools perfectly well not even use a cloud service like BigQuery? At the level you expect them to use the tool, they could have created a big query engine themselves. Isn't the whole point of these tools is to make things easier?
- Rockslide 1y ago
  Sorry but that's nonsense. Partitioning is THE central cost controlling mechanism in BigQuery and the docs clearly state this. And it's an easy to use feature, so I'm not sure what makes you think using that would be as challenging as building your own query engine.
lmkg 1y ago
I have some sympathy for people who are blindsided by surprising difference between a new tool and their old one. This post is not eliciting sympathy. They're data consultants, who don't understand a very basic and fundamental aspect of the tool that they're using and recommending. If you're a consultant you have a responsibility to RTFM, and the docs are clear that LIMIT doesn't prune queries in BigQuery. And, also, the interface tells you explicitly how much data you're gonna query before you run it. This post is also blaming Google rather than accepting their own part in this folly, and event admits the docs are clear on this matter. Cost-control in BigQuery is not difficult. One of the tradeoffs of BQ's design is that it must be configred explicitly, there's no "smart" or "automatic" query pruning, but that also makes it easier to guarantee and predict cost over time as new data arrives.
- Rockslide 1y ago
  Yes the whole consultancy situation really is the icing on the cake - as the customer you pay for (alleged) experts in the field and get this as the result...

The replies on LinkedIn are are quick to point out everything they did wrong.

Querying a partitioned table only works if you also filter on the partition key, so you can still footgun with a LIMIT if not aware of it.

kittikitti 1y ago

Wouldn't people who knew their tools perfectly well not even use a cloud service like BigQuery? At the level you expect them to use the tool, they could have created a big query engine themselves. Isn't the whole point of these tools is to make things easier?

Rockslide 1y ago
Sorry but that's nonsense. Partitioning is THE central cost controlling mechanism in BigQuery and the docs clearly state this. And it's an easy to use feature, so I'm not sure what makes you think using that would be as challenging as building your own query engine.

lmkg 1y ago

I have some sympathy for people who are blindsided by surprising difference between a new tool and their old one. This post is not eliciting sympathy. They're data consultants, who don't understand a very basic and fundamental aspect of the tool that they're using and recommending. If you're a consultant you have a responsibility to RTFM, and the docs are clear that LIMIT doesn't prune queries in BigQuery. And, also, the interface tells you explicitly how much data you're gonna query before you run it. This post is also blaming Google rather than accepting their own part in this folly, and event admits the docs are clear on this matter. Cost-control in BigQuery is not difficult. One of the tradeoffs of BQ's design is that it must be configred explicitly, there's no "smart" or "automatic" query pruning, but that also makes it easier to guarantee and predict cost over time as new data arrives.

Rockslide 1y ago
Yes the whole consultancy situation really is the icing on the cake - as the customer you pay for (alleged) experts in the field and get this as the result...