6 ms·

Amazon Redshift – New Features

29 points by ahjones 13y ago

gopalv 13y ago
HyperLogLog based COUNT DISTINCT APPROXIMATE, that's an interesting development in there (simple algorithm, but complex math). I wonder what hashes they are using.
- dmunoz 13y ago
  Only tangentially related to your comment, but last year there was a series of postings "Damn Cool Algorithms" in which the last post covered HyperLogLog in the post on cardinality estimation [0]. Some discussion in the HackerNews submission [1]. [0] http://blog.notdot.net/2012/09/Dam-Cool-Algorithms-Cardinality-Estimation http://blog.notdot.net/2012/09/Dam-Cool-Algorithms-Cardinali... [1] https://news.ycombinator.com/item?id=4488946 https://news.ycombinator.com/item?id=4488946
- vtuulos 13y ago
  one of the biggest benefits of HLL is that if you are only interested in set cardinalities, you don't have to store the original items at all, just a small HLL data structure which can be used to compute set unions, and with some caveats, intersections. It would be really great if Redshift supported HLL as a proper data type, and not just an optimization for COUNT(). A proper data type would allow us to store pre-aggregated data instead of billions of unnecessarily granular rows.
CurtMonash 13y ago
It sounds as if multi-node Redshift could actually be useful now (because it's picked up the minimally necessary data distribution options).
flavor8 13y ago
I'd be interested in hearing from anybody using RedShift in production. Currently shopping around for a database to store 1.5 billion interesting things, and RedShift's at the top of my list. Specifically - how many (small/large) nodes are you using, how big is your data, how many simultaneous users does your setup support, and what kind of performance do you see? Thanks in advance.
- sloth0000 13y ago
  I have not used Redshift in production but it's been getting pretty good reviews thus far. I did some testing using pretty small dataset which you can read more about here: http://bicortex.com/amazon-redshift-review-data-warehouse-in-the-cloud-is-here-part-1/ http://bicortex.com/amazon-redshift-review-data-warehouse-in... I don't think it's the answer to all data storage/warehousing needs but it does look promising and it should only get better.
  - rurounijones 13y ago
    Opened your page with interest but the tiny white text on black background is eye-searing. Even with manual zooming it is still a bit harsh due to the colours (Maybe just me though.).
- dude_abides 13y ago
  We're using it as a replacement for our enterprise data warehouse. Total size so far ~1 TB compressed, daily delta ~10s of GB compressed. We're using a two XL node cluster. The best part is that you can increase the cluster size without downtime (except you can't switch from XL to 8XL this way). Right now we have a single digit number of simultaneous users (analysts). Did not have to tweak the Workload Management configurations too much so far. I'd be interested to hear from others with bigger workload than ours. For example, someone from Netflix (They switched their data warehouse from Vertica to Redshift this year)
- onemoreaccount 13y ago
  We use Redshift in production to power a customer-facing app, but don't have web apps hit it directly (that really couldn't work with the concurrent query limits). Workers query RS and cache results in another database and the web app hits that database. We handle writes in the low 10s of billions per day. Our data format is very simple and we get great compression, so we currently fit everything on two clusters — one of 3 XLs and one of 6 XLs. Performance is great — except when it's not. Simple aggregations on tables that are < 1B rows and two table joins on tables that are < 100M rows are blazingly fast (maybe 1-30 seconds depending on the query). Larger tables than those and it can start to crawl.

29 points by ahjones 13y ago

HyperLogLog based COUNT DISTINCT APPROXIMATE, that's an interesting development in there (simple algorithm, but complex math). I wonder what hashes they are using.

dmunoz 13y ago
Only tangentially related to your comment, but last year there was a series of postings "Damn Cool Algorithms" in which the last post covered HyperLogLog in the post on cardinality estimation [0]. Some discussion in the HackerNews submission [1]. [0] http://blog.notdot.net/2012/09/Dam-Cool-Algorithms-Cardinality-Estimation http://blog.notdot.net/2012/09/Dam-Cool-Algorithms-Cardinali... [1] https://news.ycombinator.com/item?id=4488946 https://news.ycombinator.com/item?id=4488946
vtuulos 13y ago
one of the biggest benefits of HLL is that if you are only interested in set cardinalities, you don't have to store the original items at all, just a small HLL data structure which can be used to compute set unions, and with some caveats, intersections. It would be really great if Redshift supported HLL as a proper data type, and not just an optimization for COUNT(). A proper data type would allow us to store pre-aggregated data instead of billions of unnecessarily granular rows.

CurtMonash 13y ago

It sounds as if multi-node Redshift could actually be useful now (because it's picked up the minimally necessary data distribution options).

flavor8 13y ago

I'd be interested in hearing from anybody using RedShift in production. Currently shopping around for a database to store 1.5 billion interesting things, and RedShift's at the top of my list. Specifically - how many (small/large) nodes are you using, how big is your data, how many simultaneous users does your setup support, and what kind of performance do you see? Thanks in advance.

sloth0000 13y ago
I have not used Redshift in production but it's been getting pretty good reviews thus far. I did some testing using pretty small dataset which you can read more about here: http://bicortex.com/amazon-redshift-review-data-warehouse-in-the-cloud-is-here-part-1/ http://bicortex.com/amazon-redshift-review-data-warehouse-in... I don't think it's the answer to all data storage/warehousing needs but it does look promising and it should only get better.
- rurounijones 13y ago
  Opened your page with interest but the tiny white text on black background is eye-searing. Even with manual zooming it is still a bit harsh due to the colours (Maybe just me though.).
dude_abides 13y ago
We're using it as a replacement for our enterprise data warehouse. Total size so far ~1 TB compressed, daily delta ~10s of GB compressed. We're using a two XL node cluster. The best part is that you can increase the cluster size without downtime (except you can't switch from XL to 8XL this way). Right now we have a single digit number of simultaneous users (analysts). Did not have to tweak the Workload Management configurations too much so far. I'd be interested to hear from others with bigger workload than ours. For example, someone from Netflix (They switched their data warehouse from Vertica to Redshift this year)
onemoreaccount 13y ago
We use Redshift in production to power a customer-facing app, but don't have web apps hit it directly (that really couldn't work with the concurrent query limits). Workers query RS and cache results in another database and the web app hits that database. We handle writes in the low 10s of billions per day. Our data format is very simple and we get great compression, so we currently fit everything on two clusters — one of 3 XLs and one of 6 XLs. Performance is great — except when it's not. Simple aggregations on tables that are < 1B rows and two table joins on tables that are < 100M rows are blazingly fast (maybe 1-30 seconds depending on the query). Larger tables than those and it can start to crawl.