8 ms·
Security for Elasticsearch is now free
- mrkstu 7y agoFree, not open source version. Obviously a reaction to Amazon's fork- not wanting to give them any code to pull into their version. It will be interesting to see if this is enough to retain the majority of the userbase or if we'll still see a majority migrate to the 'Open Distro' fork.
- chelmzy 7y agoI was under the impression that they already opened the code for X-Pack features.
- lol768 7y agoWere these features licensed in such a way that you could freely use them, though? Or was it 'open' as in "you can see the code" but it's not FOSS?
- Jedd 7y agoRefer this thread from 2018-02 when the X-Pack licensing change was announced: https://news.ycombinator.com/item?id=16487440 https://news.ycombinator.com/item?id=16487440 Summary -- vague use of the word 'open' and exclusive use of free in the beer sense, led to some significant angst about what this means for end users.
- jka 7y agoGood catch and an important distinction. Some background here: https://devclass.com/2019/03/12/aws-launches-elasticsearch-distro-to-guard-customers-against-muddy-open-source-waters/ https://devclass.com/2019/03/12/aws-launches-elasticsearch-d...
- yehosef 7y agoThe features of the open distro are not enough to compare to the Elastic offerings. I think most are smart enough to see through Amazon's "generosity" and know that they (Amazon) are not a bastion of OSS.
- noname120 7y agoCould you expand on the features that Open Distro misses and that Elastic offers?
- yehosef 7y agoYou can see the OSS and Free offerings at https://www.elastic.co/subscriptions https://www.elastic.co/subscriptions. Real question is what are the Open Distro is offering: * Alerting - you can use ElasticAlert * Security - Search Guard * SQL - https://github.com/NLPchina/elasticsearch-sql https://github.com/NLPchina/elasticsearch-sql There very little reason (IMO) for users to a choose "Open Distro" except that it comes as an AWS image.
- noname120 7y agoMaybe a real reason is that it's precisely a distro which means it contains features and reasonable defaults so that users don't need to learn about them, install them, and configure them. Basically same debate as between Linux From Scratch and a full-featured distro such as Ubuntu.
- mrkstu 7y agoAlso note that you'll want to go to the features page and hit the disclosure triangle on the 'Security' feature. This is very much a subset of their security features- no IP filtering, AD/LDAP integration, SAML or many other security oriented features. https://www.elastic.co/subscriptions https://www.elastic.co/subscriptions
- lol768 7y agoI'm actually okay with these more enterprise features still being premium. Basic RBAC, TLS support and user management should've been core from the start for free, though.
- dhd415 7y agoTo be fair, the Docker Hub stats for the Open Distro for ES image don't show a very large shift away the Elastic ES image. And I doubt Amazon really cares about that, either. This is about whether their hosted ES service remains competitive with Elastic's hosted ES service.
- glloydell 7y agoTotally agree. Amazon had to do something after Elastic changed their license terms specifically to stop Amazon from competing with their own hosted service. I'm interested to see what Elastics next move will be. grabs popcorn
- jillesvangurp 7y agoOpen distro is definitely creating some pressure on ES. However, there are some misconceptions on what that is. 1) Amazon did not actually fork elasticsearch or maintains any patches against it. 2) Elasticsearch does in fact provide completely OSS distributions and docker images for their products. 3) Amazon has created several OSS plugins for Elasticsearch that they bundle with their open distro that compete directly with what Elasticsearch does in their non OSS add-ons to their product. So, obviously Elasticsearch is responding to Amazon by ensuring there's little functional gap with the stuff you get for free. I'd argue most new users are still better off on elastic cloud vs amazon's hosted version of their distro and should not be attempting to run this themselves. I've used both and would pick elastic cloud every time for the simple reason of being more reliable and easy to deal with (e.g. backups, upgrades, cluster topology changes, etc.). Also, it seems they are quite competitive on price/performance. For reference, we pay about 170 Euro a month for a simple setup that takes care of all our logging (couple of GB worth of logs / day). I'd hate running blind without that. IMHO at those prices, self hosting is not worth the effort (devops time required to do it would pay for several years of hosting).
- jjeaff 7y agoRegarding the docker images elastic provides, I find it odd that they are only hosted from their own servers (rather than docker hub) and I have looked all over the place to find the dockerfile they use to create those images. It seems that they are hiding that info and it really locks you into only doing customizations that the docker image is directly built for. I also don't like to pull images blindly. I generally fork the dockerfile source so that I can build the software from source and have a bit more control and knowledge of what I am installing.
- jillesvangurp 7y agoAfter 1 google search: https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html https://www.elastic.co/guide/en/elasticsearch/reference/curr... It includes a deep link straight into their repository right at the top: https://github.com/elastic/elasticsearch/tree/7.1/distribution/docker https://github.com/elastic/elasticsearch/tree/7.1/distributi... In short, their Docker build process is part of the elasticsearch repository. So, it's actually part of their normal build process and not something that happens with a separate build in some different repository. Personally, I think this is a good practice. That's for both the OSS and non OSS images. They produce these with every build. And they probably test them too, which I think is the responsible thing to do and something I'd expect from them. They use gradle to build their docker images from a Docker file (https://github.com/elastic/elasticsearch/blob/7.1/distribution/docker/src/docker/Dockerfile https://github.com/elastic/elasticsearch/blob/7.1/distributi...) that essentially untars the tar ball produced by their build. This looks pretty straightforward and free of magic steps. So, read the source. It's all there. You can build from source or do your own thing. A variant of their Docker file where you just wget their tar ball shouldn't be that hard to do.
- pmart123 7y agoDo you have a good understanding of the legal differences? My understanding is there now an open source license that anyone can fork, including companies basically committing IP theft, and then a basic license that is free unless you are selling Elastic as a service? The idea is that Elastic would put all improvements into Elastic Basic, and Amazon can't use this source code in its forked version? IMO, it doesn't seem like a near-term risk, but could Elastic ever change its basic license so it costs money for everyone?
- KenanSulayman 7y agoInteresting. Three hours ago someone in our Ops team shared a link to "Open Distro for Elasticsearch" [1] and it's also featured on the AWS console login page. Is this a very rushed reaction to it? Or is this related? I would really love to have a clarification of what's happening in that space. [1] https://opendistro.github.io/for-elasticsearch/ https://opendistro.github.io/for-elasticsearch/
- igama 7y agoOpendistro was announced by AWS a few weeks ago. It’s their fork of ES with security features and some of the XPack functionality included.
- aasasd 7y agoHow were you able to miss all the drama about AWS and ElasticSearch / Mongo / Nginx here on HN? It turned out that with the open core and premium-service models, the original company might not be the only one providing paid services or development. Which was a bit of a surprise to those original devs.
- yehosef 7y agosee https://grafana.com/blog/2019/03/28/everything-you-need-to-know-about-the-oss-licensing-war-part-2./ https://grafana.com/blog/2019/03/28/everything-you-need-to-k... and the first part of the story it links to.
- syrrim 7y agoopendistro has this: https://github.com/opendistro-for-elasticsearch/security https://github.com/opendistro-for-elasticsearch/security which has feature parity with the free version elastic just released afaict.
- majkinetor 7y agoNo it doesn't - for example LDAP/AD are paid feature in ES
- vorpalhex 7y agoToo little too late? Trying to charge for TLS was a very poor move and it's made me not trust ElasticSearch...
- deleted 7y ago[deleted]
- nickserv 7y agoI can understand they need to make money, but still a bit shady. Honestly though it's not something critical for my needs. Now if they could lower the resource hogging a bit, that would interest me... Maybe even pay for that.
- jrockway 7y agoI use ELK for Kubernetes and network device logs, and I'm very much with you -- full text search is great, but it sure can be slow, even when running on $1000/month of AWS hardware. The conclusion that I've reached is that the whole lucene model for logs is kind of outdated. Why am I tuning Java GC params to run "grep foo /logs". I think computers today can do fine with sharded flat files, a minimal index ("which node contains logs from pod foo-2387438-2384738 at 12:34AM"), and then just scale horizontally over (log messages, searches). I hope my friends over at Tailscale are doing that and I can just move off ES entirely ;)
- ryanworl 7y agoIs Tailscale building a logging product?
- ReidZB 7y agoI believe Loki [1] is intended to basically run "grep foo" at scale (plus some extra niceties like labels). I haven't used it, but it seems interesting. [1]: https://grafana.com/loki https://grafana.com/loki
- vorpalhex 7y agoELK stack user here - we actually found logstash to be our bottleneck. Changing it out for fluentd fixed our woes.
- haney 7y agoI know it's hard to make a buck with an open source business model but deciding to charge more for security related features is always so frustrating to me. It leads to a culture of insecure deployments in environments when the business is trying to save money. Differentiate on storage or number of cores or something, anything but auth/security. I'm glad they've finally reversed this.
- tedk-42 7y agoRunning Elasticsearch on K8s storing 16TB of compressed logs across 6 data nodes and ~4600 shards. We're a really happy ES customer. We've on ESv6 at the moment and it's been running amazingly for us. We've halved our storage and running costs by moving from 5 to 6. We've always been a licensed customer and they are in front of AWS with their features (we run our k8s stack on AWS though :) )
- KenanSulayman 7y agoHow did ES 6 reduce your operation costs by 50%? Same CPU, disk & network bandwidth?
- praseodym 7y agoSome free advice: reduce the number of shards! Each shard comes with some state management overhead. The soft limit is currently at 1000 shards per node, but you should be aiming at 25-50GB of data per shard.
- parliament32 7y agoThat's an insane number of shards, you should be closer to 500 for that amount of data and only 6 nodes.
- tedk-42 7y agoIt's due to the number of indicies/indexes stored from our various data sources. Yours and another poster's comments are interesting so we might look at ways we can reduce the shard count given the new info on overhead.
- mattupstate 7y agoThere's also a lesser known project out there: https://search-guard.com/ https://search-guard.com/ Paired with an OpenResty reverse proxy I was able to set up a reasonably secure cluster back when X-Pack was prohibitively expensive and the AWS offering wasn't under their BAA. Big thanks to that team of contributors!
- reilly3000 7y agoSome of the worst breaches of 2017-19 have been due to open ES clusters, some on AWS. This is a welcome change. I just spun our AWS ES cluster down in favor of BigQuery, but while I was setting it up security for it was a big chore, with defaults that are in no way sane. AWS EC2 does a great job at secure defaults for auth and firewalls, RDS even moreso. Why was ES left to wag in the wind out of the box?
- mrmondo 7y agoThis (while perhaps not perfect) is massive for us, it’s going to be especially useful for Kibana authentication to add readonly and write users, something we’ve wanted for a long time but haven’t been able to afford as a non-profit, charitable organisation. I know it’s not all 100% open source, but it’s better than a nginx reverse proxy hack or similar. Thank you Elastic for continuing to create fantastic software.
- dadoonet 7y agoDid you apply to this program? https://www.elastic.co/elastic-search-awards/ https://www.elastic.co/elastic-search-awards/ That could help your hopefully.