5 ms·

Show HN: LogDNA – Set up Kubernetes logging with 2 kubectl commands

76 points by leeab 10y ago

leeab 10y ago
Hi everyone, I'm a co-founder / CTO of LogDNA. We were in Y Combinator's W15 batch and launched our cloud logging platform last year (https://news.ycombinator.com/item?id=11074537 https://news.ycombinator.com/item?id=11074537) Based on user feedback, we're happy to announce our super easy Kubernetes integration. No more wrestling with fluentd configs, fiddling with Elasticsearch knobs or following 30-step guides cutting and pasting other people's configs. kubectl create secret generic logdna-agent-key --from-literal=logdna-agent-key=<YOUR LOGDNA API KEY> kubectl create -f https://raw.githubusercontent.com/logdna/logdna-agent/master/logdna-agent-ds.yaml We're looking for some feedback on how we can improve this integration. We currently extract Kubernetes metadata: pod name, container name, container id, namespace. Feel free to try it out. Happy to answer any questions!
- allanshin 10y ago
  kubernetes integration FTW! Thank you, thank you!!
  - leeab 10y ago
    :) Hope you'll give us a shot. And let us know what you think!
- HorizonXP 10y ago
  Thanks for putting this out there Lee! This is huge! How do you guys handle multi-line log entries? This is the hardest part with existing setups, and would really help us trace exceptions as they occur in real-time.
  - leeab 10y ago
    We currently treat each line separately and we can maintain line ordering okay. We have thought about this before and doing something like: if a line starts with tab or a fixed number of spaces a few times in a row, treat it as 1 line with \n's and store it as such. It would help with alerting and filtering alerts. Just not 100% sure if this will screw up anything.
- weitzj 10y ago
  It would be nice to improve the documentation a bit, and mention that only one agent is needed per node and that this agent will collect the logs across namespaces. As I have understood,you do not set any namespace for the secret/daemonset, which is fine with me. But this way, I could install multiple agents (for each namespace). So a better documentation about the best practice installation for multiple namespaces would be nice.
  - leeab 10y ago
    Ahh yes, thanks for catching that. I just made a note that we collect from all namespaces currently. Yeah, we now set our agent up w/o namespaces. During our beta, we originally had it set up inside `kube-system` but 2 of our testers mentioned that the pod wouldn't install unless it was in `default`. So we moved it out of `kube-system` but we still weren't sure what caused the issue since it worked fine on our cluster. We were using Kubernetes v1.4 so it could've been an older version issue.
    - lobster_johnson 10y ago
      Actually, kube-system is the right namespace for things like this. I don't know why it didn't work -- maybe they put the secret in the wrong namespace?
      - leeab 10y ago
        Yeah one of those mysteries of life. :/ One day we'll probably go back and try this on a few older versions to see if we can repro what happened. But yeah, it was right off the bat when we started our beta, 2 different people said they had to modify the yaml in order to get it to work (something we wanted to avoid altogether).
- od0 10y ago
  Great work with this - definitely a pain point. Getting set up and trying it out now.
  - leeab 10y ago
    Awesome! We're pretty responsive on Intercom as well if you run into issues post install. But definitely share your feedback if you notice areas that can be improved.
- weitzj 10y ago
  Also interesting: I have used the docker integration of logdna beforehand, before moving to Kubernetes. The integration was done using docker compose per docker compose environment. The logs which contained `err` were marked red as errors in logdna and I could trigger an alarm. The same containers in Kubernetes with the same logs seem to marked as `info` now. I am not sure, why this is and how I can get the same behavior as before. Is there a way to tell Kubernetes about stderr/stdout? Or how would I trigger logdna to treat a log as error instead of info?
  - leeab 10y ago
    Hmmm this may have been a bug. It should show stderr as err. Let me look into this.
    - deleted 10y ago
      [deleted]
- gkop 10y ago
  I love that you appear to have been built from the ground up to be HIPAA-compliant. I have emailed your sales team :)
manojlds 10y ago
Seems odd that this is getting upvotes. Most stuff is 2 (actually 1) commands away in Kubernetes ONCE you have the necessary manifests.
- leeab 10y ago
  Yes that's true for many things in Kubernetes but not as frequent in the logging space for Kubernetes. Many still rely on Fluentd as a parsing filter first to extract metadata like pod name or container name. And most of the logging ones require you to edit your yaml in some way first. Ours is almost copy-pasteable, provided you have a LogDNA API key. :)
- andrewstuart2 10y ago
  I don't think it's odd that it's getting upvotes, but centralized logging is already a feature (as an addon) of k8s. I have a helm chart that's a one-shot for setting up ELK and FluentD as a system service that aggregates all docker stdout logs and tags each stream with k8s metadata, so you can very easily slice/dice your logs even at scale. It includes a cronjob running es-curator so that logs older than some configurable threshold are automatically deleted. e.g. output from my dev cluster: ~CK/elk/templates git:(master) kc cluster-info Kubernetes master is running at https://192.168.16.16:8443 Elasticsearch is running at https://192.168.16.16:8443/api/v1/proxy/namespaces/kube-system/services/elasticsearch-logging Heapster is running at https://192.168.16.16:8443/api/v1/proxy/namespaces/kube-system/services/heapster KubeDNS is running at https://192.168.16.16:8443/api/v1/proxy/namespaces/kube-system/services/kube-dns monitoring-grafana is running at https://192.168.16.16:8443/api/v1/proxy/namespaces/kube-system/services/monitoring-grafana I think people want easy management of this, but I do wonder how successful this particular integration will be with such low-hanging self-managed alternatives. Where I think LogDNA will have a win is that you probably also have things running outside k8s that LogDNA helps integrate. So if you want all your k8s and non-k8s logs aggregated, and don't want to mess with it, then you might go with LogDNA.
  - ShaneOG 10y ago
    Is this extremely useful chart public?
jdotjdot 10y ago
Congrats! LogDNA is one of my favorite products. We're likely moving to Kubernetes shortly and were actually worried about logging, so this makes my life a lot easier
- leeab 10y ago
  Thanks JJ! Yeah, we hope to earn your Kubernetes business :)
jazoom 10y ago
I've used LogDNA for a while now. It has been a good experience.
- leeab 10y ago
  Awesome! And definitely let us know if we can improve in any way.

76 points by leeab 10y ago

leeab 10y ago

Hi everyone, I'm a co-founder / CTO of LogDNA. We were in Y Combinator's W15 batch and launched our cloud logging platform last year (https://news.ycombinator.com/item?id=11074537 https://news.ycombinator.com/item?id=11074537) Based on user feedback, we're happy to announce our super easy Kubernetes integration. No more wrestling with fluentd configs, fiddling with Elasticsearch knobs or following 30-step guides cutting and pasting other people's configs. kubectl create secret generic logdna-agent-key --from-literal=logdna-agent-key=<YOUR LOGDNA API KEY> kubectl create -f https://raw.githubusercontent.com/logdna/logdna-agent/master/logdna-agent-ds.yaml We're looking for some feedback on how we can improve this integration. We currently extract Kubernetes metadata: pod name, container name, container id, namespace. Feel free to try it out. Happy to answer any questions!

allanshin 10y ago
kubernetes integration FTW! Thank you, thank you!!
- leeab 10y ago
  :) Hope you'll give us a shot. And let us know what you think!
HorizonXP 10y ago
Thanks for putting this out there Lee! This is huge! How do you guys handle multi-line log entries? This is the hardest part with existing setups, and would really help us trace exceptions as they occur in real-time.
- leeab 10y ago
  We currently treat each line separately and we can maintain line ordering okay. We have thought about this before and doing something like: if a line starts with tab or a fixed number of spaces a few times in a row, treat it as 1 line with \n's and store it as such. It would help with alerting and filtering alerts. Just not 100% sure if this will screw up anything.
weitzj 10y ago
It would be nice to improve the documentation a bit, and mention that only one agent is needed per node and that this agent will collect the logs across namespaces. As I have understood,you do not set any namespace for the secret/daemonset, which is fine with me. But this way, I could install multiple agents (for each namespace). So a better documentation about the best practice installation for multiple namespaces would be nice.
- leeab 10y ago
  Ahh yes, thanks for catching that. I just made a note that we collect from all namespaces currently. Yeah, we now set our agent up w/o namespaces. During our beta, we originally had it set up inside `kube-system` but 2 of our testers mentioned that the pod wouldn't install unless it was in `default`. So we moved it out of `kube-system` but we still weren't sure what caused the issue since it worked fine on our cluster. We were using Kubernetes v1.4 so it could've been an older version issue.
  - lobster_johnson 10y ago
    Actually, kube-system is the right namespace for things like this. I don't know why it didn't work -- maybe they put the secret in the wrong namespace?
    - leeab 10y ago
      Yeah one of those mysteries of life. :/ One day we'll probably go back and try this on a few older versions to see if we can repro what happened. But yeah, it was right off the bat when we started our beta, 2 different people said they had to modify the yaml in order to get it to work (something we wanted to avoid altogether).
od0 10y ago
Great work with this - definitely a pain point. Getting set up and trying it out now.
- leeab 10y ago
  Awesome! We're pretty responsive on Intercom as well if you run into issues post install. But definitely share your feedback if you notice areas that can be improved.
weitzj 10y ago
Also interesting: I have used the docker integration of logdna beforehand, before moving to Kubernetes. The integration was done using docker compose per docker compose environment. The logs which contained `err` were marked red as errors in logdna and I could trigger an alarm. The same containers in Kubernetes with the same logs seem to marked as `info` now. I am not sure, why this is and how I can get the same behavior as before. Is there a way to tell Kubernetes about stderr/stdout? Or how would I trigger logdna to treat a log as error instead of info?
- leeab 10y ago
  Hmmm this may have been a bug. It should show stderr as err. Let me look into this.
  - deleted 10y ago
    [deleted]
gkop 10y ago
I love that you appear to have been built from the ground up to be HIPAA-compliant. I have emailed your sales team :)

manojlds 10y ago

Seems odd that this is getting upvotes. Most stuff is 2 (actually 1) commands away in Kubernetes ONCE you have the necessary manifests.

leeab 10y ago
Yes that's true for many things in Kubernetes but not as frequent in the logging space for Kubernetes. Many still rely on Fluentd as a parsing filter first to extract metadata like pod name or container name. And most of the logging ones require you to edit your yaml in some way first. Ours is almost copy-pasteable, provided you have a LogDNA API key. :)
andrewstuart2 10y ago
I don't think it's odd that it's getting upvotes, but centralized logging is already a feature (as an addon) of k8s. I have a helm chart that's a one-shot for setting up ELK and FluentD as a system service that aggregates all docker stdout logs and tags each stream with k8s metadata, so you can very easily slice/dice your logs even at scale. It includes a cronjob running es-curator so that logs older than some configurable threshold are automatically deleted. e.g. output from my dev cluster: ~CK/elk/templates git:(master) kc cluster-info Kubernetes master is running at https://192.168.16.16:8443 Elasticsearch is running at https://192.168.16.16:8443/api/v1/proxy/namespaces/kube-system/services/elasticsearch-logging Heapster is running at https://192.168.16.16:8443/api/v1/proxy/namespaces/kube-system/services/heapster KubeDNS is running at https://192.168.16.16:8443/api/v1/proxy/namespaces/kube-system/services/kube-dns monitoring-grafana is running at https://192.168.16.16:8443/api/v1/proxy/namespaces/kube-system/services/monitoring-grafana I think people want easy management of this, but I do wonder how successful this particular integration will be with such low-hanging self-managed alternatives. Where I think LogDNA will have a win is that you probably also have things running outside k8s that LogDNA helps integrate. So if you want all your k8s and non-k8s logs aggregated, and don't want to mess with it, then you might go with LogDNA.
- ShaneOG 10y ago
  Is this extremely useful chart public?

jdotjdot 10y ago

Congrats! LogDNA is one of my favorite products. We're likely moving to Kubernetes shortly and were actually worried about logging, so this makes my life a lot easier

leeab 10y ago
Thanks JJ! Yeah, we hope to earn your Kubernetes business :)

jazoom 10y ago

I've used LogDNA for a while now. It has been a good experience.

leeab 10y ago
Awesome! And definitely let us know if we can improve in any way.