6 ms·

Struggling with Container HTTP Logs? Meet Logmink

13 points by mutairibassam 2y ago

mutairibassam 2y ago
The Issue: When working with Docker containers, capturing and visualizing HTTP logs can be a pain. You usually need to modify your code or set up complicated logging systems just to see what’s happening with your app's HTTP requests. The Solution: Logmink! Logmink is an easy, plug-and-play tool that lets you capture HTTP traffic from your Docker containers without touching your app code! Just add the Logmink agent, and all your logs are sent to a central hub where you can store and visualize them. Why Use Logmink? Quick Setup: Get started in minutes, no code changes needed. Centralized Logs: View all HTTP requests from all containers in one place. Easy to Extend: Built on Node.js, so you can easily add your own features. Try it out and make monitoring your containers a breeze: Logmink GitHub Repo #Docker #DevOps #OpenSource #Logging #Containers #NodeJS
euph0ria 2y ago
What is the performance like when doing this? Doing pcap capturing and processing in node.js feels slow, but that is just a feeling and I don't have any data to back it up.. are there any performance impact studies?
- FooBarWidget 2y ago
  I also wonder whether the code actually works reliably. They have an "onpacket" handler on the pcap object, but this handler is, I think, called on every TCP packet. Which may not correspond to an HTTP session. Consider an app that starts an HTTP session by sending two TCP packets: 1. GET /foo 2. HTTP/1.1\r\n...etc... Their code would not catch this.
  - mutairibassam 2y ago
    I agree with you, currently I am only catching the request that have valid http method. I mentioned this in code comments there is todo task to capture different cases as well. Moreover, I am happy to learn more from you and if you have any recommendation please mention it. This simple implementation has met my needs. Regarding the performance, to be honest I've not tested it as I am not expert in this domain, so any hit is appreciated.
    - FooBarWidget 2y ago
      It would be better to stream the captured packets to a proper HTTP parser. That way you can also handle HTTP keepalive. Also, you can set a filter on the pcap session so that pcap only gives you http packets. Then you don't have to check yourself whether you captured non-http.
  - mutairibassam 2y ago
    I did some fixes, now it should capture all TCP packets. Appreciate your support to validate.
firesteelrain 2y ago
I am probably missing something. We just attached a Splunk UF container and forwarded all of our logs (events) to our centralized Splunk instance.
- mutairibassam 2y ago
  Please clarify the point or the challenge if this related to the package?
  - firesteelrain 2y ago
    Both tools (Splunk UF running as a sidecar and Logmink) serve the purpose of forwarding logs for centralized analysis. While Logmink has a narrow focus of just HTTP logs, Splunk is log general and can serve the same purpose. Splunk is considered more enterprise grade. Logmink focuses only on HTTP logs. But Splunk seems more appropriate here for enterprise environments.
    - szszrk 2y ago
      I am curious where this conversation will go, but still fail to see what is your actual question/doubt. These tools are vastly different and on other ends of both complexity and cost spectrums.
      - firesteelrain 2y ago
        He and I sorted it out above
    - mutairibassam 2y ago
      I completely agree with your points. I developed Logmink because tools like Splunk are very costly, are only managed by security teams (not application teams), and can take days to retrieve a single log packet, especially in production environments (or sometimes not at all due to security restrictions). As an application developer, I needed a straightforward way to check my app logs. We tried other tools like Dynatrace, but they were also expensive. We turned to open-source solutions like Logstash, Prometheus, Filebeat, and Grafana. While these tools are powerful, they require significant effort and expertise to set up. What we needed was a simpler HTTP-based solution for tracing requests across servers, which is why I developed Logmink. I hope this clarifies the difference.
PeterZaitsev 2y ago
Hm. Storing logs in MongoDB. Does it really scale ?
- szszrk 2y ago
  Can't tell if this is a joke or not [1]. I think it actually makes sense. I did a few implementations of Elastic and while it was great, but even with all aspects fully managed via code, at some point it consumed insane amount of brainpower to keep expanding, adding features, securing and patching it. It also consumed very large amounts of hardware - which I was fine with mostly, as this was onprem and costs were accepted. I tried something similar in a more cloud-based environment (both private and public) and I was shocked how much infrastructure I have to pay for just to create a reasonable PoC. Tried same with Openshift/Vector and it just gave me a headache. There are plenty of situations where you can accept a small risk of even losing some log data (due to low redundancy, scale vertically) and deploy something that is much simpler. Small apps with small volumes, or apps with large volumes, but you are OK with high latency of the solution and you are ok with slow (and cheap) VMs hosting it. It could work well with environments where you prefer to have more of the platform tools as dedicated nodes assigned to separate teams/apps (due to easier charging, or easier handling of noise neighbors, etc). [1] https://www.youtube.com/watch?v=HdnDXsqiPYo https://www.youtube.com/watch?v=HdnDXsqiPYo

13 points by mutairibassam 2y ago

The Issue: When working with Docker containers, capturing and visualizing HTTP logs can be a pain. You usually need to modify your code or set up complicated logging systems just to see what’s happening with your app's HTTP requests. The Solution: Logmink! Logmink is an easy, plug-and-play tool that lets you capture HTTP traffic from your Docker containers without touching your app code! Just add the Logmink agent, and all your logs are sent to a central hub where you can store and visualize them. Why Use Logmink? Quick Setup: Get started in minutes, no code changes needed. Centralized Logs: View all HTTP requests from all containers in one place. Easy to Extend: Built on Node.js, so you can easily add your own features. Try it out and make monitoring your containers a breeze: Logmink GitHub Repo #Docker #DevOps #OpenSource #Logging #Containers #NodeJS

euph0ria 2y ago

What is the performance like when doing this? Doing pcap capturing and processing in node.js feels slow, but that is just a feeling and I don't have any data to back it up.. are there any performance impact studies?

FooBarWidget 2y ago
I also wonder whether the code actually works reliably. They have an "onpacket" handler on the pcap object, but this handler is, I think, called on every TCP packet. Which may not correspond to an HTTP session. Consider an app that starts an HTTP session by sending two TCP packets: 1. GET /foo 2. HTTP/1.1\r\n...etc... Their code would not catch this.
- mutairibassam 2y ago
  I agree with you, currently I am only catching the request that have valid http method. I mentioned this in code comments there is todo task to capture different cases as well. Moreover, I am happy to learn more from you and if you have any recommendation please mention it. This simple implementation has met my needs. Regarding the performance, to be honest I've not tested it as I am not expert in this domain, so any hit is appreciated.
  - FooBarWidget 2y ago
    It would be better to stream the captured packets to a proper HTTP parser. That way you can also handle HTTP keepalive. Also, you can set a filter on the pcap session so that pcap only gives you http packets. Then you don't have to check yourself whether you captured non-http.
- mutairibassam 2y ago
  I did some fixes, now it should capture all TCP packets. Appreciate your support to validate.

firesteelrain 2y ago

I am probably missing something. We just attached a Splunk UF container and forwarded all of our logs (events) to our centralized Splunk instance.

mutairibassam 2y ago
Please clarify the point or the challenge if this related to the package?
- firesteelrain 2y ago
  Both tools (Splunk UF running as a sidecar and Logmink) serve the purpose of forwarding logs for centralized analysis. While Logmink has a narrow focus of just HTTP logs, Splunk is log general and can serve the same purpose. Splunk is considered more enterprise grade. Logmink focuses only on HTTP logs. But Splunk seems more appropriate here for enterprise environments.
  - szszrk 2y ago
    I am curious where this conversation will go, but still fail to see what is your actual question/doubt. These tools are vastly different and on other ends of both complexity and cost spectrums.
    - firesteelrain 2y ago
      He and I sorted it out above
  - mutairibassam 2y ago
    I completely agree with your points. I developed Logmink because tools like Splunk are very costly, are only managed by security teams (not application teams), and can take days to retrieve a single log packet, especially in production environments (or sometimes not at all due to security restrictions). As an application developer, I needed a straightforward way to check my app logs. We tried other tools like Dynatrace, but they were also expensive. We turned to open-source solutions like Logstash, Prometheus, Filebeat, and Grafana. While these tools are powerful, they require significant effort and expertise to set up. What we needed was a simpler HTTP-based solution for tracing requests across servers, which is why I developed Logmink. I hope this clarifies the difference.

PeterZaitsev 2y ago

Hm. Storing logs in MongoDB. Does it really scale ?

szszrk 2y ago
Can't tell if this is a joke or not [1]. I think it actually makes sense. I did a few implementations of Elastic and while it was great, but even with all aspects fully managed via code, at some point it consumed insane amount of brainpower to keep expanding, adding features, securing and patching it. It also consumed very large amounts of hardware - which I was fine with mostly, as this was onprem and costs were accepted. I tried something similar in a more cloud-based environment (both private and public) and I was shocked how much infrastructure I have to pay for just to create a reasonable PoC. Tried same with Openshift/Vector and it just gave me a headache. There are plenty of situations where you can accept a small risk of even losing some log data (due to low redundancy, scale vertically) and deploy something that is much simpler. Small apps with small volumes, or apps with large volumes, but you are OK with high latency of the solution and you are ok with slow (and cheap) VMs hosting it. It could work well with environments where you prefer to have more of the platform tools as dedicated nodes assigned to separate teams/apps (due to easier charging, or easier handling of noise neighbors, etc). [1] https://www.youtube.com/watch?v=HdnDXsqiPYo https://www.youtube.com/watch?v=HdnDXsqiPYo