6 ms·
DeepFlow – open-source eBPF Distributed Tracing
- bobberkarl 3y agoHow are you different from Pixie?
- sharangxy 3y agoThe biggest difference: DeepFlow enables *Distributed* Tracing. In addition, DeepFlow combines the capabilities of eBPF and cBPF to achieve full-stack tracing of syscall + network_forward. You can take a look at our documentation: https://deepflow.io/docs/about/features/ https://deepflow.io/docs/about/features/
- bigcat12345678 3y agoNice project, have been following this project casually for a while. The standout feature is to trace RPC flow across network connections, through packet tracing.
- nimrody 3y agoHow can it tie requests arriving at a service and generating additional downstream requests? Distributed tracing needs some common token all requests share to identify all RPCs that should be associated with a specific incoming request.
- archivator 3y agoTake a look at Core Feature #2 in this post - https://deepflow.io/ebpf-the-key-technology-to-observability/#0x2-Three-Core-Features-of-DeepFlow-Based-on-eBPF https://deepflow.io/ebpf-the-key-technology-to-observability... It looks like it's using tcp flow tuple + tcp_seq to join things.
- Eridrus 3y agoIt looks like it depends on applications either using threads or go routines for concurrency: > When collecting invocation logs through eBPF and cBPF, DeepFlow calculates information such as syscall_trace_id, thread_id, goroutine_id, cap_seq, tcp_seq based on the system call context. This allows for distributed tracing without modifying application code or injecting TraceID and SpanID. Currently, DeepFlow can achieve Zero Code distributed tracing for all cases except for cross-thread communication (through memory queues or channels) and asynchronous invocations.
- sharangxy 3y agoVP of DeepFlow here. Thank you for your interest in DeepFlow! Yes, we have implemented distributed tracing using eBPF. In simple terms, we use thread-id, coroutine-id, and tcp-seq to automatically correlate all spans. Most importantly, we use eBPF to calculate a syscall-trace-id (without the need to propagate it between upstream and downstream), enabling automatic correlation of a service's ingress and egress requests. For more details, you can refer to our paper presented at SIGCOMM'23: https://dl.acm.org/doi/10.1145/3603269.3604823 https://dl.acm.org/doi/10.1145/3603269.3604823. Of course, this kind of Zero Code distributed tracing currently has some limitations. For specific details, please see: https://deepflow.io/docs/features/distributed-tracing/auto-tracing/#current-limitations https://deepflow.io/docs/features/distributed-tracing/auto-t... These limitations are not entirely insurmountable. We are actively working on resolving them and continually making breakthroughs.
- inhumantsar 3y agowould it be reasonable to assume that because this entirely network-based, it would work best with systems which really emphasize the "micro" in microservices? how well does this work if, say, my system has a legacy monolith in addition to microservices?
- sharangxy 3y agoI believe the current situation is like this. The advantage of eBPF lies in *request granularity* (i.e. PRC, API, SQL, etc ...) distributed tracing. To trace the internal functions of an application, instrumentation is still required for coverage. Therefore, the finer the service decomposition, the more effective eBPF's distributed tracing becomes.
- mdaniel 3y agoApache 2, if that interests you: https://github.com/deepflowio/deepflow/blob/v6.4.7/LICENSE https://github.com/deepflowio/deepflow/blob/v6.4.7/LICENSE heh, GitHub also has "symbol navigation" turned on for that license file but I didn't dig into it to find out what source language it thinks the file is
- deleted 3y ago[deleted]
- javierhonduco 3y agoHaven’t checked the source code yet, wondering if profiling of code without frame pointes is supported. Curious on their approach.
- reactordev 3y agoIt uses eBPF to provide instrumentation of the kernel calls up as well as hooking into networking for http2 pgsql etc. Since it’s running the process in eBPF it’s essentially sandboxed and all memory, kernel function calls, and even profiling, is an option. They have an agent that collects this information and sends to the server over RPC (protobuf/grpc). You should check it out (however, some of the docs are in Chinese).
- sharangxy 3y agoWe are in the process of translating more documents into English, and we also greatly welcome the help of the community!
- progbits 3y ago> DeepFlow can even analyze code performance through network profiling under old version kernels (2.6+). This is an interesting call-out, the last release of 2.6 is from 2011. I wonder who is still running that in production.
- sharangxy 3y agoSome of our users’ ancestral processes are running on kernel 2.6, and the operations staff dare not upgrade the kernel. Indeed, there are many limitations in 2.6, but the simple traffic analysis has brought surprising insights to users. However, this also brings some troubles: even if a problem is known, no one dares to easily modify the code to fix it, unless absolutely necessary :)
- z523552999 3y ago[dead]
- ilovesnow 3y agonice
- robertheadley 3y agoI don't have any idea what this is, but the graphs are beautiful.
- jiumos 3y ago[dead]
- deleted 3y ago[deleted]
- cyber_chen 3y ago[dead]
- jinxiao2010 3y agogood project. I'll take a try. ebpf is so popular, but we're still using traditional network plugins.