7 ms·

LTTng: An open-source tracing framework for Linux

68 points by thezorg 12y ago

dmix 12y ago
Wow, open source sites are getting nice designs these days. Is there a business behind this project or is it via donation? The documentation is excellent.
- cbab 12y ago
  Thanks, we worked quite hard on this new redesign! edit: Yes, EfficiOS [1] is the open source business behind the LTTng project :). [1] - https://www.efficios.com/ https://www.efficios.com/
  - dmix 12y ago
    Some UX feedback after attempting to install it on Fedora: 1) Clicked Fedora on install docs 2) Installed lttng-tools and lttng-ust RPMs 3) Clicked Getting Started in docs. Ran "lttng list --kernel". 3a) Got an error saying I need to run the systemd service. 4) Started systemd service (didn't see this mentioned in the docs?) 5) Reran "lttng list --kernel".... error saying I'm missing a module 5a) Reread docs, realized I need to build LTTng-modules. Which led me to find out I need to also build liburcu 6) Installed liburcu but got an error building lttng-modules https://gist.github.com/dmix/d2d4c3c1b823e82486bf https://gist.github.com/dmix/d2d4c3c1b823e82486bf (I could submit this as a bug report?) So I gave up to attempt the install another time.
    - cbab 12y ago
      Thanks for reporting your issues on Fedora. 4) We will definitely fix that omission. 5a) liburcu is a dependency of the lttng-ust and lttng-tools. The dependency should be accounted for in the respective Fedora packages. 6) Yes please submit a bug for this at [1]. We will investigate that failure to build on Fedora. [1] - http://bugs.lttng.org/projects/lttng-modules http://bugs.lttng.org/projects/lttng-modules
    - mtdewcmu 12y ago
      But the website looks excellent.
jordigh 12y ago
Here is a pretty cool presentation about it, in French: https://www.youtube.com/watch?v=9XokHJadkq4 https://www.youtube.com/watch?v=9XokHJadkq4 The slides are in English, if that helps: http://lttng.org/files/papers/presentations/lttng20tracingforeveryone.pdf http://lttng.org/files/papers/presentations/lttng20tracingfo...
- cbab 12y ago
  You can also find all the presentations/publications that we did through the years here [1]. [1] - http://lttng.org/publications/ http://lttng.org/publications/
hitlin37 12y ago
how different is this from oprofile?
- abusque 12y ago
  Well, I've never used OProfile myself, so please take this answer with a grain of salt, but from a quick perusal of their website, and as the project's name seems to suggest, OProfile is a sampling profiler, which usually means it will collect data from counters at given intervals of time, producing a number of statistics over time. This generally implies a somewhat significant overhead, and consequent performance drop. LTTng, on the other hand, is a tracer, which means it collects events from the kernel (using the built-in tracepoint facilities) as they happen. It's also possible to trace userspace apps (with lttng-ust), or define your own tracepoints. This has the benefit of being much more detailed, and also has a much smaller overhead. You might be interested in reading the "What is Tracing?" section of the lttng docs [0], which does a far better job at explaining this than I do. [0] - http://lttng.org/docs/#doc-nuts-and-bolts http://lttng.org/docs/#doc-nuts-and-bolts
  - hitlin37 12y ago
    thanks for explaining. it sure looks interesting. i'm going to read now.
- krakensden 12y ago
  oprofile has been abandoned, and no longer works on recent distributions. 'perf', I think, is the usual replacement, though the tooling is not as nice.
rasur 12y ago
So how does this compare to somerthing like Sysdig, for example?
- mvlad 12y ago
  "Excluding proprietary solutions, a few competing software tracers exist for Linux. ftrace is the de facto function tracer of the Linux kernel. strace is able to record all system calls made by a user process. SystemTap is a Linux kernel and user space tracer which uses custom user scripts to produce plain text traces. sysdig also uses scripts, written in Lua, to trace and analyze the Linux kernel. The main distinctive features of LTTng is that it produces correlated kernel and user space traces, as well as doing so with the lowest overhead amongst other solutions. It produces trace files in the CTF format, an optimized file format for production and analyses of multi-gigabyte data. LTTng is the result of close to 10 years of active development by a community of passionate developers. It is currently available on all major desktop and embedded Linux distributions." Further down you'll discover that your application(s) are linked with a library provided by the project. Most of them rely on kprobes [1], or like ftrace on [2]. But there is also oprofile [3], and almighty perf [4], which rely on CPU extensions to implement tracing, (also implemented and by the HW manufacturer as well -- but in a OS-agnostic way). [1] http://lxr.free-electrons.com/source/Documentation/kprobes.txt http://lxr.free-electrons.com/source/Documentation/kprobes.t... [2] http://lxr.freeelectrons.com/source/Documentation/filesystems/debugfs.txt http://lxr.freeelectrons.com/source/Documentation/filesystem... [3] http://oprofile.sourceforge.net/doc/internals/index.html http://oprofile.sourceforge.net/doc/internals/index.html [4] https://perf.wiki.kernel.org/index.php/Main_Page https://perf.wiki.kernel.org/index.php/Main_Page
jpfr 12y ago
LTTng is closely related to liburcu (now on http://urcu.so/ http://urcu.so/). It provides lockless datastructures for screaming fast concurrency on multicore machines. You will be pleasantly surprised by the concurrent hash-map implementation that comes with it [1]. [1] here's the paper for background: https://www.usenix.org/conference/usenixatc11/resizable-scalable-concurrent-hash-tables-relativistic-programming https://www.usenix.org/conference/usenixatc11/resizable-scal...
- wrl 12y ago
  In the event LGPL is too restrictive, there's also concurrencykit (http://concurrencykit.org/ http://concurrencykit.org/), which provides similar mechanisms (both for concurrent memory reclamation and various datastructures built on top).

68 points by thezorg 12y ago

dmix 12y ago

Wow, open source sites are getting nice designs these days. Is there a business behind this project or is it via donation? The documentation is excellent.

cbab 12y ago
Thanks, we worked quite hard on this new redesign! edit: Yes, EfficiOS [1] is the open source business behind the LTTng project :). [1] - https://www.efficios.com/ https://www.efficios.com/
- dmix 12y ago
  Some UX feedback after attempting to install it on Fedora: 1) Clicked Fedora on install docs 2) Installed lttng-tools and lttng-ust RPMs 3) Clicked Getting Started in docs. Ran "lttng list --kernel". 3a) Got an error saying I need to run the systemd service. 4) Started systemd service (didn't see this mentioned in the docs?) 5) Reran "lttng list --kernel".... error saying I'm missing a module 5a) Reread docs, realized I need to build LTTng-modules. Which led me to find out I need to also build liburcu 6) Installed liburcu but got an error building lttng-modules https://gist.github.com/dmix/d2d4c3c1b823e82486bf https://gist.github.com/dmix/d2d4c3c1b823e82486bf (I could submit this as a bug report?) So I gave up to attempt the install another time.
  - cbab 12y ago
    Thanks for reporting your issues on Fedora. 4) We will definitely fix that omission. 5a) liburcu is a dependency of the lttng-ust and lttng-tools. The dependency should be accounted for in the respective Fedora packages. 6) Yes please submit a bug for this at [1]. We will investigate that failure to build on Fedora. [1] - http://bugs.lttng.org/projects/lttng-modules http://bugs.lttng.org/projects/lttng-modules
  - mtdewcmu 12y ago
    But the website looks excellent.

jordigh 12y ago

Here is a pretty cool presentation about it, in French: https://www.youtube.com/watch?v=9XokHJadkq4 https://www.youtube.com/watch?v=9XokHJadkq4 The slides are in English, if that helps: http://lttng.org/files/papers/presentations/lttng20tracingforeveryone.pdf http://lttng.org/files/papers/presentations/lttng20tracingfo...

cbab 12y ago
You can also find all the presentations/publications that we did through the years here [1]. [1] - http://lttng.org/publications/ http://lttng.org/publications/

hitlin37 12y ago

how different is this from oprofile?

abusque 12y ago
Well, I've never used OProfile myself, so please take this answer with a grain of salt, but from a quick perusal of their website, and as the project's name seems to suggest, OProfile is a sampling profiler, which usually means it will collect data from counters at given intervals of time, producing a number of statistics over time. This generally implies a somewhat significant overhead, and consequent performance drop. LTTng, on the other hand, is a tracer, which means it collects events from the kernel (using the built-in tracepoint facilities) as they happen. It's also possible to trace userspace apps (with lttng-ust), or define your own tracepoints. This has the benefit of being much more detailed, and also has a much smaller overhead. You might be interested in reading the "What is Tracing?" section of the lttng docs [0], which does a far better job at explaining this than I do. [0] - http://lttng.org/docs/#doc-nuts-and-bolts http://lttng.org/docs/#doc-nuts-and-bolts
- hitlin37 12y ago
  thanks for explaining. it sure looks interesting. i'm going to read now.
krakensden 12y ago
oprofile has been abandoned, and no longer works on recent distributions. 'perf', I think, is the usual replacement, though the tooling is not as nice.

rasur 12y ago

So how does this compare to somerthing like Sysdig, for example?

mvlad 12y ago
"Excluding proprietary solutions, a few competing software tracers exist for Linux. ftrace is the de facto function tracer of the Linux kernel. strace is able to record all system calls made by a user process. SystemTap is a Linux kernel and user space tracer which uses custom user scripts to produce plain text traces. sysdig also uses scripts, written in Lua, to trace and analyze the Linux kernel. The main distinctive features of LTTng is that it produces correlated kernel and user space traces, as well as doing so with the lowest overhead amongst other solutions. It produces trace files in the CTF format, an optimized file format for production and analyses of multi-gigabyte data. LTTng is the result of close to 10 years of active development by a community of passionate developers. It is currently available on all major desktop and embedded Linux distributions." Further down you'll discover that your application(s) are linked with a library provided by the project. Most of them rely on kprobes [1], or like ftrace on [2]. But there is also oprofile [3], and almighty perf [4], which rely on CPU extensions to implement tracing, (also implemented and by the HW manufacturer as well -- but in a OS-agnostic way). [1] http://lxr.free-electrons.com/source/Documentation/kprobes.txt http://lxr.free-electrons.com/source/Documentation/kprobes.t... [2] http://lxr.freeelectrons.com/source/Documentation/filesystems/debugfs.txt http://lxr.freeelectrons.com/source/Documentation/filesystem... [3] http://oprofile.sourceforge.net/doc/internals/index.html http://oprofile.sourceforge.net/doc/internals/index.html [4] https://perf.wiki.kernel.org/index.php/Main_Page https://perf.wiki.kernel.org/index.php/Main_Page

jpfr 12y ago

LTTng is closely related to liburcu (now on http://urcu.so/ http://urcu.so/). It provides lockless datastructures for screaming fast concurrency on multicore machines. You will be pleasantly surprised by the concurrent hash-map implementation that comes with it [1]. [1] here's the paper for background: https://www.usenix.org/conference/usenixatc11/resizable-scalable-concurrent-hash-tables-relativistic-programming https://www.usenix.org/conference/usenixatc11/resizable-scal...

wrl 12y ago
In the event LGPL is too restrictive, there's also concurrencykit (http://concurrencykit.org/ http://concurrencykit.org/), which provides similar mechanisms (both for concurrent memory reclamation and various datastructures built on top).