10 ms·

Twenty years of Valgrind

626 points by nnethercote 4y ago

syockit 4y ago
There are times when LeakSanitizer (in gcc-8.2) would not give me the full backtrace of a leak, while valgrind would, so to me it's still an indispensable tool for debugging leaks. One caveat is that it's magnitudes slower than valgrind. Now, if only I know how to make valgrind run as fast as LeakSanitizer... (command line options?)
- rigtorp 4y ago
  You might need to add -fno-omit-frame-pointer to help ASAN unwind the stack.
  - abbeyj 4y ago
    This is definitely an option you want to be using when using ASan or LSan. You may also want to consider additionally using -momit-leaf-frame-pointer to skip frame pointers only in leaf functions while keeping frame pointers for non-leaf functions. This can make small leaf functions significantly shorter, limiting some of the negative impact of using -fno-omit-frame-pointer alone. Sometimes even -fno-omit-frame-pointer won't help, like if the stack is being unwound through a system library that was built without frame pointers. In that case you can switch to the slow unwinder. Set the environment variable `ASAN_OPTIONS=fast_unwind_on_malloc=0` when running your program. But note that this will make most programs run significantly slower so you probably want to use it only when you really need it and not as the default setting for all runs.
mynegation 4y ago
I am old enough that I started with Purify and I used Valgrind starting from the version 1.0, because Purify was commercial and Solaris only. It saved my behind multiple multiple times.
- hn_go_brrrrr 4y ago
  I worked at a company 11 years ago that was still using Purify!
  - unmole 4y ago
    I used Purify 8 years ago. On Windows. I don't remember the specifics but the company kept a few XP machines around just so they could continue using Purify.
- atgreen 4y ago
  Purify was an amazing tool. I recently noticed that one of my libraries (libffi) still has an --enable-purify configure option, although it probably hasn't been exercised in.. 20 years? A Purify patent prevented work-alikes for many years, but valgrind eventually emerged as a more-than-worthy successor. Fun fact: the creator of Purify went on to found Netflix and is still their CEO.
  - mynegation 4y ago
    Ha! And I thought that the same person writing bzip2 and Val grind is my surprise for the day.
    - snovv_crash 4y ago
      And Ardupilot
- cpeterso 4y ago
  And BoundsChecker was also great! https://en.m.wikipedia.org/wiki/BoundsChecker https://en.m.wikipedia.org/wiki/BoundsChecker
  - sumtechguy 4y ago
    That tool saved me tons of time tracking down bugs. It also taught me to be a better C/C++ programmer. Run time sanitizers like Purify/Valgrind/Boundchecker do not tolerate poor C code. What is kind of cool is you can find whole classes of bugs in your code. Because as devs we get something working once we tend to copy and paste that pattern everywhere. So find a bug in one place you will probably find it a few dozen other places in your codebase.
nicoburns 4y ago
Well damn, no wonder he’s so good at optimising the Rust compiler. He literally has a PhD in profiling tools!
gkhartman 4y ago
Many thanks for Valgrind. I can honestly say that it helped me become a better C++ programmer.
junon 4y ago
Valgrind's maintainers are super pleasant and have been quite helpful in a number of cases I've personally had to reach out to them. Lovely piece of software toward which I owe a lot of gratitude.
pjmlp 4y ago
> Speaking of software quality, I think it’s fitting that I now work full time on Rust, a systems programming language that didn’t exist when Valgrind was created, but which basically prevents all the problems that Memcheck detects. Just like Ada has been doing since 1983.
- oconnor663 4y ago
  My understanding is that dynamically freeing memory is an unsafe operation in Ada, do I have that right?
  - pjmlp 4y ago
    Depends on which dynamic memory you are talking about. Ada can manage dynamic stacks, strings and arrays on its own. For example, Ada has what one could call type safe VLAs, instead of corrupting the stack like C, you get an exception and can redo the call with a smaller size, for example. As for explicit heap types and Ada.Unchecked_Deallocation, yes if we are speaking about Ada 83. Ada 95 introduced controlled types, which via Initialize, Adjust, and Finalize, provide the basis of RAII like features in Ada. Here is an example on how to implement smart pointers with controlled types, https://www.adacore.com/gems/gem-97-reference-counting-in-ada-part-1 https://www.adacore.com/gems/gem-97-reference-counting-in-ad... There is also the possiblity to wrap heap allocation primitives with safe interfaces exposed via storage pools, like on this tutorial https://blog.adacore.com/header-storage-pools https://blog.adacore.com/header-storage-pools Finally thanks to SPARK, nowadays integrated into Ada 2012[0], you can also have formal proofs that it is safe to release heap memory. In top of all this, Ada is in the process of integrating affine types as well. [0] - Supported in PTC and GNAT, remaining Ada compilers have a mix of Ada 95 - 2012 features, see https://news.ycombinator.com/item?id=27603292 https://news.ycombinator.com/item?id=27603292
    - touisteur 4y ago
      That said, I still use valgrind because we have to integrate C libraries sometimes (libpcl is my favorite culprit, only because I'm trying , and there's still possibility to blow the stack (yeah you can use gnatstack to get a good idea of your maximum stack size, but it's doesn't cover the whole Ada featureset and stack canaries - fstack-check don't catch everything. edit Also massif, call/cachegrind and hellgrind have saved our bacon many, many times. Even more interesting is writing your own tools with valgrind. Here https://github.com/AdaCore/gnatcoverage/tree/master/tools/gnatcov/trace_adapters/valgrind https://github.com/AdaCore/gnatcoverage/tree/master/tools/gn... is the code of a branch-trace adapter for valgrind (outputs all branches taken/not-taken in 'qemu' format). Very useful if you can run a pintool or Intel Processor Trace just for that. And if you keep digging, the angr symbolic execution toolkit use (used?) VEX as an intermediate representation. end of edit Ada doesn't catch uninitialized variables by default (although warnings are getting better). You can either go Spark 'bronze level' (dataflow proof, every variable is initialized) or use 'pragma Initialize_Scalars' combined with -gnatVa. Some of these techniques described in that now old blog post full of links https://blog.adacore.com/running-american-fuzzy-lop-on-your-ada-code https://blog.adacore.com/running-american-fuzzy-lop-on-your-... (shameless plug) where one can infer that even proof of absence of runtime errors isn't a panacea and fuzzing still has its use even on fully-proved SPARK code.
mukundesh 4y ago
Using Cachegrind to get hardware independent performance numbers (https://pythonspeed.com/articles/consistent-benchmarking-in-ci/ https://pythonspeed.com/articles/consistent-benchmarking-in-...) Also used by SQLite in their performance measurement workflow(https://sqlite.org/cpu.html#performance_measurement https://sqlite.org/cpu.html#performance_measurement)
amelius 4y ago
Are people using Valgrind on Python packages? It seems some packages (even basic ones) are not compatible with Valgrind, thereby spoiling the entire debugging experience.
edsiper2 4y ago
First of all congratulations to Valgrind and the team behind it! This is an essential tool that help me personally over the years while developing. What needs to be done to get Valgrind binaries available for MacOS (M1) ?, from a company perspective we are happy to support this work. If you know who's interest and can accomplish this pls drop me an email to eduardo at calyptia dot com.
cjbprime 4y ago
I wish I hadn't read this article because now I know that I've been mispronouncing Valgrind for nearly 20 years but I'm not going to stop. (Kidding. Thanks for Valgrind! I still use it for assessing memory corruption vulnerabilities along with ASan.)
- dtgriscom 4y ago
  I've been promoting proper pronunciation of Valgrind at work, an am making passable progress...
  - quickthrower2 4y ago
    Valarie smiled. Is how I will remember it. That said I sometimes get the "V" tools mixed up (Vagrant, Valgrind, Varnish)
- galangalalgol 4y ago
  Our pipelines have asan ( and cpp check clang tidy coverity and coverage stuff) but no valgrind, is there something it is good at that we are missing?
  - Jason_Gibson 4y ago
    ASAN on its own doesn't detect uninitialized memory. MSAN can, though. Valgrind is also more than just the memcheck sub-tool - there are others, like Cachegrind, which is a cache and branch-prediction profiler. https://github.com/google/sanitizers/wiki/AddressSanitizerComparisonOfMemoryTools https://github.com/google/sanitizers/wiki/AddressSanitizerCo... https://github.com/google/sanitizers/wiki/MemorySanitizer https://github.com/google/sanitizers/wiki/MemorySanitizer https://valgrind.org/docs/manual/manual.html https://valgrind.org/docs/manual/manual.html
    - deleted 4y ago
      [deleted]
  - glouwbug 4y ago
vlmutolo 4y ago
> I still use Cachegrind, Callgrind, and DHAT all the time. I’m amazed that I’m still using Cachegrind today, given that it has hardly changed in twenty years. (I only use it for instruction counts, though. I wouldn’t trust the icache/dcache results at all given that they come from a best-guess simulation of an AMD Athlon circa 2002.) I'm pretty sure I've seen people using the icache/dcache miss counts from valgrind for profiling. I wonder how unreliable these numbers are.
- andrewf 4y ago
  https://sqlite.org/cpu.html#microopt https://sqlite.org/cpu.html#microopt - Cachegrind is used to measure performance because it gives answers that are repeatable to 7 or more significant digits. In comparison, actual (wall-clock) run times are scarcely repeatable beyond one significant digit [...] The high repeatability of cachegrind allows the SQLite developers to implement and measure "microoptimizations". There's a bunch of ways for caches to behave differently but have they changed much over the past 20 years? i.e. is the difference between [2022 AMD cache, 2002 AMD cache] significantly greater than the difference between [2002 PowerPC G4 cache, 2002 AMD cache, 2002 Intel cache] ?
  - tux3 4y ago
    I don't know how sophisticates the streaming/prefetch/access pattern prediction the 2002 cpus did was. I'm speculating, but if that's not modeled, cachegrind may pessimize some less simple predictable patterns and report a lot of expected misses when the cpu would have been able to prefetch it
    - andrewf 4y ago
      Agreed, I suspect it'd be most accurate to say the SQLite folks are minimizing their working set. I picked a couple of random performance commits out of their code repo, and they look like they might keep 1 or 2 lines out of i-cache: https://sqlite.org/src/info/f48bd8f85d86fd93 https://sqlite.org/src/info/f48bd8f85d86fd93 https://sqlite.org/src/info/390717e68800af9b https://sqlite.org/src/info/390717e68800af9b
whimsicalism 4y ago
It's unfortunate that so many of these great tools (like `perf` and I believe `valgrind`) are basically not available locally on the Mac. And running in a container is not really a solution for most of these.
- wyldfire 4y ago
  Sanitizers and electric fence are ultra portable, they're definitely available on macos. The feature set from valgrind is a bit richer but not by much.
  - whimsicalism 4y ago
    I am not familiar with electric fence but I remember from my experience that there are definitely important things that I got from `perf` and `valgrind` that the alternative sanitizers did not provide. Can't recall what now of course.
    - nyanpasu64 4y ago
      asan/ubsan do not detect uninitialized memory reads (though ubsan can detect when bools take on invalid bit patterns from uninitialized memory), and msan requires rebuilding the standard library or something, so I've never used msan. Valgrind is slow, but detects uninitialized memory reads properly, and doesn't require rebuilding the app (which is useful when running a complex or prebuilt app for short periods of time). On the topic of profiling, callgrind can count exact function calls and generate accurate call graphs, which I find useful for not only profiling, but tracing the execution of unfamiliary code. I just wish rr had similarly fast tooling (pernosco is close enough to be useful, but I think there's value in exploring different workflows than what they picked).
      - 1over137 4y ago
        >msan requires rebuilding the standard library or something Yes, which is a PITA. But even then, macOS is not supported anyway: https://clang.llvm.org/docs/MemorySanitizer.html#supported-platforms https://clang.llvm.org/docs/MemorySanitizer.html#supported-p...
appleflaxen 4y ago
What other great tools are there in the vein of valgrind and AFL?
- tux3 4y ago
  rr, for record and replay I'm also a fan of systemtap, for when your probing problems push into peeking at the kernel
- tialaramex 4y ago
  In my obviously biased opinion, very specialised, but sometimes exactly what you needed (I have used this in anger maybe 2-3 times in my career since then, which is why I wrote the C version): https://github.com/tialaramex/leakdice https://github.com/tialaramex/leakdice (or https://github.com/tialaramex/leakdice-rust https://github.com/tialaramex/leakdice-rust) Leakdice implements some of Raymond Chen's "The poor man’s way of identifying memory leaks" for you. On Linux at least. https://bytepointer.com/resources/old_new_thing/20050815_224_the_poor_mans_way_of_identifying_memory_leaks.htm https://bytepointer.com/resources/old_new_thing/20050815_224... All leakdice does is: You pick a running process which you own, leakdice picks a random heap page belonging to that process and shows you that page as hex + ASCII. The Raymond Chen article explains why you might ever want to do this.
- cjbprime 4y ago
  Starting to stretch, but would have to pick strace next. Can't believe macOS devs don't get to use it (at least without hoops like disabling SIP).
- yaantc 4y ago
  Seconding `rr` as suggested by @tux3, it's great for debugging. Also, the sanitizers for GCC and Clang (https://github.com/google/sanitizers https://github.com/google/sanitizers), and the Clang static analyzer (and tidy too) through CodeChecker (https://codechecker.readthedocs.io/ https://codechecker.readthedocs.io/). For the Clang static analyzer, make sure your LLVM toolchain has the Z3 support enabled (OK in Debian stable for example), and enable cross translation units (CTU) analysis too for better results.
compiler-guy 4y ago
I sort of owe callgrind a big chunk of my career. I was working at a company full of PhDs and well seasoned veterans, who looked at me as a new kid, kind of underqualified to be working in their tools group. I had been at the firm for a while, and they were nice enough, but didn't really have me down as someone who was going to contribute as anything other than a very junior engineer. We had a severe problem with a program's performance, and no one really had any idea why. And as it was clearly not a sophisticated project, I got assigned to figure something out. I used the then very new callgrind and the accompanying flamegraph, and discovered that we were passing very large bit arrays for register allocation by value. Very, very large. They had started small enough to fit in registers, but over time had grown so large that a function call to manipulate them effectively flushed the cache, and the rest of the code assumed these operations were cheap. Profiling tools at the time were quite primitive, and the application was a morass of shared libraries, weird dynamic allocations and JIT, and a bunch of other crap. Valgrind was able to get the profiles after failing with everything else I could try. The presentation I made on that discovery, and my proposed fixes (which eventually sped everything up greatly), finally earned the respect of my colleagues, and no phd wasn't a big deal after that. Later on, those colleagues who had left the company invited me to my next gig. And the one after that. So thanks!
- azurezyq 4y ago
  I have a very similar experience, but with a different profiling tool. When I first graduated from school and joined a big internet company, I'm not that "different". The serving stack was all in C++. My colleagues were really capable but not that into "tools", they'd rather depend on themselves (guess, tune, measure). But I, as a fresh member in the team, learned and introduced Google perftools to the team and did a presentation of the breakdown of the running time of the big binary. I have to say that presentation was a life-changing moment in my career. So together with you, I really want to thank those who devoted heavily into building these tools. When I was doing the presentation, I really felt standing on the shoulders of giants and those giants were helping me. And over years, I used more and more tools like valgrind, pahole, asan, tsan. Much appreciated!
- dijonman2 4y ago
deleted 4y ago
[deleted]
Olumde 4y ago
Happy birthday Valgrind. Next year you'll be able to drink in the US! Being a UK PhD holder, a sentence stood out out to me was a commentary/comparison between UK and US PhDs: "This was a three year UK PhD, rather than a brutal six-or-more year US PhD." My cousin has a US PhD and judging from what he tells me. It is a lot more rigorous than UK PhDs.
- not2b 4y ago
  It took me four years for my US PhD, but I had a masters and industrial experience which might have helped speed things up.
- wenc 4y ago
  The UK PhD is 3 yrs, after a 1 yr Masters and 3 yr bachelors. (7 years) The US PhD is usually 4-5 years after a 4 year bachelors (8-9 years). It is a little bit longer with more graduate-level coursework. That said, the US bachelors starts at age 17 while a UK bachelors starts after 2 years of A-levels. So in terms of length it’s a wash.
  - pbhjpbhj 4y ago
    FWIW, you have to be slightly careful as Scotland has a different post-16 education provision. AIUI you can do Highers (equivalent to GCSE, at 16) and enter Uni then with sufficiently high grades (aged 16/17). Or, stay on for one more year to do Advanced Higher (most common). Uni courses can then be 4 or occasionally 3 years. Don't quote me!
  - piker 4y ago
    US college starts around age 18, which I understand is about the time A-levels are completed, so I believe there are 2 more years of education associated with a US PhD.
deleted 4y ago
[deleted]
Linda703 4y ago
[dead]
nneonneo 4y ago
Hah, I teach my students to use Valgrind, and I’ve been pronouncing it wrong this whole time. Guess I’ll have to make sure to get that right next semester :) The magic of Valgrind really lies in its ability to detect errors without recompiling the code. Sure, there’s a performance hit, but sometimes all you have is a binary. It’s damn solid on Linux, and works even with the custom threading library we use for the course; shame the macOS port is barely maintained (last I checked, it only worked on OSes from a few years back - anything more recent will execute syscalls during process startup that Valgrind doesn’t handle).
ssrs 4y ago
ive used valgrind quite extensively. a big thank you to the folks behind this!
sharmin123 4y ago
anewpersonality 4y ago
Is Valgrind any use in Rust?
- pjmlp 4y ago
  Depends how much unsafe code blocks you make use of.
- jackosdev 4y ago
  I work full-time with Rust, use it all the time to see how much memory is being allocated to the heap, make a change and then see if there's a difference, and also for cache misses: valgrind target/debug/rustbinary ==10173== HEAP SUMMARY: ==10173== in use at exit: 854,740 bytes in 175 blocks ==10173== total heap usage: 2,046 allocs, 1,871 frees, 3,072,309 bytes allocated ==10173== ==10173== LEAK SUMMARY: ==10173== definitely lost: 0 bytes in 0 blocks ==10173== indirectly lost: 0 bytes in 0 blocks ==10173== possibly lost: 1,175 bytes in 21 blocks ==10173== still reachable: 853,565 bytes in 154 blocks ==10173== suppressed: 0 bytes in 0 blocks ==10173== Rerun with --leak-check=full to see details of leaked memory valgrind --tool=cachegrind target/debug/rustbinary ==146711== ==146711== I refs: 1,054,791,445 ==146711== I1 misses: 11,038,023 ==146711== LLi misses: 62,896 ==146711== I1 miss rate: 1.05% ==146711== LLi miss rate: 0.01% ==146711== ==146711== D refs: 793,113,817 (368,907,959 rd + 424,205,858 wr) ==146711== D1 misses: 757,883 ( 535,230 rd + 222,653 wr) ==146711== LLd misses: 119,285 ( 49,251 rd + 70,034 wr) ==146711== D1 miss rate: 0.1% ( 0.1% + 0.1% ) ==146711== LLd miss rate: 0.0% ( 0.0% + 0.0% ) ==146711== ==146711== LL refs: 11,795,906 ( 11,573,253 rd + 222,653 wr) ==146711== LL misses: 182,181 ( 112,147 rd + 70,034 wr) ==146711== LL miss rate: 0.0% ( 0.0% + 0.0% )
- rwmj 4y ago
  Not used it with Rust, but have used it with OCaml, Perl, Ruby, Tcl successfully. In managed languages it's mainly useful for detecting problems in C bindings rather than the language itself. Languages where it doesn't work well: Python and Golang.
RustyRussell 4y ago
I once submitted a bug fix for an obscure issue to valgrind. They asked for a test case, which I managed to provide, but I was a bit nervous as I couldn't immediately see how to fit in their test suite. The response from Julian Seward was so nice it set a permanently high bar for me when random people I don't know report bugs on my projects! We still run our entire testsuite under valgrind in CI. Amazing tool!
- sealeck 4y ago
  What was the response?
bayindirh 4y ago
I still use Valgrind memcheck for memory leak verification of a large piece of code I have developed, with a long end-to-end test. Also, it has a nice integration with Eclipse which reflects the Valgrind memcheck output to the source files directly, enabling you to see where problems are rooted. All in all, Valgrind is a great toolset. P.S.: I was pronouncing Valgrind correctly! :)
lma21 4y ago
When we moved to Linux, Valgrind was THE tool that saved our as*s day after day after day. An issue in production? rollback, valgrind, fix, push, repeat. Thank you for all the hard work, in fact i don't i can thank you enough.
Sesse__ 4y ago
I live not far from Valgrindvegen (Valgrind road); I've always wondered whether the developers knew it existed. :-)
tarasglek 4y ago
Beyond raw technical ability, Nick and Julian were the kindest, most reasonable developers I've ever interacted with. I think a lot of Valgrind's success stems from combination of sophisticated tech and approachability of the core team.
j1elo 4y ago
Valgrind is an amazingly useful tool. The biggest pain point, though, has always been to read through and process the huge amount of false positives that typically come from 3rd-party support libraries, such as GLib. It provides some suppression files to be used with Valgrind, but still, GLib has its own memory allocator, so things tend to go awry. Running Helgrind or DRD (for threading issues) with GLib has been a bit frustrating, too. If anyone has some advice to share about this, I'm all ears! (EDIT: I had mistakenly left out the phrase about suppression files)
amelius 4y ago
One problem with Valgrind is that the thing you're debugging should have been tested with Valgrind from the start, otherwise you're just going to be flooded with false triggers. Now imagine that you're developing a new application and you want to use some library, and it hasn't been tested with valgrind and generates tons of false messages. Should you then use it? Or look for an alternative library?
t43562 4y ago
I was working on an application for Symbian mobile phones and I was able to implement large parts of it as a portable library - the bits which compressed results using a dictionary to make them tiny enough to fit into an SMS message or a UDP frame. This was before the days of flat-rate charges for internet access and we were trying to be very economical with data. I was able to build and debug them on Linux with Valgrind finding many stupid mistakes and the library worked flawlessly on Symbian. It's just one of the many times that Valgrind has saved my bacon. It's awesome.
ahartmetz 4y ago
Valgrind is fantastic. Memcheck decreases the memory safety problem of C++ by about 80% in my experience - it really is a big deal. The compiler-based tools that require recompiling every library used are a bit impractical for large stacks such as the ones under Qt-based GUI applications. Several libraries, several build systems. But I hear that they are popular for CI systems in large projects such as web browsers, which probably have dedicated CI developers. There are also some IME rare problems that these tools can find that Memcheck can't, which is due to information unavailable in compiled code. Still, Memcheck has the largest coverage by far. Callgrind and Cachegrind give very precise, repeatable results, complementary to but not replacing perf and AMD / Intel tooling which use hardware performance counters. I tend to use all of them. They all work without recompiling.
randomswede 4y ago
Not sure if it was still doing it in 2001, but in the 1997-1998 time-frame Purify also ran on HP-UX. The company I was working for at the time used it and we ended up finding a two-byte (IIRC) leak in the HP gethostbyname() library call (well, at least I think it was gethostbyname, it's more than two decades ago). That was one of the more annoying tickets to file. We could of course send them the binary, but it would not run without the Purify license file, and we weren't comfortable to send off the license file as well. But, in the end, they accepted the bug. Not sure if there was every any fix, though.
willfiveash 4y ago
I see the article mentions Solaris, an OS that I am very familiar with, which had me thinking about the memory corruption detection Solaris offerred. Among the development features Solaris supported were two memory corruption checking libraries (libumem, watchmalloc) that could easily be used without have to recompile binaries to link with them. Libumem had support for detecting memory leaks, buffer overruns, multiple frees, use of uninitialized data, use of freed data, etc... but it could not detect a read past an allocated buffer which is where watchmalloc came in handy. To use either with an executable binary was as easy as: $ LD_PRELOAD=libumem.so.1 <executable filename> I found a lot of memory corruption bugs using libumem in particular including some in MIT Kerberos that were severe enough to be considered security vulnerabilities. Sadly, Solaris is now in support mode thanks to Ellison and friends at Oracle.

626 points by nnethercote 4y ago

There are times when LeakSanitizer (in gcc-8.2) would not give me the full backtrace of a leak, while valgrind would, so to me it's still an indispensable tool for debugging leaks. One caveat is that it's magnitudes slower than valgrind. Now, if only I know how to make valgrind run as fast as LeakSanitizer... (command line options?)

rigtorp 4y ago
You might need to add -fno-omit-frame-pointer to help ASAN unwind the stack.
- abbeyj 4y ago
  This is definitely an option you want to be using when using ASan or LSan. You may also want to consider additionally using -momit-leaf-frame-pointer to skip frame pointers only in leaf functions while keeping frame pointers for non-leaf functions. This can make small leaf functions significantly shorter, limiting some of the negative impact of using -fno-omit-frame-pointer alone. Sometimes even -fno-omit-frame-pointer won't help, like if the stack is being unwound through a system library that was built without frame pointers. In that case you can switch to the slow unwinder. Set the environment variable `ASAN_OPTIONS=fast_unwind_on_malloc=0` when running your program. But note that this will make most programs run significantly slower so you probably want to use it only when you really need it and not as the default setting for all runs.

mynegation 4y ago

I am old enough that I started with Purify and I used Valgrind starting from the version 1.0, because Purify was commercial and Solaris only. It saved my behind multiple multiple times.

hn_go_brrrrr 4y ago
I worked at a company 11 years ago that was still using Purify!
- unmole 4y ago
  I used Purify 8 years ago. On Windows. I don't remember the specifics but the company kept a few XP machines around just so they could continue using Purify.
atgreen 4y ago
Purify was an amazing tool. I recently noticed that one of my libraries (libffi) still has an --enable-purify configure option, although it probably hasn't been exercised in.. 20 years? A Purify patent prevented work-alikes for many years, but valgrind eventually emerged as a more-than-worthy successor. Fun fact: the creator of Purify went on to found Netflix and is still their CEO.
- mynegation 4y ago
  Ha! And I thought that the same person writing bzip2 and Val grind is my surprise for the day.
  - snovv_crash 4y ago
    And Ardupilot
cpeterso 4y ago
And BoundsChecker was also great! https://en.m.wikipedia.org/wiki/BoundsChecker https://en.m.wikipedia.org/wiki/BoundsChecker
- sumtechguy 4y ago
  That tool saved me tons of time tracking down bugs. It also taught me to be a better C/C++ programmer. Run time sanitizers like Purify/Valgrind/Boundchecker do not tolerate poor C code. What is kind of cool is you can find whole classes of bugs in your code. Because as devs we get something working once we tend to copy and paste that pattern everywhere. So find a bug in one place you will probably find it a few dozen other places in your codebase.

nicoburns 4y ago

Well damn, no wonder he’s so good at optimising the Rust compiler. He literally has a PhD in profiling tools!

gkhartman 4y ago

Many thanks for Valgrind. I can honestly say that it helped me become a better C++ programmer.

junon 4y ago

Valgrind's maintainers are super pleasant and have been quite helpful in a number of cases I've personally had to reach out to them. Lovely piece of software toward which I owe a lot of gratitude.

pjmlp 4y ago

> Speaking of software quality, I think it’s fitting that I now work full time on Rust, a systems programming language that didn’t exist when Valgrind was created, but which basically prevents all the problems that Memcheck detects. Just like Ada has been doing since 1983.

oconnor663 4y ago
My understanding is that dynamically freeing memory is an unsafe operation in Ada, do I have that right?
- pjmlp 4y ago
  Depends on which dynamic memory you are talking about. Ada can manage dynamic stacks, strings and arrays on its own. For example, Ada has what one could call type safe VLAs, instead of corrupting the stack like C, you get an exception and can redo the call with a smaller size, for example. As for explicit heap types and Ada.Unchecked_Deallocation, yes if we are speaking about Ada 83. Ada 95 introduced controlled types, which via Initialize, Adjust, and Finalize, provide the basis of RAII like features in Ada. Here is an example on how to implement smart pointers with controlled types, https://www.adacore.com/gems/gem-97-reference-counting-in-ada-part-1 https://www.adacore.com/gems/gem-97-reference-counting-in-ad... There is also the possiblity to wrap heap allocation primitives with safe interfaces exposed via storage pools, like on this tutorial https://blog.adacore.com/header-storage-pools https://blog.adacore.com/header-storage-pools Finally thanks to SPARK, nowadays integrated into Ada 2012[0], you can also have formal proofs that it is safe to release heap memory. In top of all this, Ada is in the process of integrating affine types as well. [0] - Supported in PTC and GNAT, remaining Ada compilers have a mix of Ada 95 - 2012 features, see https://news.ycombinator.com/item?id=27603292 https://news.ycombinator.com/item?id=27603292
  - touisteur 4y ago
    That said, I still use valgrind because we have to integrate C libraries sometimes (libpcl is my favorite culprit, only because I'm trying , and there's still possibility to blow the stack (yeah you can use gnatstack to get a good idea of your maximum stack size, but it's doesn't cover the whole Ada featureset and stack canaries - fstack-check don't catch everything. edit Also massif, call/cachegrind and hellgrind have saved our bacon many, many times. Even more interesting is writing your own tools with valgrind. Here https://github.com/AdaCore/gnatcoverage/tree/master/tools/gnatcov/trace_adapters/valgrind https://github.com/AdaCore/gnatcoverage/tree/master/tools/gn... is the code of a branch-trace adapter for valgrind (outputs all branches taken/not-taken in 'qemu' format). Very useful if you can run a pintool or Intel Processor Trace just for that. And if you keep digging, the angr symbolic execution toolkit use (used?) VEX as an intermediate representation. end of edit Ada doesn't catch uninitialized variables by default (although warnings are getting better). You can either go Spark 'bronze level' (dataflow proof, every variable is initialized) or use 'pragma Initialize_Scalars' combined with -gnatVa. Some of these techniques described in that now old blog post full of links https://blog.adacore.com/running-american-fuzzy-lop-on-your-ada-code https://blog.adacore.com/running-american-fuzzy-lop-on-your-... (shameless plug) where one can infer that even proof of absence of runtime errors isn't a panacea and fuzzing still has its use even on fully-proved SPARK code.

mukundesh 4y ago

Using Cachegrind to get hardware independent performance numbers (https://pythonspeed.com/articles/consistent-benchmarking-in-ci/ https://pythonspeed.com/articles/consistent-benchmarking-in-...) Also used by SQLite in their performance measurement workflow(https://sqlite.org/cpu.html#performance_measurement https://sqlite.org/cpu.html#performance_measurement)

amelius 4y ago

Are people using Valgrind on Python packages? It seems some packages (even basic ones) are not compatible with Valgrind, thereby spoiling the entire debugging experience.

edsiper2 4y ago

First of all congratulations to Valgrind and the team behind it! This is an essential tool that help me personally over the years while developing. What needs to be done to get Valgrind binaries available for MacOS (M1) ?, from a company perspective we are happy to support this work. If you know who's interest and can accomplish this pls drop me an email to eduardo at calyptia dot com.

cjbprime 4y ago

I wish I hadn't read this article because now I know that I've been mispronouncing Valgrind for nearly 20 years but I'm not going to stop. (Kidding. Thanks for Valgrind! I still use it for assessing memory corruption vulnerabilities along with ASan.)

dtgriscom 4y ago
I've been promoting proper pronunciation of Valgrind at work, an am making passable progress...
- quickthrower2 4y ago
  Valarie smiled. Is how I will remember it. That said I sometimes get the "V" tools mixed up (Vagrant, Valgrind, Varnish)
galangalalgol 4y ago
Our pipelines have asan ( and cpp check clang tidy coverity and coverage stuff) but no valgrind, is there something it is good at that we are missing?
- Jason_Gibson 4y ago
  ASAN on its own doesn't detect uninitialized memory. MSAN can, though. Valgrind is also more than just the memcheck sub-tool - there are others, like Cachegrind, which is a cache and branch-prediction profiler. https://github.com/google/sanitizers/wiki/AddressSanitizerComparisonOfMemoryTools https://github.com/google/sanitizers/wiki/AddressSanitizerCo... https://github.com/google/sanitizers/wiki/MemorySanitizer https://github.com/google/sanitizers/wiki/MemorySanitizer https://valgrind.org/docs/manual/manual.html https://valgrind.org/docs/manual/manual.html
  - deleted 4y ago
    [deleted]
- glouwbug 4y ago

vlmutolo 4y ago

> I still use Cachegrind, Callgrind, and DHAT all the time. I’m amazed that I’m still using Cachegrind today, given that it has hardly changed in twenty years. (I only use it for instruction counts, though. I wouldn’t trust the icache/dcache results at all given that they come from a best-guess simulation of an AMD Athlon circa 2002.) I'm pretty sure I've seen people using the icache/dcache miss counts from valgrind for profiling. I wonder how unreliable these numbers are.

andrewf 4y ago
https://sqlite.org/cpu.html#microopt https://sqlite.org/cpu.html#microopt - Cachegrind is used to measure performance because it gives answers that are repeatable to 7 or more significant digits. In comparison, actual (wall-clock) run times are scarcely repeatable beyond one significant digit [...] The high repeatability of cachegrind allows the SQLite developers to implement and measure "microoptimizations". There's a bunch of ways for caches to behave differently but have they changed much over the past 20 years? i.e. is the difference between [2022 AMD cache, 2002 AMD cache] significantly greater than the difference between [2002 PowerPC G4 cache, 2002 AMD cache, 2002 Intel cache] ?
- tux3 4y ago
  I don't know how sophisticates the streaming/prefetch/access pattern prediction the 2002 cpus did was. I'm speculating, but if that's not modeled, cachegrind may pessimize some less simple predictable patterns and report a lot of expected misses when the cpu would have been able to prefetch it
  - andrewf 4y ago
    Agreed, I suspect it'd be most accurate to say the SQLite folks are minimizing their working set. I picked a couple of random performance commits out of their code repo, and they look like they might keep 1 or 2 lines out of i-cache: https://sqlite.org/src/info/f48bd8f85d86fd93 https://sqlite.org/src/info/f48bd8f85d86fd93 https://sqlite.org/src/info/390717e68800af9b https://sqlite.org/src/info/390717e68800af9b

whimsicalism 4y ago

It's unfortunate that so many of these great tools (like `perf` and I believe `valgrind`) are basically not available locally on the Mac. And running in a container is not really a solution for most of these.

wyldfire 4y ago
Sanitizers and electric fence are ultra portable, they're definitely available on macos. The feature set from valgrind is a bit richer but not by much.
- whimsicalism 4y ago
  I am not familiar with electric fence but I remember from my experience that there are definitely important things that I got from `perf` and `valgrind` that the alternative sanitizers did not provide. Can't recall what now of course.
  - nyanpasu64 4y ago
    asan/ubsan do not detect uninitialized memory reads (though ubsan can detect when bools take on invalid bit patterns from uninitialized memory), and msan requires rebuilding the standard library or something, so I've never used msan. Valgrind is slow, but detects uninitialized memory reads properly, and doesn't require rebuilding the app (which is useful when running a complex or prebuilt app for short periods of time). On the topic of profiling, callgrind can count exact function calls and generate accurate call graphs, which I find useful for not only profiling, but tracing the execution of unfamiliary code. I just wish rr had similarly fast tooling (pernosco is close enough to be useful, but I think there's value in exploring different workflows than what they picked).
    - 1over137 4y ago
      >msan requires rebuilding the standard library or something Yes, which is a PITA. But even then, macOS is not supported anyway: https://clang.llvm.org/docs/MemorySanitizer.html#supported-platforms https://clang.llvm.org/docs/MemorySanitizer.html#supported-p...

appleflaxen 4y ago

What other great tools are there in the vein of valgrind and AFL?

tux3 4y ago
rr, for record and replay I'm also a fan of systemtap, for when your probing problems push into peeking at the kernel
tialaramex 4y ago
In my obviously biased opinion, very specialised, but sometimes exactly what you needed (I have used this in anger maybe 2-3 times in my career since then, which is why I wrote the C version): https://github.com/tialaramex/leakdice https://github.com/tialaramex/leakdice (or https://github.com/tialaramex/leakdice-rust https://github.com/tialaramex/leakdice-rust) Leakdice implements some of Raymond Chen's "The poor man’s way of identifying memory leaks" for you. On Linux at least. https://bytepointer.com/resources/old_new_thing/20050815_224_the_poor_mans_way_of_identifying_memory_leaks.htm https://bytepointer.com/resources/old_new_thing/20050815_224... All leakdice does is: You pick a running process which you own, leakdice picks a random heap page belonging to that process and shows you that page as hex + ASCII. The Raymond Chen article explains why you might ever want to do this.
cjbprime 4y ago
Starting to stretch, but would have to pick strace next. Can't believe macOS devs don't get to use it (at least without hoops like disabling SIP).
yaantc 4y ago
Seconding `rr` as suggested by @tux3, it's great for debugging. Also, the sanitizers for GCC and Clang (https://github.com/google/sanitizers https://github.com/google/sanitizers), and the Clang static analyzer (and tidy too) through CodeChecker (https://codechecker.readthedocs.io/ https://codechecker.readthedocs.io/). For the Clang static analyzer, make sure your LLVM toolchain has the Z3 support enabled (OK in Debian stable for example), and enable cross translation units (CTU) analysis too for better results.

compiler-guy 4y ago

I sort of owe callgrind a big chunk of my career. I was working at a company full of PhDs and well seasoned veterans, who looked at me as a new kid, kind of underqualified to be working in their tools group. I had been at the firm for a while, and they were nice enough, but didn't really have me down as someone who was going to contribute as anything other than a very junior engineer. We had a severe problem with a program's performance, and no one really had any idea why. And as it was clearly not a sophisticated project, I got assigned to figure something out. I used the then very new callgrind and the accompanying flamegraph, and discovered that we were passing very large bit arrays for register allocation by value. Very, very large. They had started small enough to fit in registers, but over time had grown so large that a function call to manipulate them effectively flushed the cache, and the rest of the code assumed these operations were cheap. Profiling tools at the time were quite primitive, and the application was a morass of shared libraries, weird dynamic allocations and JIT, and a bunch of other crap. Valgrind was able to get the profiles after failing with everything else I could try. The presentation I made on that discovery, and my proposed fixes (which eventually sped everything up greatly), finally earned the respect of my colleagues, and no phd wasn't a big deal after that. Later on, those colleagues who had left the company invited me to my next gig. And the one after that. So thanks!

azurezyq 4y ago
I have a very similar experience, but with a different profiling tool. When I first graduated from school and joined a big internet company, I'm not that "different". The serving stack was all in C++. My colleagues were really capable but not that into "tools", they'd rather depend on themselves (guess, tune, measure). But I, as a fresh member in the team, learned and introduced Google perftools to the team and did a presentation of the breakdown of the running time of the big binary. I have to say that presentation was a life-changing moment in my career. So together with you, I really want to thank those who devoted heavily into building these tools. When I was doing the presentation, I really felt standing on the shoulders of giants and those giants were helping me. And over years, I used more and more tools like valgrind, pahole, asan, tsan. Much appreciated!
dijonman2 4y ago

deleted 4y ago

[deleted]

Olumde 4y ago

Happy birthday Valgrind. Next year you'll be able to drink in the US! Being a UK PhD holder, a sentence stood out out to me was a commentary/comparison between UK and US PhDs: "This was a three year UK PhD, rather than a brutal six-or-more year US PhD." My cousin has a US PhD and judging from what he tells me. It is a lot more rigorous than UK PhDs.

not2b 4y ago
It took me four years for my US PhD, but I had a masters and industrial experience which might have helped speed things up.
wenc 4y ago
The UK PhD is 3 yrs, after a 1 yr Masters and 3 yr bachelors. (7 years) The US PhD is usually 4-5 years after a 4 year bachelors (8-9 years). It is a little bit longer with more graduate-level coursework. That said, the US bachelors starts at age 17 while a UK bachelors starts after 2 years of A-levels. So in terms of length it’s a wash.
- pbhjpbhj 4y ago
  FWIW, you have to be slightly careful as Scotland has a different post-16 education provision. AIUI you can do Highers (equivalent to GCSE, at 16) and enter Uni then with sufficiently high grades (aged 16/17). Or, stay on for one more year to do Advanced Higher (most common). Uni courses can then be 4 or occasionally 3 years. Don't quote me!
- piker 4y ago
  US college starts around age 18, which I understand is about the time A-levels are completed, so I believe there are 2 more years of education associated with a US PhD.

deleted 4y ago

[deleted]

Linda703 4y ago

[dead]

nneonneo 4y ago

Hah, I teach my students to use Valgrind, and I’ve been pronouncing it wrong this whole time. Guess I’ll have to make sure to get that right next semester :) The magic of Valgrind really lies in its ability to detect errors without recompiling the code. Sure, there’s a performance hit, but sometimes all you have is a binary. It’s damn solid on Linux, and works even with the custom threading library we use for the course; shame the macOS port is barely maintained (last I checked, it only worked on OSes from a few years back - anything more recent will execute syscalls during process startup that Valgrind doesn’t handle).

ssrs 4y ago

ive used valgrind quite extensively. a big thank you to the folks behind this!

sharmin123 4y ago

anewpersonality 4y ago

Is Valgrind any use in Rust?

pjmlp 4y ago
Depends how much unsafe code blocks you make use of.
jackosdev 4y ago
I work full-time with Rust, use it all the time to see how much memory is being allocated to the heap, make a change and then see if there's a difference, and also for cache misses: valgrind target/debug/rustbinary ==10173== HEAP SUMMARY: ==10173== in use at exit: 854,740 bytes in 175 blocks ==10173== total heap usage: 2,046 allocs, 1,871 frees, 3,072,309 bytes allocated ==10173== ==10173== LEAK SUMMARY: ==10173== definitely lost: 0 bytes in 0 blocks ==10173== indirectly lost: 0 bytes in 0 blocks ==10173== possibly lost: 1,175 bytes in 21 blocks ==10173== still reachable: 853,565 bytes in 154 blocks ==10173== suppressed: 0 bytes in 0 blocks ==10173== Rerun with --leak-check=full to see details of leaked memory valgrind --tool=cachegrind target/debug/rustbinary ==146711== ==146711== I refs: 1,054,791,445 ==146711== I1 misses: 11,038,023 ==146711== LLi misses: 62,896 ==146711== I1 miss rate: 1.05% ==146711== LLi miss rate: 0.01% ==146711== ==146711== D refs: 793,113,817 (368,907,959 rd + 424,205,858 wr) ==146711== D1 misses: 757,883 ( 535,230 rd + 222,653 wr) ==146711== LLd misses: 119,285 ( 49,251 rd + 70,034 wr) ==146711== D1 miss rate: 0.1% ( 0.1% + 0.1% ) ==146711== LLd miss rate: 0.0% ( 0.0% + 0.0% ) ==146711== ==146711== LL refs: 11,795,906 ( 11,573,253 rd + 222,653 wr) ==146711== LL misses: 182,181 ( 112,147 rd + 70,034 wr) ==146711== LL miss rate: 0.0% ( 0.0% + 0.0% )
rwmj 4y ago
Not used it with Rust, but have used it with OCaml, Perl, Ruby, Tcl successfully. In managed languages it's mainly useful for detecting problems in C bindings rather than the language itself. Languages where it doesn't work well: Python and Golang.

RustyRussell 4y ago

I once submitted a bug fix for an obscure issue to valgrind. They asked for a test case, which I managed to provide, but I was a bit nervous as I couldn't immediately see how to fit in their test suite. The response from Julian Seward was so nice it set a permanently high bar for me when random people I don't know report bugs on my projects! We still run our entire testsuite under valgrind in CI. Amazing tool!

sealeck 4y ago
What was the response?

bayindirh 4y ago

I still use Valgrind memcheck for memory leak verification of a large piece of code I have developed, with a long end-to-end test. Also, it has a nice integration with Eclipse which reflects the Valgrind memcheck output to the source files directly, enabling you to see where problems are rooted. All in all, Valgrind is a great toolset. P.S.: I was pronouncing Valgrind correctly! :)

lma21 4y ago

When we moved to Linux, Valgrind was THE tool that saved our as*s day after day after day. An issue in production? rollback, valgrind, fix, push, repeat. Thank you for all the hard work, in fact i don't i can thank you enough.

Sesse__ 4y ago

I live not far from Valgrindvegen (Valgrind road); I've always wondered whether the developers knew it existed. :-)

tarasglek 4y ago

Beyond raw technical ability, Nick and Julian were the kindest, most reasonable developers I've ever interacted with. I think a lot of Valgrind's success stems from combination of sophisticated tech and approachability of the core team.

j1elo 4y ago

Valgrind is an amazingly useful tool. The biggest pain point, though, has always been to read through and process the huge amount of false positives that typically come from 3rd-party support libraries, such as GLib. It provides some suppression files to be used with Valgrind, but still, GLib has its own memory allocator, so things tend to go awry. Running Helgrind or DRD (for threading issues) with GLib has been a bit frustrating, too. If anyone has some advice to share about this, I'm all ears! (EDIT: I had mistakenly left out the phrase about suppression files)

amelius 4y ago

One problem with Valgrind is that the thing you're debugging should have been tested with Valgrind from the start, otherwise you're just going to be flooded with false triggers. Now imagine that you're developing a new application and you want to use some library, and it hasn't been tested with valgrind and generates tons of false messages. Should you then use it? Or look for an alternative library?

t43562 4y ago

I was working on an application for Symbian mobile phones and I was able to implement large parts of it as a portable library - the bits which compressed results using a dictionary to make them tiny enough to fit into an SMS message or a UDP frame. This was before the days of flat-rate charges for internet access and we were trying to be very economical with data. I was able to build and debug them on Linux with Valgrind finding many stupid mistakes and the library worked flawlessly on Symbian. It's just one of the many times that Valgrind has saved my bacon. It's awesome.

ahartmetz 4y ago

Valgrind is fantastic. Memcheck decreases the memory safety problem of C++ by about 80% in my experience - it really is a big deal. The compiler-based tools that require recompiling every library used are a bit impractical for large stacks such as the ones under Qt-based GUI applications. Several libraries, several build systems. But I hear that they are popular for CI systems in large projects such as web browsers, which probably have dedicated CI developers. There are also some IME rare problems that these tools can find that Memcheck can't, which is due to information unavailable in compiled code. Still, Memcheck has the largest coverage by far. Callgrind and Cachegrind give very precise, repeatable results, complementary to but not replacing perf and AMD / Intel tooling which use hardware performance counters. I tend to use all of them. They all work without recompiling.

randomswede 4y ago

Not sure if it was still doing it in 2001, but in the 1997-1998 time-frame Purify also ran on HP-UX. The company I was working for at the time used it and we ended up finding a two-byte (IIRC) leak in the HP gethostbyname() library call (well, at least I think it was gethostbyname, it's more than two decades ago). That was one of the more annoying tickets to file. We could of course send them the binary, but it would not run without the Purify license file, and we weren't comfortable to send off the license file as well. But, in the end, they accepted the bug. Not sure if there was every any fix, though.

willfiveash 4y ago

I see the article mentions Solaris, an OS that I am very familiar with, which had me thinking about the memory corruption detection Solaris offerred. Among the development features Solaris supported were two memory corruption checking libraries (libumem, watchmalloc) that could easily be used without have to recompile binaries to link with them. Libumem had support for detecting memory leaks, buffer overruns, multiple frees, use of uninitialized data, use of freed data, etc... but it could not detect a read past an allocated buffer which is where watchmalloc came in handy. To use either with an executable binary was as easy as: $ LD_PRELOAD=libumem.so.1 <executable filename> I found a lot of memory corruption bugs using libumem in particular including some in MIT Kerberos that were severe enough to be considered security vulnerabilities. Sadly, Solaris is now in support mode thanks to Ellison and friends at Oracle.