felixge

9 ms·

1. 1.▲
   by felixge 6mo ago

   We had reached out to y'all last year to explore taking ideas from your format. It definitely looks interesting! But IIRC nobody from your team ended up making it to one of our SIG meetings? https://github.com/yandex&#x
2. 2.▲
   by felixge 6mo ago

   OTel Profiling SIG maintainer here: I understand your concern, but we’ve tried our best to make things efficient across the protocol and all involved components. Please let us know if you find any issues with what we are shipping right now.
3. 3.▲
   Optimizing Ruby performance: Observations from real-world services(datadoghq.com)

   3 points by felixge 10mo ago|1 comments
4. 4.▲
   by felixge 1y ago

   Do you think the GC roots alone (goroutine stacks with goroutine id, package globals) would be enough? I think in many cases you'd want the reference chains. The GC could certainly keep track of those, but at the expense of making thin
5. 5.▲
   by felixge 1y ago

   I'd love to hear more! What kind of profiling issues are you running into? I'm assuming the inuse memory profiles are sometimes not good enough to track down leaks since they only show the allocation stack traces? Have you tried g
6. 6.▲
   by felixge 1y ago

   Hacking into the Go runtime with eBPF is definitely fun. But for a more long term solution in terms of reliability and overhead, it might be worth raising this as a feature request for the Go runtime itself. Type information could be provid
7. 7.▲
   by felixge 1y ago

   +1. In particular []byte slice allocations are often a significant driver of GC pace while also being relatively easy to optimize (e.g. via sync.Pool reuse).
8. 8.▲
   Don't Clobber the Frame Pointer(nsrip.com)

   77 points by felixge 2y ago|42 comments
9. 9.▲
   Orchestrion: Compile-time auto-instrumentation for Go(datadoghq.com)

   3 points by felixge 2y ago|0 comments
10. 10.▲
    We used Datadog to save $17.5M annually(datadoghq.com)

    3 points by felixge 2y ago|0 comments
11. 11.▲
    by felixge 2y ago

    This is great, thank you.
12. 12.▲
    The One-Instruction Window(nsrip.com)

    2 points by felixge 2y ago|0 comments
13. 13.▲
    by felixge 2y ago

    I'm skeptical that it's worth it myself, this was just a fun research project for me. But once hardware shadow stacks are available, I think this could be great. To answer your first question: For most Go applications, the average
14. 14.▲
    by felixge 2y ago

    That seems to be windows only? My main target OS is Linux.
15. 15.▲
    by felixge 2y ago

    Thank you so much, this is very helpful and interesting. I'll try to experiment with this at some point.
16. 16.▲
    by felixge 2y ago

    Thanks for the reply! What does the API for accessing the shadow stack from user space look like? I didn't see anything for it in the kernel docs [1]. I agree about the need for switching the shadow stacks in the Go scheduler. But this
17. 17.▲
    by felixge 2y ago

    I don't think any obvious 10%+ opportunities have been overlooked. Go is optimizing for fast and simple builds, which is a bit at odds with optimal code gen. So I think the biggest opportunity is to use Go implementations that are base
18. 18.▲
    by felixge 2y ago

    That's what hardware shadow stacks in modern intel/arm CPUs can do! It just needs to be exposed to user space and become widely available.
19. 19.▲
    by felixge 2y ago

    I know that at least two engineers from the runtime team have seen the post in the #darkarts channel of gopher slack. One of them left a fire emoji :). I'll probably bring it up in the by-weekly Go runtime diagnostics sync [1] next Thu
20. 20.▲
    by felixge 2y ago

    Thanks! And to answer you question: No, it won't speed up Go programs for now. This was mostly a fun research project for me. The low hanging fruits to speed up stack unwinding in the Go runtime is to switch to frame pointer unwinding
21. 21.▲
    by felixge 2y ago

    OP here, happy to answer any question.
22. 22.▲
    by felixge 2y ago

    Thanks for the reply. I ended up implementing this idea in Go and wrote a blog post about the results: https://blog.felixge.de/blazingly-fast-shadow-stacks-for-go/ I'm curious if you've done any benchmarking
23. 23.▲
    Blazingly Fast Shadow Stacks for Go(blog.felixge.de)

    3 points by felixge 2y ago|0 comments
24. 24.▲
    by felixge 2y ago

    Looking at the code [1] it seems like the library is actively trying to handle this problem. [1] https://github.com/koute/not-perf/blob/master/nwind/src/loca...
25. 25.▲
    by felixge 2y ago

    Dynamic patching of return addresses is a very cool trick. I don't think I've seen this before. Have you run into any situations where this crashes programs or otherwise interferes with their execution?
26. 26.▲
    Save up to 14 percent CPU with continuous profile-guided optimization for Go(datadoghq.com)

    1 points by felixge 2y ago|0 comments
27. 27.▲
    by felixge 2y ago

    This sounds like it was copied from PostreSQL which SQLite cites as a strong source of inspiration.
28. 28.▲
    by felixge 2y ago

    I wouldn't be so sure. ByteDance (TikTok) implemented a moving GC for Go: https://youtu.be/DpQgJ06ZjGc?si=FviWDGa7HhBRBrO1
29. 29.▲
    by felixge 2y ago

    I think e-bikes are a good solution for this problem.
30. 30.▲
    by felixge 3y ago

    Great comments, thanks for sharing. The non-atomic frame setup is indeed problematic for CPU profilers, but it's not an issue for allocation profiling, Off-CPU profiling or other types off non-interrupt driven profiling. But as you men