5 ms·

For anyone looking for more examples of 1BRC in Go, we had a friendly competition at work and collected the results here: https://github.com/dhartunian/1brcgo/

For anyone looking for more examples of 1BRC in Go, we had a friendly competition at work and collected the results here: https://github.com/dhartunian/1brcgo/ https://github.com/dhartunian/1brcgo/ In addition to the loop-unrolling and bit-twiddling tricks that also show up in the fastest Java and C++ versions, some Go-specific things I learned were: - unsafe.Pointer can be used to read memory without bounds checks - many functions in the bytes and bits packages in the standard library are written in assembly - debug.SetGCPercent and SetMemoryLimit to turn off GC - runtime.LockOSThread to lock a goroutine to a thread - print is slightly faster than fmt.Printf (but writes to stderr)

benhoyt 3y ago
Oh, I'd missed those solutions, thanks. You guys got way more hard core than I did -- nice work! Looking forward to reading the code for those solutions this week. Update: for reference, Jason Chu's solution (https://github.com/dhartunian/1brcgo/blob/494eabd6ea958cc193648fe34859ac53d86c8743/jason.go https://github.com/dhartunian/1brcgo/blob/494eabd6ea958cc193...) seems to be the fastest on my machine, and runs in about 1.3s!
- michae2 3y ago
  I think we all ended up using unsafe, though there were some solutions without mmap. It would have been interesting if we had adhered to the same constraints you did!
- markoman 3y ago
  Could you say why you find using memory-mapped files to be a portability issue? Thanks.
  - benhoyt 3y ago
    Well, I guess it's more that the standard library doesn't have a cross-platform way to access them, not that memory-mapped files themselves can't be done on (say) Windows. It looks like there's a fairly popular 3rd party package that supports at least Linux, macOS, and Windows: https://github.com/edsrzf/mmap-go https://github.com/edsrzf/mmap-go
  - WesolyKubeczek 3y ago
    Not all filesystems support memory-mapped files equally, and for some that do, the support comes with caveats and could be slower than non-memory-mapped access.
Exuma 3y ago
Damn, that is fine work. I know nothing and am humbled

benhoyt 3y ago

Oh, I'd missed those solutions, thanks. You guys got way more hard core than I did -- nice work! Looking forward to reading the code for those solutions this week. Update: for reference, Jason Chu's solution (https://github.com/dhartunian/1brcgo/blob/494eabd6ea958cc193648fe34859ac53d86c8743/jason.go https://github.com/dhartunian/1brcgo/blob/494eabd6ea958cc193...) seems to be the fastest on my machine, and runs in about 1.3s!

michae2 3y ago
I think we all ended up using unsafe, though there were some solutions without mmap. It would have been interesting if we had adhered to the same constraints you did!
markoman 3y ago
Could you say why you find using memory-mapped files to be a portability issue? Thanks.
- benhoyt 3y ago
  Well, I guess it's more that the standard library doesn't have a cross-platform way to access them, not that memory-mapped files themselves can't be done on (say) Windows. It looks like there's a fairly popular 3rd party package that supports at least Linux, macOS, and Windows: https://github.com/edsrzf/mmap-go https://github.com/edsrzf/mmap-go
- WesolyKubeczek 3y ago
  Not all filesystems support memory-mapped files equally, and for some that do, the support comes with caveats and could be slower than non-memory-mapped access.

Exuma 3y ago

Damn, that is fine work. I know nothing and am humbled