10 ms·

Just curious, what were the tradeoffs for profiling the Python code and rewriting slow parts in C vs total rewrite in Go? For high level languages, the usual a

Just curious, what were the tradeoffs for profiling the Python code and rewriting slow parts in C vs total rewrite in Go? For high level languages, the usual argument has been to rewrite just the slow parts in C or some other low level language.

codygman 14y ago
One thing you don't get in the C/Python combo is total static typing. The other big win of Go is elimination of a lot of runtime errors. So it comes down to which will be more of a win for you and your product. Go has a big advantage here IMO though because of static typing AND performance increase.
- pjmlp 14y ago
  Like any other language with static typing and available compiler implementations.
  - codygman 14y ago
    Is that relevant though? The parent is asking about a comparison between python+c and Go for the OP's software, so doesn't it make sense to answer to that? Of course, you are correct. Rewriting in any other language with static typing or an available compiler implementation would also provide the benefits of static typing and detection of runtime errors.
    - pjmlp 14y ago
      Because those benefits are not some special feature of Go. On the other hand I think youth generation of programmers lack exposition to programming languages, like we used to have in the old days, hence they make quite limited comparisons.
waterside81 14y ago
And that's what we did during our first pass: identify the slow parts and port them to C. This gave us about 30% better performance. Without getting too much into the nitty gritty of our specific case, this wasn't enough. We needed some massive speed improvements, I'm talking in the order of 100X faster. The nature of our algorithms was such that they could be done in parallel (i.e map/reduce) - an ideal candidate for Go's goroutines. We actually tried to make it parallelizable at first in Python, using gevent (and even just multiprocessing) and the results were not great. One other aspect that really guided us towards Go was memory usage. Python was just sucking up so much memory whereas our Go implementation thus far is so much thinner.
- danieldk 14y ago
  One other aspect that really guided us towards Go was memory usage. Python was just sucking up so much memory whereas our Go implementation thus far is so much thinner. It's funny that you mention that, because in computationally intensive work memory management is an issue with Go. E.g. I wrote a maximum entropy parameter estimator in Go, which was terribly slow until I circumvented the garbage collector by preallocating a huge block of memory and doing my own management. In C malloc() and free()-ing had practically no overhead. After putting the Go garbage collector out of the game, the Go version was approximately within 2x of the C version. I am interested how Go gave you one or two orders of magnitude speedup, while rewriting hot spots in C didn't...
  - Ziomislaw 14y ago
    > I am interested how Go gave you one or two orders of magnitude speedup, while rewriting hot spots in C didn't... paralellization
    - danieldk 14y ago
      One can easily do that in C with OpenMP.
      - Symmetry 14y ago

codygman 14y ago

One thing you don't get in the C/Python combo is total static typing. The other big win of Go is elimination of a lot of runtime errors. So it comes down to which will be more of a win for you and your product. Go has a big advantage here IMO though because of static typing AND performance increase.

pjmlp 14y ago
Like any other language with static typing and available compiler implementations.
- codygman 14y ago
  Is that relevant though? The parent is asking about a comparison between python+c and Go for the OP's software, so doesn't it make sense to answer to that? Of course, you are correct. Rewriting in any other language with static typing or an available compiler implementation would also provide the benefits of static typing and detection of runtime errors.
  - pjmlp 14y ago
    Because those benefits are not some special feature of Go. On the other hand I think youth generation of programmers lack exposition to programming languages, like we used to have in the old days, hence they make quite limited comparisons.

waterside81 14y ago

And that's what we did during our first pass: identify the slow parts and port them to C. This gave us about 30% better performance. Without getting too much into the nitty gritty of our specific case, this wasn't enough. We needed some massive speed improvements, I'm talking in the order of 100X faster. The nature of our algorithms was such that they could be done in parallel (i.e map/reduce) - an ideal candidate for Go's goroutines. We actually tried to make it parallelizable at first in Python, using gevent (and even just multiprocessing) and the results were not great. One other aspect that really guided us towards Go was memory usage. Python was just sucking up so much memory whereas our Go implementation thus far is so much thinner.

danieldk 14y ago
One other aspect that really guided us towards Go was memory usage. Python was just sucking up so much memory whereas our Go implementation thus far is so much thinner. It's funny that you mention that, because in computationally intensive work memory management is an issue with Go. E.g. I wrote a maximum entropy parameter estimator in Go, which was terribly slow until I circumvented the garbage collector by preallocating a huge block of memory and doing my own management. In C malloc() and free()-ing had practically no overhead. After putting the Go garbage collector out of the game, the Go version was approximately within 2x of the C version. I am interested how Go gave you one or two orders of magnitude speedup, while rewriting hot spots in C didn't...
- Ziomislaw 14y ago
  > I am interested how Go gave you one or two orders of magnitude speedup, while rewriting hot spots in C didn't... paralellization
  - danieldk 14y ago
    One can easily do that in C with OpenMP.
    - Symmetry 14y ago