8 ms·

It's worth noting that it's much less of a problem in Python due to the lack of ergonomic closures/lambdas. You have to construct rather esoteric looking code f

It's worth noting that it's much less of a problem in Python due to the lack of ergonomic closures/lambdas. You have to construct rather esoteric looking code for it to be a problem. add_n = [] for n in range(10): add_n.append(lambda x: x + n) add_n[9](10) # 19 add_n[0](10) # 19 This isn't to say it's *not* a footgun (and it has bit me in Python before), but it's much worse in Go due to the idiomatic use of goroutines in a loop: for i := 0; i < 10; i++ { go func() { fmt.Printf("num: %d\n", i) }() }

hinkley 3y ago
Everyone else solved this problem by using list comprehensions instead. Rob has surely heard of those.
- lmm 3y ago
  Of the two comprehension syntaxes in Haskell, Python picked the wrong one. Do notation (or, equivalently, Scala-style for/yield) feels much more consistent and easy to use - in particular the clauses are in the same order as a regular for loop, rather than the middle-endian order used by list comprehensions.
  - eru 3y ago
    Haskell has both do-notation and list comprehension. Comprehension in both Python and Haskell (for both lists and other structures) use the same order in both language, as far as I remember.
    - lmm 3y ago
      > Haskell has both do-notation and list comprehension. Right, and do-notation is the one everyone uses, because it's better. Python picked the wrong one. > Comprehension in both Python and Haskell (for both lists and other structures) use the same order in both language, as far as I remember. It may be the same order as Haskell but it's a terrible confusing order. In particular if you want to go from a nested list comprehension to a flat one (or vice versa) then you have to completely rearrange the order it's written in, whereas if you go from nested do-blocks to flat do-blocks then it all makes sense.
      - eru 3y ago
        I see what you mean, but I don't find the order that confusing in neither Haskell or Python. However, I can imagine a feature that we could add to Python to fix this: make it possible for statements to have a value. Perhaps something like this: my_generator = \ for i in "abc": for b in range(3): print("foo") yield (i, b) or perhaps have the last statement in block be its value (just like Rust or Ruby or Haskell do with the last statement in a block), and make the value of a for-loop be a generator of the individual values: my_list = list( for i in "abc": for b in range(3): (i, b)) Though there's a bit of confusion here whether the latter examples should be a flat structure or a nested one. You could probably use a similiar mechanism as the existing 'yield from' to explicitly ask for the flat version, and otherwise get the nested one: my_list = list( for i in "abc": yield from for b in range(3): (i, b)) Making Python statements have values looks to me like the more generally useful change than tweaking comprehensions. You'd probably not need comprehension at all in that case. Especially since you can already write loop header and body on a single line like for i in range(10): print(i) if they are short enough.
        nerdponx 3y ago
        for i in range(10): print(i) But what would that return? [None] * 10? The limited whitespace-based syntax limits the potential for fun inline statement things, but it also completely dodges the question of what any particular statement should evaluate to when used as an expression.
        eru 3y ago
        > But what would that return? [None] * 10? Yes, I guess something like that. That was just meant as an example of how existing Python allows you to write loops on one line. It's not a good example for a meaningful comprehension in our alternative made-up Python dialect. > The limited whitespace-based syntax limits the potential for fun inline statement things, [...] Python already mostly allows you to use parens to override the indentation. They would just need to generalise that a bit. Btw, Haskell already does that: Officially, Haskell has a syntax with curly braces and semicolons; and they define the indentation based syntax as syntactic sugar that desugars to ; and {}. But almost everyone uses indentation based syntax. The exception are perhaps code generators and when posting on a website that messes with indentation. (And, because it's Haskell, the {}; syntax is just another layer of syntactic sugar for 'weird-operator'-based based syntax like >>=.)
- panzi 3y ago
  How does list comprehension change anything here? This has the same problem: add_n = [lambda x: x + n for n in range(10)] add_n[9](10) # 19 add_n[0](10) # 19
  - billyjmc 3y ago
    I’m not sure what they mean by list comprehensions, either, but for completeness’s sake, I must point out that this is solvable by adding `n` as a keyword argument defaulting to `n`: add_n = [lambda x, n=n: x + n for n in range(10)] add_n[9](10) # 19 add_n[0](10) # 10
    - drekipus 3y ago
      This is the way Also pylsp warns you about this
- deleted 3y ago
  [deleted]
jshen 3y ago
Doesn’t go vet complain about your code? I’m not at my computer right now so can’t check.
- _ikke_ 3y ago
  > Tools have been written to identify these mistakes, but it is hard to analyze whether references to a variable outlive its iteration or not. These tools must choose between false negatives and false positives. The loopclosure analyzer used by go vet and gopls opts for false negatives, only reporting when it is sure there is a problem but missing others. So it will warn in certain situations, but not all of them
- simiones 3y ago
  Why would it? It's perfectly correct code, it's just not doing what you'd expect. It might complain about the race condition, to be fair, but the same issue can be reproduced without goroutines and it would be completely correct code per the semantics.
  - jshen 3y ago
    In many languages "if x = 3" is perfectly valid code, but almost certainly not what the person intended "if x == 3". It's very smart to warn someone in a scenario like this.
    - simiones 3y ago
      I don't really write C too much, but I thought `if err = functionWithErrorReturn() { handleError(err) }` was a somewhat common idiom.
      - defrost 3y ago
        It's a common enough idiom from "stone age" bare bones K&R C, absolutely. It's also one of the great foot-guns of C programming as there are so many other almost but not that idioms and it's never clear on casual inspection whether the result of an assignment was meant to be examined or the result of a comparison. With the evolution of C and C sanity tools that rightfully flag such statements for double checking and the desire to not have spurious flagging, etc. it's more common in later C code to see (say) if ((err = someFunction()) != NOERROR) { errorHandle(err) } that optimises down to the same intermediate code where NOERROR is 0, sure, but it makes it very clear what is going on, an intended assignment and then an intended comparison. As with all idoms the general practice in the larger codebase and house code standard rules apply - there are other ways of doing similar things.
Spivak 3y ago
Ignoring the strange nature of this code in the first place the more pythonic way to do it would be from functools import partial from operators import add add_n = [partial(add, n)) for n in range(10)] assert add_n[5](4) == 9 Look ma, no closures.
- eru 3y ago
  'partial' creates a closure for you.
  - Spivak 3y ago
    Unless you're talking philosophically how classes and closures are actually isomorphic then no, it doesn't. None of the variables in the outer scope are captured in the class instance. https://github.com/python/cpython/blob/main/Lib/functools.py#L276 https://github.com/python/cpython/blob/main/Lib/functools.py... Here's a simplified version of that code that demonstrates the pattern. class partial: def __init__(self, func, *args, **kwargs): self.func = func self.args = args self.kwargs = kwargs def __call__(self, *args, **kwargs): return self.func(*self.args, *args, **(self.kwargs | kwargs)) p = partial(add, 5) # -> an instance of partial with self.args = (5,) res = p(4) # -> calls __call__ which merges the args and calls add(5, 4)
    - eru 3y ago
      I was talking 'philosophically' in that sense. The partial object does create a new scope that binds a few of those variables. But you are also right that the mechanisms in Python are different (on some suitable mid-level of abstraction) for those two.
o11c 3y ago
It's actually worse in Python since there's no support for variable lifetimes within a function, so the `v2` workaround is still broken. (the default-argument workaround "works" but is scary) This makes it clear: the underlying problem is NOT about for loops - it's closures that are broken.
- nerdponx 3y ago
  It's not broken, it's a different design. Maybe worse in a lot of cases, but it's not broken. It's working as intended.
  - eru 3y ago
    You could say the design is broken, but the implementation is working as intended by the design.
Frotag 3y ago
I've run into this once. IIRC the workaround was to add a n=n arg to the lambda
eru 3y ago
In Python you are much more likely to hit that problem not with closures constructed with an explicit 'lambda', but with generator-comprehension expressions. (((i, j) for i in "abc") for j in range(3)) The values of the above depends on in which order you evaluate the whole thing. (Do take what I wrote with a grain of salt. Either the above is already a problem, or perhaps you also need to mix in list-comprehension expressions, too, to surface the bug.)
- nerdponx 3y ago
  Yeah, this one is weird: gs1 = (((i, j) for i in "abc") for j in range(3)) gs2 = [((i, j) for i in "abc") for j in range(3)] print(list(map(list, gs1))) print(list(map(list, gs2))) Results: [[('a', 0), ('b', 0), ('c', 0)], [('a', 1), ('b', 1), ('c', 1)], [('a', 2), ('b', 2), ('c', 2)]] [[('a', 2), ('b', 2), ('c', 2)], [('a', 2), ('b', 2), ('c', 2)], [('a', 2), ('b', 2), ('c', 2)]] That's a nice "wat" right there. I believe the explanation is that in gs2, the range() is iterated through immediately, so j is always set to 2 before you have a chance to access any of the inner generators. Whereas in gs1 the range() is still being iterated over as you access each inner generator, so when you access the first generator j=1, then j=2, etc. Equivalents: def make_gs1(): for j in range(2): yield ((i, j) for i in "abc") def make_gs2(): gs = [] for j in range(2): gs.append(((i, j) for i in "abc")) return gs Late binding applies in both cases of course, but in the first case it doesn't matter, whereas in the latter case it matters. I think early binding would produce the same result in both cases.
  - _ZeD_ 3y ago
    or you could just be eager and use lists: >>> [[(i, j) for i in "abc"] for j in range(3)] [[('a', 0), ('b', 0), ('c', 0)], [('a', 1), ('b', 1), ('c', 1)], [('a', 2), ('b', 2), ('c', 2)]]
    - nerdponx 3y ago
      Right, creating generators in a loop is not usually something you want to do, but it's meant to demonstrate the complexity that arises from late binding rather than demonstrate something you would actually want to do in a real program.
simiones 3y ago
I don't think anyone is puzzled by the Go snippet being wrong. The bigger problem in Go is the for with range loop: pointersToV := make([]*val, len(values)) for i, v := range values { go func() { fmt.Printf("num: %v\n", v) } () //race condition pointersToV[i] = &v //will contain len(values) copies of a pointer to the last item in values } This is the one they are changing. Edit: it looks like they're actually changing both of these, which is more unexpected to me. I think the C# behavior makes more sense, where only the foreach loop has a new binding of the variable in each iteration, but the normal for loop has a single lifetime.

hinkley 3y ago

Everyone else solved this problem by using list comprehensions instead. Rob has surely heard of those.

lmm 3y ago
Of the two comprehension syntaxes in Haskell, Python picked the wrong one. Do notation (or, equivalently, Scala-style for/yield) feels much more consistent and easy to use - in particular the clauses are in the same order as a regular for loop, rather than the middle-endian order used by list comprehensions.
- eru 3y ago
  Haskell has both do-notation and list comprehension. Comprehension in both Python and Haskell (for both lists and other structures) use the same order in both language, as far as I remember.
  - lmm 3y ago
    > Haskell has both do-notation and list comprehension. Right, and do-notation is the one everyone uses, because it's better. Python picked the wrong one. > Comprehension in both Python and Haskell (for both lists and other structures) use the same order in both language, as far as I remember. It may be the same order as Haskell but it's a terrible confusing order. In particular if you want to go from a nested list comprehension to a flat one (or vice versa) then you have to completely rearrange the order it's written in, whereas if you go from nested do-blocks to flat do-blocks then it all makes sense.
    - eru 3y ago
      I see what you mean, but I don't find the order that confusing in neither Haskell or Python. However, I can imagine a feature that we could add to Python to fix this: make it possible for statements to have a value. Perhaps something like this: my_generator = \ for i in "abc": for b in range(3): print("foo") yield (i, b) or perhaps have the last statement in block be its value (just like Rust or Ruby or Haskell do with the last statement in a block), and make the value of a for-loop be a generator of the individual values: my_list = list( for i in "abc": for b in range(3): (i, b)) Though there's a bit of confusion here whether the latter examples should be a flat structure or a nested one. You could probably use a similiar mechanism as the existing 'yield from' to explicitly ask for the flat version, and otherwise get the nested one: my_list = list( for i in "abc": yield from for b in range(3): (i, b)) Making Python statements have values looks to me like the more generally useful change than tweaking comprehensions. You'd probably not need comprehension at all in that case. Especially since you can already write loop header and body on a single line like for i in range(10): print(i) if they are short enough.
      - nerdponx 3y ago
        for i in range(10): print(i) But what would that return? [None] * 10? The limited whitespace-based syntax limits the potential for fun inline statement things, but it also completely dodges the question of what any particular statement should evaluate to when used as an expression.
        eru 3y ago
        > But what would that return? [None] * 10? Yes, I guess something like that. That was just meant as an example of how existing Python allows you to write loops on one line. It's not a good example for a meaningful comprehension in our alternative made-up Python dialect. > The limited whitespace-based syntax limits the potential for fun inline statement things, [...] Python already mostly allows you to use parens to override the indentation. They would just need to generalise that a bit. Btw, Haskell already does that: Officially, Haskell has a syntax with curly braces and semicolons; and they define the indentation based syntax as syntactic sugar that desugars to ; and {}. But almost everyone uses indentation based syntax. The exception are perhaps code generators and when posting on a website that messes with indentation. (And, because it's Haskell, the {}; syntax is just another layer of syntactic sugar for 'weird-operator'-based based syntax like >>=.)
panzi 3y ago
How does list comprehension change anything here? This has the same problem: add_n = [lambda x: x + n for n in range(10)] add_n[9](10) # 19 add_n[0](10) # 19
- billyjmc 3y ago
  I’m not sure what they mean by list comprehensions, either, but for completeness’s sake, I must point out that this is solvable by adding `n` as a keyword argument defaulting to `n`: add_n = [lambda x, n=n: x + n for n in range(10)] add_n[9](10) # 19 add_n[0](10) # 10
  - drekipus 3y ago
    This is the way Also pylsp warns you about this
deleted 3y ago
[deleted]

jshen 3y ago

Doesn’t go vet complain about your code? I’m not at my computer right now so can’t check.

_ikke_ 3y ago
> Tools have been written to identify these mistakes, but it is hard to analyze whether references to a variable outlive its iteration or not. These tools must choose between false negatives and false positives. The loopclosure analyzer used by go vet and gopls opts for false negatives, only reporting when it is sure there is a problem but missing others. So it will warn in certain situations, but not all of them
simiones 3y ago
Why would it? It's perfectly correct code, it's just not doing what you'd expect. It might complain about the race condition, to be fair, but the same issue can be reproduced without goroutines and it would be completely correct code per the semantics.
- jshen 3y ago
  In many languages "if x = 3" is perfectly valid code, but almost certainly not what the person intended "if x == 3". It's very smart to warn someone in a scenario like this.
  - simiones 3y ago
    I don't really write C too much, but I thought `if err = functionWithErrorReturn() { handleError(err) }` was a somewhat common idiom.
    - defrost 3y ago
      It's a common enough idiom from "stone age" bare bones K&R C, absolutely. It's also one of the great foot-guns of C programming as there are so many other almost but not that idioms and it's never clear on casual inspection whether the result of an assignment was meant to be examined or the result of a comparison. With the evolution of C and C sanity tools that rightfully flag such statements for double checking and the desire to not have spurious flagging, etc. it's more common in later C code to see (say) if ((err = someFunction()) != NOERROR) { errorHandle(err) } that optimises down to the same intermediate code where NOERROR is 0, sure, but it makes it very clear what is going on, an intended assignment and then an intended comparison. As with all idoms the general practice in the larger codebase and house code standard rules apply - there are other ways of doing similar things.

Spivak 3y ago

Ignoring the strange nature of this code in the first place the more pythonic way to do it would be from functools import partial from operators import add add_n = [partial(add, n)) for n in range(10)] assert add_n[5](4) == 9 Look ma, no closures.

eru 3y ago
'partial' creates a closure for you.
- Spivak 3y ago
  Unless you're talking philosophically how classes and closures are actually isomorphic then no, it doesn't. None of the variables in the outer scope are captured in the class instance. https://github.com/python/cpython/blob/main/Lib/functools.py#L276 https://github.com/python/cpython/blob/main/Lib/functools.py... Here's a simplified version of that code that demonstrates the pattern. class partial: def __init__(self, func, *args, **kwargs): self.func = func self.args = args self.kwargs = kwargs def __call__(self, *args, **kwargs): return self.func(*self.args, *args, **(self.kwargs | kwargs)) p = partial(add, 5) # -> an instance of partial with self.args = (5,) res = p(4) # -> calls __call__ which merges the args and calls add(5, 4)
  - eru 3y ago
    I was talking 'philosophically' in that sense. The partial object does create a new scope that binds a few of those variables. But you are also right that the mechanisms in Python are different (on some suitable mid-level of abstraction) for those two.

o11c 3y ago

It's actually worse in Python since there's no support for variable lifetimes within a function, so the `v2` workaround is still broken. (the default-argument workaround "works" but is scary) This makes it clear: the underlying problem is NOT about for loops - it's closures that are broken.

nerdponx 3y ago
It's not broken, it's a different design. Maybe worse in a lot of cases, but it's not broken. It's working as intended.
- eru 3y ago
  You could say the design is broken, but the implementation is working as intended by the design.

Frotag 3y ago

I've run into this once. IIRC the workaround was to add a n=n arg to the lambda

eru 3y ago

In Python you are much more likely to hit that problem not with closures constructed with an explicit 'lambda', but with generator-comprehension expressions. (((i, j) for i in "abc") for j in range(3)) The values of the above depends on in which order you evaluate the whole thing. (Do take what I wrote with a grain of salt. Either the above is already a problem, or perhaps you also need to mix in list-comprehension expressions, too, to surface the bug.)

nerdponx 3y ago
Yeah, this one is weird: gs1 = (((i, j) for i in "abc") for j in range(3)) gs2 = [((i, j) for i in "abc") for j in range(3)] print(list(map(list, gs1))) print(list(map(list, gs2))) Results: [[('a', 0), ('b', 0), ('c', 0)], [('a', 1), ('b', 1), ('c', 1)], [('a', 2), ('b', 2), ('c', 2)]] [[('a', 2), ('b', 2), ('c', 2)], [('a', 2), ('b', 2), ('c', 2)], [('a', 2), ('b', 2), ('c', 2)]] That's a nice "wat" right there. I believe the explanation is that in gs2, the range() is iterated through immediately, so j is always set to 2 before you have a chance to access any of the inner generators. Whereas in gs1 the range() is still being iterated over as you access each inner generator, so when you access the first generator j=1, then j=2, etc. Equivalents: def make_gs1(): for j in range(2): yield ((i, j) for i in "abc") def make_gs2(): gs = [] for j in range(2): gs.append(((i, j) for i in "abc")) return gs Late binding applies in both cases of course, but in the first case it doesn't matter, whereas in the latter case it matters. I think early binding would produce the same result in both cases.
- _ZeD_ 3y ago
  or you could just be eager and use lists: >>> [[(i, j) for i in "abc"] for j in range(3)] [[('a', 0), ('b', 0), ('c', 0)], [('a', 1), ('b', 1), ('c', 1)], [('a', 2), ('b', 2), ('c', 2)]]
  - nerdponx 3y ago
    Right, creating generators in a loop is not usually something you want to do, but it's meant to demonstrate the complexity that arises from late binding rather than demonstrate something you would actually want to do in a real program.

simiones 3y ago

I don't think anyone is puzzled by the Go snippet being wrong. The bigger problem in Go is the for with range loop: pointersToV := make([]*val, len(values)) for i, v := range values { go func() { fmt.Printf("num: %v\n", v) } () //race condition pointersToV[i] = &v //will contain len(values) copies of a pointer to the last item in values } This is the one they are changing. Edit: it looks like they're actually changing both of these, which is more unexpected to me. I think the C# behavior makes more sense, where only the foreach loop has a new binding of the variable in each iteration, but the normal for loop has a single lifetime.