8 ms·

I like differentiable programming a lot, but can anyone explain the motivation for Swift as their language of choice for this? I assume it was simply because th

I like differentiable programming a lot, but can anyone explain the motivation for Swift as their language of choice for this? I assume it was simply because they wanted to get Chris Lattner, and Swift was his baby. Julia seems like a much better target for this, I wish they had just dedicated some money or resources to that. I don't have a problem with Swift's language design per-se (although Julia's dynamic nature really lends itself more naturally to the sort of problems you might be trying to solve using differential programming, in my opinion) but its cross-platform support is terrible. Why would you choose Mac's boutique language as the foundation of your new paradigm?

nnq 7y ago
> Julia seems like a much better target for this Exactly, at least from an outside perspective, why tf?! It seems like people love so much bolting stuff to "classic, conservative" languages that they do it even if it's 10x more work than doing it in a language already having full fledged mature real macros. And this is not Common Lisp we're talking about... Julia is a pretty "normal" language that wouldn't scare anyone off! (EDIT+: though, as a software engineer primarily, I do have a slight personal preference for more classic single-dispatchy languages like Swift... I imagine that in general it's some king of Stockholm-syndrome we developers experience :P)
- deleted 7y ago
  [deleted]
- MiroF 7y ago
  > "classic, conservative" Very odd to see these words used in conjunction with a modern language like Swift.
- dnautics 7y ago
  > classic single-dispatchy languages like Swift Is C++ not classic enough these days?
  - madhadron 7y ago
    Hard to categorize C++. The base object system is single dispatch, but the way you use the STL is multiple dispatch.
    - dnautics 7y ago
      You can overload any function in c++. It's multiple dispatch.
      - ddragon 7y ago
        Multiple dispatch is a little more than just overload. In C++ if you have a base class Animal with two derived Dog and Cat, and you have an Animal * pointer to Dog or Cat and do animal->walk() it will dispatch to either the Dog or Cat method (single dispatch). If you do animal->meet(animal) it will dispatch to either dog.meet(Animal * ) or cat.meet(Animal * ), not dog.meet(Dog * ) or cat.meet(Cat * ) like multiple dispatch languages do. You need a visitor pattern in C++ (or apparently templates) to get double dispatch: https://en.wikipedia.org/wiki/Visitor_pattern#C++_example https://en.wikipedia.org/wiki/Visitor_pattern#C++_example
- cbkeller 7y ago
  Julia seems to be making good progress on this front anyways :) https://github.com/FluxML/Zygote.jl https://github.com/FluxML/Zygote.jl
asplake 7y ago
Most likely explanation: they were going to do it in Swift anyway, and they’re doing out in the open, for others to scrutinise, copy and potentially improve upon. That’s cool, right?
RossBencina 7y ago
If I remember correctly, there is an FAQ somewhere that says that they chose Swift because they were already familiar with it, and didn't know anything about Julia.
- nnq 7y ago
  ...practically makes sense, but it's also kind of the very definition of NIH!
- tonynguyen1 7y ago
  https://github.com/tensorflow/swift/blob/master/docs/WhySwiftForTensorFlow.md https://github.com/tensorflow/swift/blob/master/docs/WhySwif.... Maybe this one
  - RossBencina 7y ago
    I think so. Under "Final Decision": > picked Swift over Julia because Swift has a much larger community, is syntactically closer to Python, and because we were more familiar with its internal implementation details - which allowed us to implement a prototype much faster. Personally I find it hard to believe that Swift has a larger community of people doing numerical computation, but maybe I'm out of touch.
    - dual_basis 7y ago
      The community of people using Swift for numerical computation essentially didn't exist prior to their announcement, and after their announcement it seems to only consist of the team developing the framework and Jeremey Howard.
- dual_basis 7y ago
  That might explain why they chose it, but it doesn't give me (or anyone else who isn't familiar with Swift already) any reason to consider it.
enos_feedler 7y ago
While this is true, the prospect seems exciting to me because best in class ML might be a thing that could drive Swift towards better cross platform support.
- kkarakk 7y ago
  Cross platform is never going to be a target for current gen Apple execs. It would require a seismic level shift in their tech focus They don't have a real presence in traditional servers, they don't have any real presence in cloud, they have minimal presence in non consumer focused services area. Cross platform at this point would just obsolete their platform in favor of more widely present OS'es and mobile tech
  - dharma1 7y ago
    ..or it could be an opportunity for them. Apple relies more and more on services for growth. For instance, why not make iMessage into a cross-platform app, as a real privacy-focused competitor to Whatsapp, Telegram etc?
    - dual_basis 7y ago
      Because they don't make money directly from iMessage, they make money by selling the hardware, and the fact that iMessage only works on their hardware increases it's value and further locks you into their ecosystem.
      - dharma1 7y ago
        It's certainly true iMessage isn't monetised today, but it doesn't mean it couldn't be in the future. If China is anything to go by, people do everything on their IM app. Lock in - how many people are locked into Apple because of iMessage? I think it's just an ancillary service to most people, with WhatsApp, Messenger, Telegram, or WeChat in China being their main IM app. The longer this is the case, the less relevant iMessage becomes.
uoaei 7y ago
It's already been done by another group of researchers.[1] I'm not sure if that was a factor in their choice of Swift. [1] https://arxiv.org/abs/1907.07587 https://arxiv.org/abs/1907.07587
ninjin 7y ago
I think it is a bit unfair to say “I assume it was simply because they wanted to get Chris Lattner, and Swift was his baby”. Swift is an interesting programming language in its own right – and I am saying that as someone coding pretty much exclusively in Julia – and as I have stated before “If you are sitting on a team deeply familiar and passionate about a language – Swift – what kind of managerial fool would not let them take a stab at it? Especially with Lattner’s excellent track record” [1]. [1]: https://news.ycombinator.com/item?id=19717815 https://news.ycombinator.com/item?id=19717815 Only yesterday I gave a lecture to my cohort of MSc students on precisely this topic; there is history going back to the 60s, implementations alive since the 80s, and so much development over the last five years. As someone that cares deeply about the science (and as an engineer at heart), I simply can not be too sad that Googled picked up Lattner et al., poured money over them, and asked them to push the envelope of what is possible with differential programming. After all, what will stop us from lifting over advances to Julia et al. later down the line? Sure, I can wish that it would have been Bezanson et al. and not Lattner et al. that got the money poured over them, but that feels very petty. Let the “best” language win, and I still feel the same way that I felt in 2018 when Swift first planted their flag: “Swift has a non-existing scientific computing community […] they will have to build it entirely from scratch and community building is difficult. […] My decision to side with Julia is partially to stay my own course, partially a preference for ‘the bazaar’ development model, and partially because I have a hunch that Julia has a better chance to capture the scientific computing community as a whole which is likely to yield benefits down the line”. [2]: https://news.ycombinator.com/item?id=16939525 https://news.ycombinator.com/item?id=16939525
- high_derivative 7y ago
  I disagree, I don't think it's unfair to say at all. It is simply mismanagement to input resources into a language rework that is dead on arrival, community wise. This is of course par for the course for big tech research orgs where big names get a lot of free rein, but that does not mean it's not a strategic mistake here. This is simply about focus as an org, and this is the reason why PyTorch is getting so popular. There seems to be a massive lack of focus and direction in the TF org, too many egos wanting to put their stamp all over the APIs and subsystems (tf.keras anyone?). TensorFlow eager with autograph or Pytorch solve all differentiation problems as far as researchers and practitioners are concerned.
gwd 7y ago
> I like differentiable programming a lot, but can anyone explain the motivation for Swift as their language of choice for this? Back in January, Jeremy Howard (of fast.ai fame) wrote up a blog post talking about why he was exploring using Swift for this purpose: https://www.fast.ai/2019/01/10/swift-numerics/ https://www.fast.ai/2019/01/10/swift-numerics/ The post is quite in-depth; but just to give a brief take on other potential languages: 8<--- Here’s my personal view of some languages that I’ve used and enjoyed, but all of which have limitations I’ve found frustrating at times: Python: Slow at runtime, poor support for parallel processing (but very easy to use) C, C++: hard to use (and C++ is slow at compile time), but fast and (for C++) expressive Javascript: Unsafe (unless you use Typescript); somewhat slow (but easy to use and flexible) Julia: Poor support for general purpose programming, but fast and expressive for numeric programming. ( Edit: this may be a bit unfair to Julia; it’s come a long way since I’ve last looked at it!) Java: verbose (but getting better, particularly if you use Kotlin), less flexible (due to JVM issues), somewhat slow (but overall a language that has many useful application areas) C# and F#: perhaps the fewest compromises of any major programming language, but still requires installation of a runtime, limited flexibility due to garbage collection, and difficulties making code really fast (except on Windows, where you can interface via C++/CLI)
- C1sc0cat 7y ago
  Not sure why Julia is ruled out as not a general purpose language here - I got the impression that Julia was for numerical ie Technical programming
  - DagAgren 7y ago
    Because it would be a challenge to implement, for instance, an actual user-facing application in Julia.
    - ddragon 7y ago
      Mostly because the end-user application community is not as strong in Julia (though you can create web services/sites and GUIs right now), in the same way the scientific/numerical computing/machine learning/HPC isn't as strong in Swift. Perhaps the point he made was that since his area is ML, then he could contribute better in trying to create a community around that in Swift than he could bring the other domains to Julia up to par with those languages by adding another competing library. And in terms of the language itself, Julia is very much a general purpose language. Exceptionally general purpose as it's basically a Lisp below the surface, so it can not only support any domain, it can be comparatively easily extended to better support them. Differentiable programming is one such example, as the compiler was not designed for that, but you can just import this functionality as a library. Though the focus is still from desktop to HPC, as opposed to mobile/IoT to desktop (and apple focused) like Swift, which makes so having both languages supporting differentiable programming an overall positive over having only one of these languages.
ddragon 7y ago
Julia and Swift are different enough that there is benefit in having both with that functionality. Julia is a dynamic and heavily interactive language (especially when it improves more JIT latency and "time to first plot") for exploratory analysis (like what Python provides) and development without compromising performance in any step, with strong focus on HPC/Distributed computing. Swift has some interactivity, but it's core execution strategy is to create static binaries to be deployed in mobile and desktop, with much better support for high quality end user interfaces (especially in apple devices). While as a researcher and mostly linux developer I will probably not use Swift, I can easily see situations in which Swift would be the clear choice given both Zygote and Swift for Tensorflow equally mature, even when Julia's scientific computing support is much broader.
- dual_basis 7y ago
  s/especially/exclusively/
- taharvey 7y ago
  Or... as Jeremy Howard said it, having a real compiler with a strongly typed language is "mind blowing" after working in Python. A compiler isn't in your way, with modern JIT compilation and a language with an actual type system, its actually an productivity assistant. Shouldn't AI researchers have automation too?
taharvey 7y ago
Swift is: 1) A fast general purpose systems language (like C++) 2) With high level semantics and usability (like Python) 3) With type system from functional programming (like Haskel) 4) With concurrent memory model (like Rust) 5) That uses compile-time analysis for static guarantees 6) That runs everywhere from CLI scripting to ML to App devlopment 7) With a huge user base (millions of devs) 8) And two large corporate backers (Apple & Google) 9) The developers include the originators of Rust and the developers of LLVM & Clang. Thats a killer list. And Julia only addresses 1 & 2. It is further believed by the core group at Google and Fast.ai (and history) that the future must be a static type checked language to scale beyond the script-level experimentation of python to large application development where the lines between ML, application, and system development will blur. Its will also be critical to next-gen performance breakthroughs and heterogeneous computing. And the type system provides a fully hackable language for fast experimentation. Also Richard Wei, not Lattner, I believe initiated the project with his DLVM thesis... specifically because Swift was such a good fit and is so hackable.
- dual_basis 7y ago
  Except it basically only runs on MacOS, which means that it doesn't have strong datacenter support, which means that it won't be used in any serious applications. It's also not dynamic, which is almost a must-have for exploratory data analysis. I will avoid a point-by-point rebuttal, mostly to avoid seeming too antagonistic, but Julia supports many of the other features on the list you provided, and (more importantly, for the case at hand) already has proven it's capability in the differentiable programming paradigm with Zygote.
  - taharvey 7y ago
    You apparently aren't familiar with Swift at all. I'd recommend exploring more before answering. Swift had Linux support since it open sourced 4 years ago. IBM, AWS, Google all have very performant server-side swift packages for manner of things from servers to protocols. Our company is using Swift for a Linux-only embedded environments. If you think the future is dynamic, you're not regressing the past, or the trajectory of the future. Dynamic languages keep a compiler from understanding the code. Compilers are just code optimization heuristics - like AI for developers. If you make a language and compiler that has a strong ontology for the intent of code (a type system), it can break code down into functional proofs, then understand how to optimize it and in ways that aren't possible in a dynamic language that breaks the chain of intent. Julia is still pretty niche at this point, and just recently got tools as fundamental as a debugger.

nnq 7y ago

> Julia seems like a much better target for this Exactly, at least from an outside perspective, why tf?! It seems like people love so much bolting stuff to "classic, conservative" languages that they do it even if it's 10x more work than doing it in a language already having full fledged mature real macros. And this is not Common Lisp we're talking about... Julia is a pretty "normal" language that wouldn't scare anyone off! (EDIT+: though, as a software engineer primarily, I do have a slight personal preference for more classic single-dispatchy languages like Swift... I imagine that in general it's some king of Stockholm-syndrome we developers experience :P)

deleted 7y ago
[deleted]
MiroF 7y ago
> "classic, conservative" Very odd to see these words used in conjunction with a modern language like Swift.
dnautics 7y ago
> classic single-dispatchy languages like Swift Is C++ not classic enough these days?
- madhadron 7y ago
  Hard to categorize C++. The base object system is single dispatch, but the way you use the STL is multiple dispatch.
  - dnautics 7y ago
    You can overload any function in c++. It's multiple dispatch.
    - ddragon 7y ago
      Multiple dispatch is a little more than just overload. In C++ if you have a base class Animal with two derived Dog and Cat, and you have an Animal * pointer to Dog or Cat and do animal->walk() it will dispatch to either the Dog or Cat method (single dispatch). If you do animal->meet(animal) it will dispatch to either dog.meet(Animal * ) or cat.meet(Animal * ), not dog.meet(Dog * ) or cat.meet(Cat * ) like multiple dispatch languages do. You need a visitor pattern in C++ (or apparently templates) to get double dispatch: https://en.wikipedia.org/wiki/Visitor_pattern#C++_example https://en.wikipedia.org/wiki/Visitor_pattern#C++_example
cbkeller 7y ago
Julia seems to be making good progress on this front anyways :) https://github.com/FluxML/Zygote.jl https://github.com/FluxML/Zygote.jl

asplake 7y ago

Most likely explanation: they were going to do it in Swift anyway, and they’re doing out in the open, for others to scrutinise, copy and potentially improve upon. That’s cool, right?

RossBencina 7y ago

If I remember correctly, there is an FAQ somewhere that says that they chose Swift because they were already familiar with it, and didn't know anything about Julia.

nnq 7y ago
...practically makes sense, but it's also kind of the very definition of NIH!
tonynguyen1 7y ago
https://github.com/tensorflow/swift/blob/master/docs/WhySwiftForTensorFlow.md https://github.com/tensorflow/swift/blob/master/docs/WhySwif.... Maybe this one
- RossBencina 7y ago
  I think so. Under "Final Decision": > picked Swift over Julia because Swift has a much larger community, is syntactically closer to Python, and because we were more familiar with its internal implementation details - which allowed us to implement a prototype much faster. Personally I find it hard to believe that Swift has a larger community of people doing numerical computation, but maybe I'm out of touch.
  - dual_basis 7y ago
    The community of people using Swift for numerical computation essentially didn't exist prior to their announcement, and after their announcement it seems to only consist of the team developing the framework and Jeremey Howard.
dual_basis 7y ago
That might explain why they chose it, but it doesn't give me (or anyone else who isn't familiar with Swift already) any reason to consider it.

enos_feedler 7y ago

While this is true, the prospect seems exciting to me because best in class ML might be a thing that could drive Swift towards better cross platform support.

kkarakk 7y ago
Cross platform is never going to be a target for current gen Apple execs. It would require a seismic level shift in their tech focus They don't have a real presence in traditional servers, they don't have any real presence in cloud, they have minimal presence in non consumer focused services area. Cross platform at this point would just obsolete their platform in favor of more widely present OS'es and mobile tech
- dharma1 7y ago
  ..or it could be an opportunity for them. Apple relies more and more on services for growth. For instance, why not make iMessage into a cross-platform app, as a real privacy-focused competitor to Whatsapp, Telegram etc?
  - dual_basis 7y ago
    Because they don't make money directly from iMessage, they make money by selling the hardware, and the fact that iMessage only works on their hardware increases it's value and further locks you into their ecosystem.
    - dharma1 7y ago
      It's certainly true iMessage isn't monetised today, but it doesn't mean it couldn't be in the future. If China is anything to go by, people do everything on their IM app. Lock in - how many people are locked into Apple because of iMessage? I think it's just an ancillary service to most people, with WhatsApp, Messenger, Telegram, or WeChat in China being their main IM app. The longer this is the case, the less relevant iMessage becomes.

uoaei 7y ago

It's already been done by another group of researchers.[1] I'm not sure if that was a factor in their choice of Swift. [1] https://arxiv.org/abs/1907.07587 https://arxiv.org/abs/1907.07587

ninjin 7y ago

I think it is a bit unfair to say “I assume it was simply because they wanted to get Chris Lattner, and Swift was his baby”. Swift is an interesting programming language in its own right – and I am saying that as someone coding pretty much exclusively in Julia – and as I have stated before “If you are sitting on a team deeply familiar and passionate about a language – Swift – what kind of managerial fool would not let them take a stab at it? Especially with Lattner’s excellent track record” [1]. [1]: https://news.ycombinator.com/item?id=19717815 https://news.ycombinator.com/item?id=19717815 Only yesterday I gave a lecture to my cohort of MSc students on precisely this topic; there is history going back to the 60s, implementations alive since the 80s, and so much development over the last five years. As someone that cares deeply about the science (and as an engineer at heart), I simply can not be too sad that Googled picked up Lattner et al., poured money over them, and asked them to push the envelope of what is possible with differential programming. After all, what will stop us from lifting over advances to Julia et al. later down the line? Sure, I can wish that it would have been Bezanson et al. and not Lattner et al. that got the money poured over them, but that feels very petty. Let the “best” language win, and I still feel the same way that I felt in 2018 when Swift first planted their flag: “Swift has a non-existing scientific computing community […] they will have to build it entirely from scratch and community building is difficult. […] My decision to side with Julia is partially to stay my own course, partially a preference for ‘the bazaar’ development model, and partially because I have a hunch that Julia has a better chance to capture the scientific computing community as a whole which is likely to yield benefits down the line”. [2]: https://news.ycombinator.com/item?id=16939525 https://news.ycombinator.com/item?id=16939525

high_derivative 7y ago
I disagree, I don't think it's unfair to say at all. It is simply mismanagement to input resources into a language rework that is dead on arrival, community wise. This is of course par for the course for big tech research orgs where big names get a lot of free rein, but that does not mean it's not a strategic mistake here. This is simply about focus as an org, and this is the reason why PyTorch is getting so popular. There seems to be a massive lack of focus and direction in the TF org, too many egos wanting to put their stamp all over the APIs and subsystems (tf.keras anyone?). TensorFlow eager with autograph or Pytorch solve all differentiation problems as far as researchers and practitioners are concerned.

gwd 7y ago

> I like differentiable programming a lot, but can anyone explain the motivation for Swift as their language of choice for this? Back in January, Jeremy Howard (of fast.ai fame) wrote up a blog post talking about why he was exploring using Swift for this purpose: https://www.fast.ai/2019/01/10/swift-numerics/ https://www.fast.ai/2019/01/10/swift-numerics/ The post is quite in-depth; but just to give a brief take on other potential languages: 8<--- Here’s my personal view of some languages that I’ve used and enjoyed, but all of which have limitations I’ve found frustrating at times: Python: Slow at runtime, poor support for parallel processing (but very easy to use) C, C++: hard to use (and C++ is slow at compile time), but fast and (for C++) expressive Javascript: Unsafe (unless you use Typescript); somewhat slow (but easy to use and flexible) Julia: Poor support for general purpose programming, but fast and expressive for numeric programming. ( Edit: this may be a bit unfair to Julia; it’s come a long way since I’ve last looked at it!) Java: verbose (but getting better, particularly if you use Kotlin), less flexible (due to JVM issues), somewhat slow (but overall a language that has many useful application areas) C# and F#: perhaps the fewest compromises of any major programming language, but still requires installation of a runtime, limited flexibility due to garbage collection, and difficulties making code really fast (except on Windows, where you can interface via C++/CLI)

C1sc0cat 7y ago
Not sure why Julia is ruled out as not a general purpose language here - I got the impression that Julia was for numerical ie Technical programming
- DagAgren 7y ago
  Because it would be a challenge to implement, for instance, an actual user-facing application in Julia.
  - ddragon 7y ago
    Mostly because the end-user application community is not as strong in Julia (though you can create web services/sites and GUIs right now), in the same way the scientific/numerical computing/machine learning/HPC isn't as strong in Swift. Perhaps the point he made was that since his area is ML, then he could contribute better in trying to create a community around that in Swift than he could bring the other domains to Julia up to par with those languages by adding another competing library. And in terms of the language itself, Julia is very much a general purpose language. Exceptionally general purpose as it's basically a Lisp below the surface, so it can not only support any domain, it can be comparatively easily extended to better support them. Differentiable programming is one such example, as the compiler was not designed for that, but you can just import this functionality as a library. Though the focus is still from desktop to HPC, as opposed to mobile/IoT to desktop (and apple focused) like Swift, which makes so having both languages supporting differentiable programming an overall positive over having only one of these languages.

ddragon 7y ago

Julia and Swift are different enough that there is benefit in having both with that functionality. Julia is a dynamic and heavily interactive language (especially when it improves more JIT latency and "time to first plot") for exploratory analysis (like what Python provides) and development without compromising performance in any step, with strong focus on HPC/Distributed computing. Swift has some interactivity, but it's core execution strategy is to create static binaries to be deployed in mobile and desktop, with much better support for high quality end user interfaces (especially in apple devices). While as a researcher and mostly linux developer I will probably not use Swift, I can easily see situations in which Swift would be the clear choice given both Zygote and Swift for Tensorflow equally mature, even when Julia's scientific computing support is much broader.

dual_basis 7y ago
s/especially/exclusively/
taharvey 7y ago
Or... as Jeremy Howard said it, having a real compiler with a strongly typed language is "mind blowing" after working in Python. A compiler isn't in your way, with modern JIT compilation and a language with an actual type system, its actually an productivity assistant. Shouldn't AI researchers have automation too?

taharvey 7y ago

Swift is: 1) A fast general purpose systems language (like C++) 2) With high level semantics and usability (like Python) 3) With type system from functional programming (like Haskel) 4) With concurrent memory model (like Rust) 5) That uses compile-time analysis for static guarantees 6) That runs everywhere from CLI scripting to ML to App devlopment 7) With a huge user base (millions of devs) 8) And two large corporate backers (Apple & Google) 9) The developers include the originators of Rust and the developers of LLVM & Clang. Thats a killer list. And Julia only addresses 1 & 2. It is further believed by the core group at Google and Fast.ai (and history) that the future must be a static type checked language to scale beyond the script-level experimentation of python to large application development where the lines between ML, application, and system development will blur. Its will also be critical to next-gen performance breakthroughs and heterogeneous computing. And the type system provides a fully hackable language for fast experimentation. Also Richard Wei, not Lattner, I believe initiated the project with his DLVM thesis... specifically because Swift was such a good fit and is so hackable.

dual_basis 7y ago
Except it basically only runs on MacOS, which means that it doesn't have strong datacenter support, which means that it won't be used in any serious applications. It's also not dynamic, which is almost a must-have for exploratory data analysis. I will avoid a point-by-point rebuttal, mostly to avoid seeming too antagonistic, but Julia supports many of the other features on the list you provided, and (more importantly, for the case at hand) already has proven it's capability in the differentiable programming paradigm with Zygote.
- taharvey 7y ago
  You apparently aren't familiar with Swift at all. I'd recommend exploring more before answering. Swift had Linux support since it open sourced 4 years ago. IBM, AWS, Google all have very performant server-side swift packages for manner of things from servers to protocols. Our company is using Swift for a Linux-only embedded environments. If you think the future is dynamic, you're not regressing the past, or the trajectory of the future. Dynamic languages keep a compiler from understanding the code. Compilers are just code optimization heuristics - like AI for developers. If you make a language and compiler that has a strong ontology for the intent of code (a type system), it can break code down into functional proofs, then understand how to optimize it and in ways that aren't possible in a dynamic language that breaks the chain of intent. Julia is still pretty niche at this point, and just recently got tools as fundamental as a debugger.