7 ms·

Using aligned word vectors for instant translations with Python and Rust

112 points by beau 5y ago

PaulHoule 5y ago
Nice example. The short text and that fact that your application would tolerate or celebrate catchy neologisms plays to fasttext's strengths.
- beau 5y ago
  Thank you!
beau 5y ago
We've released the underlying Rust implementation here: https://github.com/InstantDomain/instant-distance https://github.com/InstantDomain/instant-distance with Python bindings at https://pypi.org/project/instant-distance https://pypi.org/project/instant-distance — feedback welcome!
- Fiahil 5y ago
  I’ve not much to say on the actual lib, it seems great! However, don’t feel compelled to put all your rust code into a single lib.rs. You can split your work into several files and use ‘pub use’ and ‘mod’ in lib.rs to re-export your functions & types into a public API of your choosing. cargo check and format time might also slightly improve!
  - ibraheemdev 5y ago
    Funny, I often say the opposite. Don't feel compelled to split up your lib.rs. It's really refreshing to see a nice, compact library in one or two files. Much easier to follow, especially over "type per file". Of course, there are limits, but for a small lib like this, I personally would keep it in a single, or maybe two files.
  - dochtman 5y ago
    I have a fair bit of experience writing Rust code and the current status is totally deliberate. I find module file sizes of about 400-800 lines of code optimal in terms of my ability to find things vs the unnecessary complexity of having to skip around files when changing something that touches an API boundary.
- arbol 5y ago
  Did you try spacy's most similar method? It's written in cython so is presumably quite fast as well. Thanks for the rust implementation though, I will most likely use this.
- habibur 5y ago
  For Linux, in the Makefile change the copy command to cp target/release/libinstant_distance.so instant-distance-py/test/instant_distance.so and it works. Built and running. The main tree was MacOS only. Here's resource consumption in a sample run. Time: 4.49s, Memory: 1552 mb. Single word. Three langs including en.
  - planckscnst 5y ago
    How did you figure this out? I've done lots of Linux software build troubleshooting as a result of using Gentoo, BuildRoot, and pacaur, but this doesn't ring any bells for a common issue
    - fnord123 5y ago
      They probably tried it, it couldn't find a dynlib which is a Macos shared object file, opened the three line makefile, and then fixed it to copy a .so
deleted 5y ago
[deleted]
deleted 5y ago
[deleted]
denysvitali 5y ago
> For example, here are the results of translating the English word "hello": > Language: fr, Translation: bonjours > Language: fr, Translation: bonsoir > Language: fr, Translation: salutations > Language: it, Translation: buongiorno > Language: it, Translation: buonanotte > Language: fr, Translation: rebonjour > Language: it, Translation: auguri > Language: fr, Translation: bonjour, > Language: it, Translation: buonasera > Language: it, Translation: chiamatemi Is it just me or these machine translations are worse than ... Google Translate?
- toxik 5y ago
  Google Translate is state of the art, so I’m not sure why that would be surprising. That said, is there something wrong with the translations offered?
  - dataflow 5y ago
    > That said, is there something wrong with the translations offered? I think in French hello = "bonjour" and hi = "salut"... not sure where "bonjours" and "salutations" came from.
    - T-A 5y ago
      The Italian "auguri" means "best wishes"; "chiamatemi" means "call me". Neither is a plausible translation of "hello". The obvious one, "ciao", is missing.
      - e17 5y ago
        I thought Hello was invented with the telephone. Prior to that, English greetings were good morning/evening. What do Italians and French say when they pick up the phone? Allora?
    - shakow 5y ago
      "Bonjours" doesn't exist, and "salutations" is a tad quirky, but OK in informal settings, especially when addressing many people at once.
aitk 5y ago
At first glance at the title, I thought it was translating Python code to Rust code.
- adsharma 5y ago
  It may not be a bad candidate for writing the rust part in python and then running it through py2many to generate the rust.
shakow 5y ago
> fast translates to vite in French Only as an adverb, it should be "rapide" otherwise.
dukeofdoom 5y ago
Can something like this be done to compare/translate subsequences COVID genetic code to SARS and other virus genetic codes. Would be interesting how much overlap there is. And would further the research into where it came from. Full genome of COVID-19 is available: https://www.snapgene.com/resources/coronavirus-resources/?resource=SARS-CoV-2_(COVID-19)_Genome https://www.snapgene.com/resources/coronavirus-resources/?re...
- nestorD 5y ago
  Bioinformaticists have been able to do that with traditional algorithms for years (dynamic programming gets you a long way to compute an edit distance for example). It is probably the first thing that was done once the COVID-19 genome was made public. A quick googling gave me that summary of the results: https://www.news-medical.net/health/How-Does-the-SARS-Virus-Genome-Compare-to-Other-Viruses.aspx https://www.news-medical.net/health/How-Does-the-SARS-Virus-...
  - dukeofdoom 5y ago
    Great article, thank you
- mattkrause 5y ago
  It sounds like you're thinking of "sequence alignment", which is a pretty standard bioinformatics tool. BLAST (=Basic Local Alignment Search Tool) is one common version, and the NIH'S NCBI has a variety of nice online tools here: https://blast.ncbi.nlm.nih.gov/Blast.cgi https://blast.ncbi.nlm.nih.gov/Blast.cgi Note that it does take a little bit of background knowledge to interpret:some motifs are just really common, others are shared.
fulafel 5y ago
Was disappointed this can't translate from Python to Rust.
maeln 5y ago
This webpage use a significant amount of CPU constantly for no apparent reason (as far as I can see it is mostly a static webpage). What the hell ? Is it mining crypto in the background ?
- chakkepolja 5y ago
  Compiling rust.
- maybevain 5y ago
  At a quick glance it seems like some React component is constantly re-rending. Quick glance in this case: took a couple second snapshot on the Performance tab and saw a lot of React related calls.
- beau 5y ago
  Sorry, this page had a useEffect/setState render loop. We are running react@experimental with concurrent mode, and missed the error. Rolling out a fix now. Thanks!

112 points by beau 5y ago

Nice example. The short text and that fact that your application would tolerate or celebrate catchy neologisms plays to fasttext's strengths.

beau 5y ago
Thank you!

beau 5y ago

We've released the underlying Rust implementation here: https://github.com/InstantDomain/instant-distance https://github.com/InstantDomain/instant-distance with Python bindings at https://pypi.org/project/instant-distance https://pypi.org/project/instant-distance — feedback welcome!

Fiahil 5y ago
I’ve not much to say on the actual lib, it seems great! However, don’t feel compelled to put all your rust code into a single lib.rs. You can split your work into several files and use ‘pub use’ and ‘mod’ in lib.rs to re-export your functions & types into a public API of your choosing. cargo check and format time might also slightly improve!
- ibraheemdev 5y ago
  Funny, I often say the opposite. Don't feel compelled to split up your lib.rs. It's really refreshing to see a nice, compact library in one or two files. Much easier to follow, especially over "type per file". Of course, there are limits, but for a small lib like this, I personally would keep it in a single, or maybe two files.
- dochtman 5y ago
  I have a fair bit of experience writing Rust code and the current status is totally deliberate. I find module file sizes of about 400-800 lines of code optimal in terms of my ability to find things vs the unnecessary complexity of having to skip around files when changing something that touches an API boundary.
arbol 5y ago
Did you try spacy's most similar method? It's written in cython so is presumably quite fast as well. Thanks for the rust implementation though, I will most likely use this.
habibur 5y ago
For Linux, in the Makefile change the copy command to cp target/release/libinstant_distance.so instant-distance-py/test/instant_distance.so and it works. Built and running. The main tree was MacOS only. Here's resource consumption in a sample run. Time: 4.49s, Memory: 1552 mb. Single word. Three langs including en.
- planckscnst 5y ago
  How did you figure this out? I've done lots of Linux software build troubleshooting as a result of using Gentoo, BuildRoot, and pacaur, but this doesn't ring any bells for a common issue
  - fnord123 5y ago
    They probably tried it, it couldn't find a dynlib which is a Macos shared object file, opened the three line makefile, and then fixed it to copy a .so

deleted 5y ago

[deleted]

deleted 5y ago

[deleted]

denysvitali 5y ago

> For example, here are the results of translating the English word "hello": > Language: fr, Translation: bonjours > Language: fr, Translation: bonsoir > Language: fr, Translation: salutations > Language: it, Translation: buongiorno > Language: it, Translation: buonanotte > Language: fr, Translation: rebonjour > Language: it, Translation: auguri > Language: fr, Translation: bonjour, > Language: it, Translation: buonasera > Language: it, Translation: chiamatemi Is it just me or these machine translations are worse than ... Google Translate?

toxik 5y ago
Google Translate is state of the art, so I’m not sure why that would be surprising. That said, is there something wrong with the translations offered?
- dataflow 5y ago
  > That said, is there something wrong with the translations offered? I think in French hello = "bonjour" and hi = "salut"... not sure where "bonjours" and "salutations" came from.
  - T-A 5y ago
    The Italian "auguri" means "best wishes"; "chiamatemi" means "call me". Neither is a plausible translation of "hello". The obvious one, "ciao", is missing.
    - e17 5y ago
      I thought Hello was invented with the telephone. Prior to that, English greetings were good morning/evening. What do Italians and French say when they pick up the phone? Allora?
  - shakow 5y ago
    "Bonjours" doesn't exist, and "salutations" is a tad quirky, but OK in informal settings, especially when addressing many people at once.

aitk 5y ago

At first glance at the title, I thought it was translating Python code to Rust code.

adsharma 5y ago
It may not be a bad candidate for writing the rust part in python and then running it through py2many to generate the rust.

shakow 5y ago

> fast translates to vite in French Only as an adverb, it should be "rapide" otherwise.

dukeofdoom 5y ago

Can something like this be done to compare/translate subsequences COVID genetic code to SARS and other virus genetic codes. Would be interesting how much overlap there is. And would further the research into where it came from. Full genome of COVID-19 is available: https://www.snapgene.com/resources/coronavirus-resources/?resource=SARS-CoV-2_(COVID-19)_Genome https://www.snapgene.com/resources/coronavirus-resources/?re...

nestorD 5y ago
Bioinformaticists have been able to do that with traditional algorithms for years (dynamic programming gets you a long way to compute an edit distance for example). It is probably the first thing that was done once the COVID-19 genome was made public. A quick googling gave me that summary of the results: https://www.news-medical.net/health/How-Does-the-SARS-Virus-Genome-Compare-to-Other-Viruses.aspx https://www.news-medical.net/health/How-Does-the-SARS-Virus-...
- dukeofdoom 5y ago
  Great article, thank you
mattkrause 5y ago
It sounds like you're thinking of "sequence alignment", which is a pretty standard bioinformatics tool. BLAST (=Basic Local Alignment Search Tool) is one common version, and the NIH'S NCBI has a variety of nice online tools here: https://blast.ncbi.nlm.nih.gov/Blast.cgi https://blast.ncbi.nlm.nih.gov/Blast.cgi Note that it does take a little bit of background knowledge to interpret:some motifs are just really common, others are shared.

fulafel 5y ago

Was disappointed this can't translate from Python to Rust.

maeln 5y ago

This webpage use a significant amount of CPU constantly for no apparent reason (as far as I can see it is mostly a static webpage). What the hell ? Is it mining crypto in the background ?

chakkepolja 5y ago
Compiling rust.
maybevain 5y ago
At a quick glance it seems like some React component is constantly re-rending. Quick glance in this case: took a couple second snapshot on the Performance tab and saw a lot of React related calls.
beau 5y ago
Sorry, this page had a useEffect/setState render loop. We are running react@experimental with concurrent mode, and missed the error. Rolling out a fix now. Thanks!