wooorm

8 ms·

1.
by wooorm 11y ago
alex is open to suggestion. See http://alexjs.com/#contributing on how to contribute. So if you have a word to add or phrasing to remove let us know!
2.
Show HN: Alex – Catch insensitive, inconsiderate writing(github.com)
4 points by wooorm 11y ago|1 comments
3.
Show HN: Mdast – Markdown processor powered by plugins(mdast.js.org)
3 points by wooorm 11y ago|0 comments
4.
by wooorm 12y ago
Perhaps you should ;) If, I’d be interest to know how it goes!
5.
by wooorm 12y ago
It’s a very interesting idea. Would it work accurate enough when scaled to 160+ languages?
6.
by wooorm 12y ago
I pushed a fix, incorporating your suggestions, and your examples in the specs. Thanks a lot!
7.
by wooorm 12y ago
It’s an interesting thought. I might fiddle on it, but I’m not sure it would work in practice (d’oh). Thanks!
8.
by wooorm 12y ago
I agree the task is neither impossible nor useless. There’s work to do. Short passages should be supported. I do however think franc does a good job, and adds support for some languages which before today have never (I think) been supported
9.
by wooorm 12y ago
Thanks ;)
10.
by wooorm 12y ago
By `correct language` I mean the language you expect, by `second` and `third` I mean `2.` and `3.` in the previously mentioned demo: http://wooorm.github.io/franc/ ). I think we’re talking about the same thing! Anyway
11.
by wooorm 12y ago
Ha! Some very nice examples, I have to say :) Anyway, You’re completely right. Italian is `und` due to LTE 10 characters, the others are slightly off due to short input too, but the demo ( http://wooorm.github.io/franc/
12.
by wooorm 12y ago
No full-frequency data is kept, only 300 top-trigrams are identified. A quick through the source also reveals wooorm/trigrams, and wooorm/udhr, as sources!
13.
by wooorm 12y ago
Oh you’re right. I think I have a fix in mind, will work on it. Thanks so much!
14.
by wooorm 12y ago
I’ll investigate this, but I think I excluded the preamble’s for trigram creation. Sure, the words will be a bit similar, but it’ll be a lot of work to compile 380 fixtures from other sources. I’ll investigate that too. But it’s lots of wor
15.
by wooorm 12y ago
Thanks! Currently, the UDHRs are crawled, and I’d rather not include exceptions and maintain their plain-text and XML/JSON versions by hand. If you’re into growing the language, I suggest contacting the Office of the High Commissioner
16.
by wooorm 12y ago
Franc seems to work well on longer passages. Such as these: https://github.com/wooorm/franc/blob/master/spec/fixtures.js... It’s interesting though, I’ll take a look at it!
17.
by wooorm 12y ago
It sucks, right? Currently, it’s good at long passages. But for shorter values, the results are pretty poor. The amount of supported languages is just too damn high!
18.
by wooorm 12y ago
That would be awesome :)
19.
by wooorm 12y ago
I’m not sure. I don’t know any CJK languages myself. I’d like some test-cases where the current methods do not work, as the example in the Readme seems to work pretty well: `এটি একটি ভাষা একক IBM স্ক্রিপ্ট` is classified as Bengali?
20.
by wooorm 12y ago
And it doesn’t have a Universal Declaration of Human rights: http://www.unicode.org/udhr/index_by_name.html
21.
by wooorm 12y ago
Fries as in Frisian? I don’t think it has one million speakers (right?) :p
22.
by wooorm 12y ago
You seem to be completely right, I hand-crawled the data ( https://github.com/wooorm/speakers ), but seem to have made big typo there! Thanks!
23.
by wooorm 12y ago
One of franc’s focusses was to be pretty small, and usable on the client-side, that’s why no actual training is done and this simple method is used. Also, I’m interest in a test-suite, before we start talking about accuracy-percentages :p
24.
by wooorm 12y ago
Yeah, so I’d like to add an easier way to support more, or less, languages through the Node API. Currently, there’s a number (1e6), the amount of speakers of a given language, which is hard-coded in the generation file (I added a link this
25.
by wooorm 12y ago
Agreed :)
26.
by wooorm 12y ago
I’m also really interested in trying something like this: http://www.slideshare.net/shuyo/short-text-language-detectio... (slide 6). But I’d need a lot of training data, more than UDHR.
27.
by wooorm 12y ago
That’s because Haitians always say that! No, joking, it’s just that because of so may supported languages, the accuracy for very short inputs is extremely low.
28.
by wooorm 12y ago
You are completely right, franc doesn’t state how language are detected. The detection is based on (1) unicode-script usage and (2) trigram-counts. Some scripts are only used by one language. Other scripts, such as Cyrillic, come with many
29.
Show HN: Franc – Detect natural languages(github.com)
51 points by wooorm 12y ago|60 comments
30.
Natural Language Parsing with Retext(dailyjs.com)
1 points by wooorm 12y ago|0 comments

More ›