Comments on: N-Gram Language Guessing with NGramJ

By: rsmudge

rsmudge — Wed, 24 Mar 2010 15:46:18 +0000

Thanks, this is fixed now.

By: rsmudge

rsmudge — Wed, 24 Mar 2010 15:44:24 +0000

I haven’t tried these languages yet. One key I’ve found is to make sure your character encoding is correct all around. I use UTF-8.

1. Set it on the command line

export LC_CTYPE=en_US.UTF-8
export LANG=en_US.UTF-8

2. Make sure your files are encoded with UTF-8
3. Make sure Java is getting the UTF-8 hint with -Dfile.encoding=UTF-8

etc.

By: Dominique

Dominique — Tue, 23 Mar 2010 20:11:11 +0000

There is a syntax error in sortid.sl

return $2[1] <=> $1[1];

By: Dominique

Dominique — Tue, 23 Mar 2010 20:07:42 +0000

3 very interesting articles (with “Generating a Plain Text Corpus from Wikipedia” and “All about Language Model”). Did you add Arabic or Cyrillic languages or Japanese, Chinese ?

I created an Arabian ngp file, it works fine, but strangly Persian doesn’t work (score is always 0). Same thing, with Russian doesn’t work (with the ru.ngp file provided in ngramj). I suppose I make a mistake with the cngram API.

By: rsmudge

rsmudge — Thu, 11 Mar 2010 13:32:36 +0000

They’re probably pretty similar. I use ngramj as it’s Java and AtD is written mostly in Sleep/Java. In my own tests I’ve found one or two words is a coin toss as to what it will characterize it as. Once you get beyond a full sentence that says something it’s always correct. I have to add the says something caveat because I’ve found it will mischaracterize a list of names, addresses, and phone numbers.

By: Kevin

Kevin — Thu, 11 Mar 2010 08:59:50 +0000

How does it compare with libtextcat? I found libtextcat extremely simple to use, building an LM is just $ createfp lang-fingerprint.txt and it seemed to work fine to guess between the two variants of Norwegian (it characterised my bad attempts at spelling Dutch as Middle-Frisian =P)