After the Deadline

One thing that makes AtD so powerful is using the right tool for the right job.  AtD uses a rule-based approach to find some errors and a statistical approach to find others.  The misused word detection feature is the heaviest user of the statistical approach.

Last week I made great progress on the misused word detection making it look at the two words (trigrams) before the phrase in question instead of just one (bigrams).  There were several memory challenges to get past to make this work in production, but it happened and I was happy with the results.

Before this, I just looked at the one word before and after the current word.  All well and good but for certain kinds of words (too/to, their/there, your/you’re) this proved useless as it almost always guessed wrong.  For these words I moved to a rule-based approach.  Rules are great because they work and they’re easy to make.  The disadvantage of rules is a lot of them are required to get broad coverage of the errors people really make.  The statistical approach is great because it can pick up more cases.

The Experiment

So with the advent of the trigrams I decided to revisit the statistical technology for detecting errors I use rules for.  I focused on three types of errors just to see what kind of difference there was.  These types were:

• Wrong verb tense (irregular verbs only) – Writers tend to have trouble with irregular verbs as each case has to be memorized.  Native English speakers aren’t too bad but those just learning have a lot of trouble with these.  An example set would be throw vs. threw vs. thrown vs. throws vs. throwing.
• Agreement errors (irregular nouns only) – Irregular nouns are nouns that have plural and singular cases one has to memorize.  You can’t just add an s to convert them.  An example is die vs. dice.
• Confused words – And finally there are several confused words that the statistical approach just didn’t do much good for.  These include it’s/its, their/there, and where/were.

You’ll notice that each of these types of errors relies on fixed words with a fixed set of alternatives.  Perfect for use with the misused word detection technology.

My next step was to generate datasets for training and evaluating a neural network for each of these errors.  With this data I then trained the neural network and compared the test results of this neural network (Problem AI) to how the misused word detector (Misused AI) did as-is against these types of errors.  The results surprised me.

The Results

Table 1. Statistical Grammar Checking with Trigrams

First, you’ll notice the results aren’t that great.  Imagine what they were like before I had trigrams. 🙂  Actually, I ran those numbers too.  Here are the results:

Table 2. Statistical Grammar Checking with Bigrams

So, my instinct was at least validated.  There was a big improvement going from bigrams to trigrams… just not big enough.  What surprised me from these results is how close the AI trained for detecting the misused words performed to the AI trained for the problem at hand.

So one this experiment showed there is no need to train a separate model for dealing with these other classes of errors.  There is a slight accuracy difference but it’s not significant enough to justify a separate model.

Real World Results

The experiment shows that all I need (in theory) is a higher bias to protect against false positives.  With higher biasing in place (my target is always <0.05% false positives), I decided to run this feature against real writing to see what kind of errors it finds.  This is an important step because the tests come from fabricated data.  If the mechanism fails to find meaningful errors in a real corpus of writing then it’s not worth adding to AtD.

The irregular verb rules found 500 “errors” in my writing corpus.  Some of the finds were good and wouldn’t be picked up by rules, for example:

Romanian justice had no saying whatsoever
[ACCEPT] no, saying -> @('say')
However it will mean that once 6th May 2010 comes around the thumping that Labour get will be historic
[ACCEPT] Labour, get -> @('getting', 'gets', 'got', 'gotten')

Unfortunately, most of the errors are noise.  The confused words and nouns have similar stories.

Conclusions

The misused word detection is my favorite feature in AtD.  I like it because it catches a lot of errors with little effort on my part.  I was hoping to get the same win catching verb tenses and other errors.  This experiment showed misused word detection is fine for picking a better fit from a set and I don’t need to train separate models for different types of errors.  This gives me another tool to use when creating rules and also gives me a flexible tool to use when trying to mine errors from raw text.

I’ve been hard at work improving the statistical misused word detection in After the Deadline.  Nothing in the world of NLP is ever perfect.  If computers understood English we’d have bigger problems like rapidly evolving robots that try to enslave us.  I’m no longer interested in making robots to enslave people, that’s why I left my old job.

AtD’s statistical misused word detection relies on a database of confusion sets.  A confusion set is a group of words that are likely to be mistaken for each other.  An example is cite, sight, and site.  I tend to type right when I mean write, but only when I’m really tired.

AtD keeps track of about 1,500 of these likely to be confused words.  When it encounters one in your writing, it checks if any of the other words from the confusion set are a statistically better fit.  If one of the words is a better fit then your old word is marked as an error and you’re presented with the options.

AtD’s misused word detection uses the immediate left and right context of the word to decide which word is the better fit. I have access to this information using the bigram language model I assembled from public domain books, Wikipedia, and many of your blog entries.  Here is how well the statistical misused word detection fares with this information:

Table 1. AtD Misused Word Detection Using Bigrams

Of course, I’ve been working to improve this. Today I got my memory usage under control and I can now afford to use trigrams or sequences of three words. With a trigram I can use the previous two words to try and predict the third word. Here is how the statistical misused word detection fares with this extra information:

Table 2. AtD Misused Word Detection Using Bigrams and Trigrams

AtD uses neural networks to decide how to weight all the information fed to it.  These tables represent the before and after of introducing trigrams.  As you can see the use of trigrams significantly boosts precision (how often AtD is right when it marks a word as wrong) and recall (how often AtD marks a word as wrong when it is wrong).

Lies…

If you think these numbers are impressive (and they’re pretty good), you should know a few things.  The separation between the training data and testing data isn’t very good.  I generated datasets for training and testing, but both datasets were drawn from text used in the AtD corpus.  More clearly put–these numbers assume 100% coverage of all use cases in the English language because all the training and test cases have trigrams and bigrams associated with them.  If I were writing an academic paper, I’d take care to make this separation better.  Since I’m benchmarking a feature and how new information affects it, I’m sticking with what I’ve got.

This feature is pretty good, and I’m gonna let you see it, BUT…

I have some work to do yet.  In production I bias against false positives which hurts the recall of the system.  I do this because you’re more likely to use the correct word and I don’t want AtD flagging your words unless it’s sure the word really is wrong.

The biasing of the bigram model put the statistical misused word detection at a recall of 65-70%.  I expect to have 85-90% when I’m done experimenting with the trigrams.  My target false positive rate is under 0.5% or 1:200 correctly used words identified as wrong.

With any luck this updated feature will be pushed into production tomorrow.