Comments on: Learning from your mistakes — Some ideas

By: Can we use crowd sourcing to improve AtD? « After the Deadline

Can we use crowd sourcing to improve AtD? « After the Deadline — Thu, 27 May 2010 18:02:43 +0000

[...] to improve AtD? Posted in Uncategorized by rsmudge on May 27, 2010 I’ve written about learning from AtD use in the past. The main ideas I had back then were to bring more data into AtD’s corpus and [...]

By: Matt

Matt — Tue, 01 Dec 2009 03:04:05 +0000

Human spot-check it.

By: rsmudge

rsmudge — Tue, 01 Dec 2009 01:38:44 +0000

The trick is deciding what to do with the information when the system gets it. If we know a rule failed in a certain case, how should we feed this to the system so it benefits? One possibility is to add the sentence with that context into the corpora, that would at least aid the statistical filtering and improve the precision.

I haven’t done any research into mining rules or developing an algorithmic approach to improving the grammar checker recall, but I think this would be an interesting area to work in.

In case someone wants to pick it up, AtD has two data sets for evaluating the grammar checker. They’re data/tests/grammar_gutenberg.txt and data/tests/grammar_wikipedia.txt. These two files are merely phrases taken from the Wikipedia common grammar errors list merged with context from the sources in their name. There is also a script bin/testgr.sh that tests the precision and recall of the AtD grammar checker using these sources.

Before bringing user data in, I think using these datasets to measure the effectiveness of such and approach would make sense.

By: Chris

Chris — Tue, 01 Dec 2009 01:28:00 +0000

I’m wondering what could be done to increase the recall or precision of the grammar checker? It seems if you gave users the ability to tag grammar corrections as “false positive” than you could build up a corpus of sentences where the rules failed. With this data you could train your rules to improve precision.

Conversely, you could give users the ability to flag grammar mistakes or, more likely, sentences as having one or more grammatical errors. In other words, you could build up a corpus of “false negatives” allowing you to improve recall.

What are your thoughts on this?

By: Matt

Matt — Mon, 23 Nov 2009 23:05:17 +0000

There’s no reason to have everything be 100% algorithmic, a hybrid approach with certain human-edited rules (like for “alot”) is probably the best bet long-term.