Comments for After the Deadline

Comment on Generating a Plain Text Corpus from Wikipedia by Aly

Aly — Tue, 10 Aug 2010 21:54:09 +0000

Hmm, it worked fine on a different version of wikipedia (the latest one: 20100728). Not sure what happened. Thanks for your quick reply anyway.

Comment on Generating a Plain Text Corpus from Wikipedia by rsmudge

rsmudge — Tue, 10 Aug 2010 02:38:28 +0000

File not found eh… make sure you have enough disk space and enough free inodes on that disk to create all the files you will need. This script creates a lot of them. It’s been a long time since I’ve run this process, but I was able to pull it off for 10 wikipedias (including en). I have faith you’ll get it too.

Comment on Generating a Plain Text Corpus from Wikipedia by Aly

Aly — Mon, 09 Aug 2010 02:41:19 +0000

I failed with your script on step 3.
Any ideas what’s wrong? Thanks!

Traceback (most recent call last):
File “./xmldump2files.py”, line 93, in
xml.sax.parse(sys.argv[1], WikiPageSplitter(sys.argv[2]))
File “/tmp/python.6884/usr/lib/python2.5/xml/sax/__init__.py”, line 33, in parse
File “/tmp/python.6884/usr/lib/python2.5/xml/sax/expatreader.py”, line 107, in parse
File “/tmp/python.6884/usr/lib/python2.5/xml/sax/xmlreader.py”, line 123, in parse
File “/tmp/python.6884/usr/lib/python2.5/xml/sax/expatreader.py”, line 207, in feed
File “/tmp/python.6884/usr/lib/python2.5/xml/sax/expatreader.py”, line 304, in end_element
File “./xmldump2files.py”, line 79, in endElement
writeArticle(self.root, self.title, self.text)
File “./xmldump2files.py”, line 41, in writeArticle
out = open(filename, “w”)
IOError: [Errno 2] No such file or directory: ‘../enwiki-20100130-out/90/f9/Con.txt’

Comment on Generating a Plain Text Corpus from Wikipedia by rsmudge

rsmudge — Wed, 28 Jul 2010 14:29:07 +0000

And so I did. Thanks for pointing this out. I’ve corrected the post.

Comment on Generating a Plain Text Corpus from Wikipedia by yhj

yhj — Wed, 28 Jul 2010 07:34:53 +0000

thanks for your post. it is very useful for me. there is just a small issue on the last step. i think you have missed a sleep.jar after “-jar” : )

Comment on After the Deadline Bigram Corpus – Our Gift to You… by tszming

tszming — Tue, 27 Jul 2010 01:41:24 +0000

I agree more isn’t better, so currently I am doing research to mine those set from Wikipedia’ edit history.

Comment on After the Deadline Bigram Corpus – Our Gift to You… by rsmudge

rsmudge — Tue, 27 Jul 2010 00:08:36 +0000

I maintain the set by hand. There are other sets (for example, http://www.dcs.bbk.ac.uk/~jenny/resources.html) that contain more entries. More isn’t always better though. More entries (especially arbitrary ones that haven’t been tested) may lead to more false positives.

Comment on After the Deadline Bigram Corpus – Our Gift to You… by tszming

tszming — Mon, 26 Jul 2010 09:39:21 +0000

How do you generate the “confusion set” and keep it updated?

To me, this seems to be not a big set, e.g. bar, bra not appear in the file.

Comment on After the Deadline Bigram Corpus – Our Gift to You… by rsmudge

rsmudge — Wed, 21 Jul 2010 03:33:29 +0000

Thanks for the comment. It’s always good to read something constructive from a member of the community. Yes, I should not make mistakes. I’m trying to make software that will help writers break their bad habits. One of those bad habits, unfortunately, is forgetting to run the tool.

Comment on After the Deadline Bigram Corpus – Our Gift to You… by Dick Jenkin

Dick Jenkin — Wed, 21 Jul 2010 00:00:40 +0000

There’s only one trouble with doing your sort of work – you cannot afford to make any spelling or grammar mistakes yourselves! Have a look at the “it’s” in the last line of this paragraph!

bigrams.txt.gz
This file is a dump of each two-word sequence that occurs in our corpus. It has 5,612,483 word pairs associated with a count. You can use this information to calculate the probability of a word given it’s next or previous words.

HOWEVER – I see that it has been fixed already (apostrophe removed)!! Full credit for that!
Thanks – Dick Jenkin.