Vietnamese Language Tools: Developing Keyboards and Spell-checker

Mentoring Mozilla Vietnamese localization team to develop Vietnamese keyboard and spell checker for Firefox OS. We built and tested Vietnamese keyboard during Mozilla Vietnam L10N sprints in Hanoi. We will integrate this into Firefox OS after Mozilla UX team improves the keyboard design to accommodate large set of alternative character keys.

Lewtds created Vietnamese spell checker using the Ho Ngoc Duc's word list. The Vietnamese language processing work is very limited. We haven't found any good data to that create a frequency list for the Vietnamese spell checker. Any pointers would be much appreciated.

Please read the Mozilla FirefoxOS keyboard documentation to learn more about developing keyboards in your language.

Read More

Linux at Sanskrit Computational Linguistics Symposium workshop

Last week I traveled to Hyderabad to talk about 'Linux for Linguists' at the Sanskrit Computational Linguistics Symposium workshop. Department of Sanskrit studies, University of Hyderabad organized this national workshop titled "Sanskrit and Computers, Getting equipped to face new challenges".

Few years ago I spent sometime working with newly created department of Sanskrit Studies learning the fundamentals of computational linguistics. I return to the department every year to work with the researchers. They use Ubuntu Linux for their work, the Ubuntu logo kolam at the entrance of the workshop was a surprise.

Linux Talk In Sanskrit

The event had yet another surprise in store for me. Thirumala Kulkarni introduced Linux in the Sanskrit, one of oldest language of the world. Can you match that ! .

Linux For Linguists Slides

The first part of my talk was a demo of Ubuntu Linux LiveCD and how to install Ubuntu Linux on their personal computers. In my talk I introduced Linux and its numerous toolset that can be used for linguistics research. The talk was peppered with humour rather than technical jargon and I ended my talk asking the research to contribute back to the Ubuntu Linux project.

Linux For Linguists

q

The response from the participants of the workshop was very positive and inspiring. The event did generate a lot of interest and hope it will help more people adopt gnu/Linux for their research work.

Read More

Machine Translation Humor

Studying computational linguistics isn't dull. There is always something to laugh about, including the subject itself.

Courtesy: xkcd #114

Lets look at that classic illustration of Machine Translation(MT) of the English phrase The spirit is willing, but the flesh is weak to Russian and back. Lets see what would we get if we try with Japanese today.

• The spirit is willing, but the flesh is weak Let's start



• 精神が、肉体が弱い喜んで into Japanese



• Spirit, willing flesh weak back into English


• 精神は、喜んで、肉は弱い back into Japanese


• Spirit is willing, flesh is weak back into English


• 精神、肉体が弱い喜んで back into Japanese


• Spirit, I'd have a weak body back into English


• 精神を、私は弱い体がしたい back into Japanese


• Spirit, I want a weak body back into English


• 精神を、私は、弱い体を back into Japanese


• Spirit, I have a weak body back into English


• 精神を、私は弱い肉体を持って back into Japanese


• Spirit, I have a weak body back into English

View the complete results at Translation party.

Read More

NLTK Installation with Python easy_install

Few weeks ago I wrote the NLTK on Ubuntu Quick Start Guide. Now with the release of NLTK (Natural Language Toolkit) 2.0b5 today the NLTK installation has been greatly simplified thanks to the nltk python egg (See Changelog).

To get started with NLTK install, you first need the python-setuptools package.

$ sudo apt-get install python-setuptools 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  python-setuptools
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 195kB of archives.
After this operation, 909kB of additional disk space will be used.
Get:1 http://in.archive.ubuntu.com karmic/main python-setuptools 0.6c9-0ubuntu4 [195kB]
Fetched 195kB in 9s (20.2kB/s)                                                 
Selecting previously deselected package python-setuptools.
(Reading database ... 106971 files and directories currently installed.)
Unpacking python-setuptools (from .../python-setuptools_0.6c9-0ubuntu4_all.deb) ...
Setting up python-setuptools (0.6c9-0ubuntu4) ...



 Now lets install the NLTK with easy_install program.


$ sudo easy_install http://nltk.googlecode.com/files/nltk-2.0b5-py2.6.egg
 
Downloading http://nltk.googlecode.com/files/nltk-2.0b5-py2.6.egg
Processing nltk-2.0b5-py2.6.egg
creating /usr/local/lib/python2.6/dist-packages/nltk-2.0b5-py2.6.egg
Extracting nltk-2.0b5-py2.6.egg to /usr/local/lib/python2.6/dist-packages
Adding nltk 2.0b5 to easy-install.pth file

Installed /usr/local/lib/python2.6/dist-packages/nltk-2.0b5-py2.6.egg
Processing dependencies for nltk==2.0b5
Searching for PyYAML==3.08
Reading http://pypi.python.org/simple/PyYAML/
Reading http://pyyaml.org/wiki/PyYAML
Best match: PyYAML 3.08
Downloading http://pyyaml.org/download/pyyaml/PyYAML-3.08.zip
Processing PyYAML-3.08.zip
Running PyYAML-3.08/setup.py -q bdist_egg --dist-dir /tmp/easy_install-T7Y0La/PyYAML-3.08/egg-dist-tmp-vRjvDM
build/temp.linux-i686-2.6/check_libyaml.c:2:18: error: yaml.h: No such file or directory
build/temp.linux-i686-2.6/check_libyaml.c: In function ‘main’:
build/temp.linux-i686-2.6/check_libyaml.c:5: error: ‘yaml_parser_t’ undeclared (first use in this function)
build/temp.linux-i686-2.6/check_libyaml.c:5: error: (Each undeclared identifier is reported only once
build/temp.linux-i686-2.6/check_libyaml.c:5: error: for each function it appears in.)
build/temp.linux-i686-2.6/check_libyaml.c:5: error: expected ‘;’ before ‘parser’
build/temp.linux-i686-2.6/check_libyaml.c:6: error: ‘yaml_emitter_t’ undeclared (first use in this function)
build/temp.linux-i686-2.6/check_libyaml.c:6: error: expected ‘;’ before ‘emitter’
build/temp.linux-i686-2.6/check_libyaml.c:8: warning: implicit declaration of function ‘yaml_parser_initialize’
build/temp.linux-i686-2.6/check_libyaml.c:8: error: ‘parser’ undeclared (first use in this function)
build/temp.linux-i686-2.6/check_libyaml.c:9: warning: implicit declaration of function ‘yaml_parser_delete’
build/temp.linux-i686-2.6/check_libyaml.c:11: warning: implicit declaration of function ‘yaml_emitter_initialize’
build/temp.linux-i686-2.6/check_libyaml.c:11: error: ‘emitter’ undeclared (first use in this function)
build/temp.linux-i686-2.6/check_libyaml.c:12: warning: implicit declaration of function ‘yaml_emitter_delete’

libyaml is not found or a compiler error: forcing --without-libyaml
(if libyaml is installed correctly, you may need to
 specify the option --include-dirs or uncomment and
 modify the parameter include_dirs in setup.cfg)
zip_safe flag not set; analyzing archive contents...
Adding PyYAML 3.08 to easy-install.pth file

Installed /usr/local/lib/python2.6/dist-packages/PyYAML-3.08-py2.6-linux-i686.egg
Finished processing dependencies for nltk==2.0b5



 Now you done, import the NLTK and start downloading the NTLK data.


$ python
Python 2.6.2+ (release26-maint, Jun 19 2009, 15:14:35) 
[GCC 4.4.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import nltk
>>> nltk.download()
NLTK Downloader
---------------------------------------------------------------------------
    d) Download      l) List      c) Config      h) Help      q) Quit
---------------------------------------------------------------------------
Downloader> l

Packages:
/usr/local/lib/python2.6/dist-packages/nltk-2.0b5-py2.6.egg/nltk/__init__.py:588: DeprecationWarning: object.__new__() takes no parameters
  [ ] maxent_ne_chunker... ACE Named Entity Chunker (Maximum entropy)
  [ ] abc................. Australian Broadcasting Commission 2006
  [ ] brown............... Brown Corpus
  [ ] alpino.............. Alpino Dutch Treebank
  [ ] cess_cat............ CESS-CAT Treebank
  [ ] brown_tei........... Brown Corpus (TEI XML Version)
  [ ] cmudict............. The Carnegie Mellon Pronouncing Dictionary (0.6)
  [ ] biocreative_ppi..... BioCreAtIvE (Critical Assessment of Information
                           Extraction Systems in Biology)
  [ ] cess_esp............ CESS-ESP Treebank
  [ ] chat80.............. Chat-80 Data Files
  [ ] city_database....... City Database
  [ ] conll2002........... CONLL 2002 Named Entity Recognition Corpus
  [ ] conll2000........... CONLL 2000 Chunking Corpus
  [ ] conll2007........... Dependency Treebanks from CoNLL 2007 (Catalan
                           and Basque Subset)
  [ ] dependency_treebank. Dependency Parsed Treebank
  [ ] floresta............ Portuguese Treebank
  [ ] genesis............. Genesis Corpus
  [ ] gazetteers.......... Gazeteer Lists
Hit Enter to continue: 

Read More

NLTK on Ubuntu Quick Start Guide

Update July 19, 2009 : You can now use nltk python egg instead, read the NLTK Installation with Python setuptools post.

While attending a short program in computational linguistics at Dravidian University, Dr. Arul introduced me to NLTK (Natural Language Toolkit). It was full two years before that I finally decided to have a close look at it. Like most linguists at the lab I used Perl programming language. With new version of NLTK 2.0 released last month, NLTK now works with python 2.6. Here a quick start guide for NLTK on Ubuntu Linux.

Installing NLTK on Ubuntu with Python 2.6

At the time of writing this post the Debian package on NLTK download page is built for Python 2.5. Ubuntu ships with Python 2.6 by default. So you need to download the source package from the NLTK download page.

NLTK needs some dependency modules, lets install them.

sudo apt-get install python-numpy python-matplotlib prover9

Uncompress the source package and run the NLTK setup.

$ unzip nltk-2.0b3.zip

$ cd nltk-2.0b3/

$ ls build LICENSE.txt nltk PKG-INFO README.txt setup.py yaml

$ sudo python setup.py install

After finishing the NLTK setup, you should download the NLTK data which contains various corpora, tagsets and treebank data etc.

$ python

Python 2.6.2+ (release26-maint, Jun 19 2009, 15:14:35)

[GCC 4.4.0] on linux2

Type "help", "copyright", "credits" or "license" for more information.

>>> import nltk

>>> nltk.download()



Learning NLTK

The best place to start is the NLTK book Natural Language Processing with Python Analyzing Text with the Natural Language Toolkit. The book is released under public domain, so you can read it online on NLTK website itself. I would recommand you to buy a copy of this book as the procceds will go into the future development of NLTK.

There aren't many videos about NLTK. I recently stumbled upon this video lecture by the trinity of NLTK Steven Bird, Ewan Klein, and Edward Loper.

If you are new to computational linguistics and need good grounding in this field you should also consider reading these texts.

Speech and Language Processing (2nd Edition)

Natural Language Understanding (2nd Edition)

Foundations of Statistical Natural Language Processing

Read More

OPEN SOURCE TOOLS FOR MACHINE TRANSLATION

The Third Machine Translation Marathon, which will take place
January 26-31 in Prague, Czech Republic, is hosting a Open Source
Convention to advance the state of the art in machine translation.

We invite developers of open source tools to present their work
and submit a paper of up to 10 pages that (a) describes the
underlying methodology and (b) instructions how to use the tools.
We are looking for stand-alone tools and extensions of existing
tools, such as the Moses open source systems. Accepted papers will
be presented during the MT Marathon and published as a special
issue of the Prague Bulletin of Mathematical Linguistics
(http://ufal.mff.cuni.cz/pbml).

Possible topics:

• training of machine translation models


• machine translation decoders


• tuning of machine translation systems


• evaluation of machine translation


• visualization, annotation or debugging tools


• tools for human translators


• interfaces for web-based services or APIs


• extensions of existing tools


• other tools for machine translation

Program will be selected from papers submitted by a program
committee that will ask additional reviewers from the PBML's
editorial board for at least two reviews per paper.

Important dates:

Deadline for paper submission: December 12, 2008

Notification of acceptance: December 19, 2008

Camera-ready paper due: January 8, 2009

Presentations: January 26-30, 2009 (at the MT Marathon in Prague)

Please send non-anonymized full-version submissions in .pdf format
to Philipp Koehn . Please use the PBML style files from http://ufal.mff.cuni.cz/pbml-instructions.html (follow the "short paper" track instructions) or download the style package directly from http://ufal.mff.cuni.cz/pbml-style-standard.html.

Program Committee

Philipp Koehn (chair)

Chris Callison-Bruch

Jan Hajic

About the MT Marathon

MT Marathon is organized yearly by the EuroMatrix machine
translation research project funded by the European Union under
its "Cooperation programme" as a STREP project FP6-IST-5-034291-STP. In January 2009, it will be third MT Marathon organized by EuroMatrix. MT Marathon consists of several
events taking place at the same place to allow for free flow of thoughts and exchange of information and experience: a "spring school" (this time more like a "winter school") with associated lab lessons, invited research talks, and a hands-on experience
with Open Source MT tools. Participants will also experience
evaluating Machine Translation systems (with some hands-on
experience in actual subjective evaluation of MT systems taking
part in the WMT 2009 competition - see more at http://www.statmt.org/wmt09/). This year, talks presenting some of the available OpenSource tools in more detail will also be planned throughout the week (see the call for papers). Please find more about the current MT Marathon and the previous ones at http://ufal.mff.cuni.cz/euromatrix/mtmarathon.

About Euromatrix

The EuroMatrix project (http://www.euromatrix.net) aims at a major push in machine translation (MT) technology applying the most advanced MT technologies systematically to all pairs of EU languages. Special attention is being paid to the languages of the new and near-term prospective member states. As part of this
application development, EuroMatrix designs and investigates novel combinations of statistical techniques and linguistic knowledge sources as well as hybrid MT architectures. EuroMatrix addresses urgent European economic and social needs by concentrating on European languages and on high-quality translation to be employed for the publication of technical, social, legal and political documents. EuroMatrix aims at enriching the statistical MT approach with novel learning paradigms and experiment with new combinations of methods and resources from statistical MT,rule-based MT, shallow language processing and computational lexicography/morphology.

The main objectives of the project are:

• Translation systems for all pairs of EU languages, with a
  special focus on the languages of new and near-term prospective
  member states


• Efficient inclusion of linguistic knowledge into statistical
  machine translation


• The development and testing of hybrid architectures for the
  integration of rule-based and statistical approaches


• Organization, analysis and interpretation of a competitive
  annual international evaluation of machine translation with a
  strong focus on European economic and social needs


• The provision of open source machine translation technology
  including research tools, software and data


• A systematically compiled and constantly updated detailed survey
  of the state of MT technology for all EU language pairs based on
  the developed systematic translation between all EU languages,
  the comparative MT evaluations and an inventory of available and
  needed tools, components, lingware and data.

Read More

THIRD MACHINE TRANSLATION MARATHON 2009

The MT Marathon 2009, organized by the Institute of Formal and
Applied Linguistics of the Charles University in Prague, Czech
Republic, is the third in a series of MT Marathons organized by
the EU Euromatrix research project on Machine Translation.

The EuroMatrix consortium invites researchers, developers,
students, and users of machine translation for participation. The
event will feature

• Winter School classes on current methods in statistical MT



• Research showcase



• Open source convention on resources for machine translation,
  this time with an open call for papers (separate announcement
  will follow)



• Lab hands-on experience for system developers, students and
  programmers



• Workshop on evaluation of European language translation




• Where:
  Prague's Lesser Town, the newly renovated historical
  building of the Computer Science School of the Charles
  University in Prague



• When: January 26-31, 2009



• How: Registration is now possible!



• How much: Attendance is free of charge, but limited.

For more information and online registration please go to http://ufal.mff.cuni.cz/euromatrix/mtmarathon.

About the MT Marathon

MT Marathon is organized yearly by the EuroMatrix machine translation research project funded by the European Union under its "Cooperation programme" as a STREP project FP6-IST-5-034291-STP. In January 2009, it will be third MT Marathon organized by EuroMatrix. MT Marathon consists of several events taking place at the same place to allow for free flow of thoughts and exchange of information and experience: a "spring school" (this time more like a "winter school") with associated lab lessons, invited research talks, and a hands-on experience with Open Source MT tools. Participants will also experience evaluating Machine Translation systems (with some hands-on experience in actual subjective evaluation of MT systems taking part in the WMT 2009 competition - see more at http://www.statmt.org/wmt09/). This year, talks presenting some of the available OpenSource tools in more detail will also be planned throughout the week (see the call for papers). Please find more about the current MT Marathon and the previous ones at http://ufal.mff.cuni.cz/euromatrix/mtmarathon.

About Euromatrix

The EuroMatrix project (http://www.euromatrix.net) aims at a major push in machine translation (MT) technology applying the most advanced MT technologies systematically to all pairs of EU
languages. Special attention is being paid to the languages of the new and near-term prospective member states. As part of this application development, EuroMatrix designs and investigates novel
combinations of statistical techniques and linguistic knowledge sources as well as hybrid MT architectures. EuroMatrix addresses urgent European economic and social needs by concentrating on
European languages and on high-quality translation to be employed for the publication of technical, social, legal and political
documents. EuroMatrix aims at enriching the statistical MT
approach with novel learning paradigms and experiment with new
combinations of methods and resources from statistical MT,
rule-based MT, shallow language processing and computational
lexicography/morphology.

The main objectives of the project are:

• Translation systems for all pairs of EU languages, with a
  special focus on the languages of new and near-term prospective
  member states



• Efficient inclusion of linguistic knowledge into statistical
  machine translation



• The development and testing of hybrid architectures for the
  integration of rule-based and statistical approaches



• Organization, analysis and interpretation of a competitive
  annual international evaluation of machine translation with a
  strong focus on European economic and social needs



• The provision of open source machine translation technology
  including research tools, software and data



• A systematically compiled and constantly updated detailed survey
  of the state of MT technology for all EU language pairs based on
  the developed systematic translation between all EU languages,
  the comparative MT evaluations and an inventory of available and
  needed tools, components, lingware and data.

Read More

Apertium: Open source machine translation

If you are interested in Machine Translation(MT) then you should read this great introductory article on Apertium MT system. Well I didn't know that Apertium is the open-source Systran (the engine that powers Babelfish) Did you? .

At the time of writing this post, there are no stable language pair for Indian languages in Apertium, If you are interested in working on one, do send me a shout out.

Kudos to Jimmy O'Regan for writing this one.

Read More

Character Encoding Problem in Anusaaraka Urdu to Devanagari transliteration script

While test driving the latest firefox3 I found that the Anusaaraka Urdu to Devanagari transliteration system was not working as expected. After looking around I figured out that the CGI script was setting the page encoding to Western (ISO-8859-1) instead of standard unicode(UTF-8). Perhaps the developers should note that setting correct parameters in HTML/XHTML meta elements is not enough. You need to set the proper encoding via the HTTP headers returned by the Web server, in this case I guess the CGI script didn't do that and the web server sent the default headers.

Oops

Often character encoding issue like this is mistaken for missing fonts or worse lack of proper support in the web browser (sic).

After changing to Unicode (UTF-8)

Read More

Lttoolbox Install Notes

Lttoolbox is a lexical and morphological analyser package from the Apertium machine translation system. Here is a brief install notes taken from my NLP summer school lab notes.

Build Lttoolbox

Download the lttoolbox source package from sourceforge.


tar zxvf lttoolbox-xxxxx.tgz
cd lttoolbox-xxxxx/
make
make install


You will need root level permission to install other dependency packages for lttoolbox . GNU Debian users however should use the lttoolbox debian package available in stable/testing/unstable and also read the Installing Apertium on Debian documentation page for more information.

Usage

If the tool is installed properly then you can look up the program install path with which lt-comp command.

Here I am compiling the Hindi dictionary with a left-to-right option lr, the output is stored in binary format in file 'mydata'. This binary file is used by the lt-proc processor to do the morph analysis.


lt-comp lr hi.dix.noun_f mydata

lt-proc ./mydata



For more information do read the manual pages for the lt-comp and lt-proc tools or fire me a email, I would love to you help you.

Read More

Kompare to the rescue

If you have been reading this blog for the past week then you already know that I am in NLP Summer School in IIIT-Hyderabad. And while doing a project comparing the results of various dependency parsers we had to read through large text files to find the errors and its a real pain. Well, I strong believe in the motto of letting the computer do the dirty work. I would have preferred to use a editor like Vim or Emacs with multiple spilt windows to compare the files and their diff's output. However I found that the KDE's GUI based Kompare (frontend to diff, comp or cmp) to be better suited for this project since my team mates are linguists and quite new to gnu/Linux. Also almost all the lab machines have it installed KDE as default desktop envirnoment.

The above screenshot shows kompare in side-by-side mode and just by using arrow keys or the mouse we could manipulate both the windows at the same time and tally our results. It did help to finish the job quickly but its still a pain.

Read More

Off to Computational Linguistics workshop

I hope this post ends yet another spell of absense from writing to this blog. Thanks for bearing with me.

So, What's new? Well my tickets are booked and bags almost packed. I am off to the workshop I was planning to attend for so long. Guess, I'll meet some of you over there.

Workshop on
Advanced Course in Computational Linguistics 16-25, March 2008

Organized by

Department of Dravidian & Computational Linguistics

Dravidian University, Kuppam -517425

In collaboration with

Central Institute of Indian Languages (CIIL), Mysore
The Department

The Department of Dravidian and Computational Linguistics was established with a vision to fulfill the main objectives of the university and to meet the challenges of technological developments. It has identified the areas of Dravidian Linguistics and Computational Linguistics as the thrust areas of research. The Department offers part-time Certificate & Diploma courses in Dravidian Linguistics and Computational Linguistics.

The Department undertakes projects relating to Natural Language Processing such as building WordNets, Language Technology Tools, preparing Machine Readable Dictionaries and developing Machine Translation Systems pertaining to Dravidian Languages. At present, the Department is working on the Indian Language to Indian Language (Telugu-Tamil) Machine Translation System.

The Workshop

Development of language technology tools has emerged as a prime area of research in the technology age. That is why the role of language and the linguistic aspects have gained enormous importance. Various institutions in India are involved in the language technology activity, especially in developing tools in different Indian languages. The tools thus developed are intended to cater to the needs of researchers and language teachers working in the educational institutions all over the country. Further, the applicational aspects of these tools in areas such as language teaching, dictionary compilation, machine translation, etc. are yet to be utilized in a proper manner.

The Department organized a Workshop on Introduction to Computational Linguistics from 8-17th December 2006. It aimed at introducing the participants various computational techniques that are used to build language technology tools.

The present Workshop, Advanced Course in Computational Linguistics to be organized from 16-25, March 2008 will expose the participants to the advanced techniques in computational linguistics and its applications. This ten-day workshop will provide the participants an opportunity to get hands-on experience with the state-of-the-art systems.

A candidate who has attended the first workshop and participates in the advanced Workshop is eligible for a P.G. Diploma in Computational Linguistics provided he/she takes the requisite examination of two papers and submits a project work in the area of his choice in computational linguistics. A candidate who attends afresh for the Advanced Workshop will have to undergo the preparatory program of three-days prior to the Advanced Workshop will be awarded a Certificate in Computational Linguistics provided he clears the requisite two exams.

Course Content

The following areas would be covered in detail through lectures, labs and tutorials:

• Machine Translation



• Computational Semantics & ontology



• Information Extraction



• Bioinformatics & NLP



• Knowledge Representation and Reasoning



• Intelligent Tutoring Systems



• Artificial Intelligence in Education



• Dialogue Analysis



• Named Entity Recognition



• Cross Language Information Access



• Summarization



• Language Technology Policy and Planning

The Workshop Faculty

Prof. G. UmaMaheshwar Rao, Head, Centre for ALTS, University of Hyderabad, Hyderabad

Dr. Achuth Sankar S. Nair, Director, Centre for Bioinformatics, University of Kerala, Thiruvananthapuram

Dr. P. Umarani, Formerly Research Associate, University of Pittsburgh, USA

Mrs. Amba P. Kulkarni, Reader, Department of Sanskrit Studies, University of Hyderabad, Hyderabad.

Dr. L. Sobha, Research Scientist, Computational Linguistics Group, AU-KBC Research Centre, MIT Campus of Anna University, Chennai.

Dr. Vasudeva Varma, Associate Professor, IIIT, Hyderabad

Mr. S. Viswanathan, Senior Programmar, Capgemini, Chennai.

Read More