From: rushan chen <chenrslucky@gmail.com>
To: Mark H Weaver <mhw@netris.org>
Cc: guile-devel@gnu.org
Subject: Re: a passionate guy who want to join in as a developer
Date: Sun, 12 Aug 2012 22:31:22 +0800 [thread overview]
Message-ID: <CAP-p8Ea_aaH2kU6C7zu+tgSn9RTN0=PX-b-iRYNz-rx=r_9Lfg@mail.gmail.com> (raw)
In-Reply-To: <5024F643.8040706@netris.org>
[-- Attachment #1: Type: text/plain, Size: 2015 bytes --]
Hi Mark,
Very appreciate for your reply.
I see you mention that it's useful to implement a larger library of
efficient data structure, and I'm interested in that very much. I used to
work on projects which involve complicated but very interesting data
structures, implementing them could be challenging, but once done I feel a
great sense of achievement.
One such project is implementing a language model (LM) which is a core
component of speech recognition and machine translation. I don't know if
you heard of it before. Unfortunately, I can't cover it too detailed here,
that would complicate things too much.
Basically, one of the key operations LM supports is it should return a
probability associated with any given id sequence. All id sequences are of
the same length, and there are a mass amount of such id sequences (a
commonly-seen LM may contain billions of them). So it's required to store
LM in a concise way, and at the same time make the search for each id
sequence very quickly.
Trie is finally chosen to be the data structure for LM (there were many
papers discussing this issue). All id sequences with the same prefix share
the same internal node, for example, for <1, 2, 3, 4> and <1, 2, 3, 5>,
only one copy of <1, 2, 3> will be stored in LM, and a search for a id
sequence is done by a sequence of binary search until the leaf is met. One
extra thing worth mentioning is that I store the whole trie structure in a
single large piece of memory (usually around 2 gigabytes), which makes
it convenient to write out to disk and load into memory by simply using
mmap, and I think it also makes the system faster than if you allocate
memory every time it's needed.
There are some other projects I worked or working on like Spell Corrector,
which also involve complicated data structures, but due to privacy policy,
I can't say much about it.
All in all, I'm very interested in it, and I really really hope I can help.
Looking forward to your reply. Thanks in advance.
Have fun!
Rushan Chen
[-- Attachment #2: Type: text/html, Size: 2332 bytes --]
next prev parent reply other threads:[~2012-08-12 14:31 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-08-06 14:50 a passionate guy who want to join in as a developer rushan chen
2012-08-10 11:53 ` Mark H Weaver
2012-08-12 14:31 ` rushan chen [this message]
2012-08-13 10:18 ` nalaginrut
2012-08-13 10:41 ` Tristan Colgate
2012-08-13 15:47 ` rushan chen
2012-08-13 16:47 ` Optimizing vlists Ludovic Courtès
2012-12-13 8:58 ` a passionate guy who want to join in as a developer Abhinav Jauhri
2012-12-13 15:31 ` Ludovic Courtès
2012-12-13 23:32 ` Abhinav Jauhri
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://www.gnu.org/software/guile/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAP-p8Ea_aaH2kU6C7zu+tgSn9RTN0=PX-b-iRYNz-rx=r_9Lfg@mail.gmail.com' \
--to=chenrslucky@gmail.com \
--cc=guile-devel@gnu.org \
--cc=mhw@netris.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).