Chapter
1
Introduction
rmmseg-cpp is a high performance Chinese word segmentation utility for
Ruby. It features full Ferret integration
as well as support for normal Ruby program usage.
rmmseg-cpp is a re-written of the original
RMMSeg gem in C++. RMMSeg is written
in pure Ruby. Though I tried hard to tweak RMMSeg, it just consumes
lots of memory and the segmenting process is rather slow.
The interface is almost identical to RMMSeg but the performance is
much better. This gem is always preferable in production
use. However, if you want to understand how the MMSEG segmenting
algorithm works, the source code of RMMSeg is a better choice than
this.
Chapter
2
Setup
2.1 Requirements
Your system needs the following software to run RMMSeg.
| Software | Notes |
|---|---|
| Ruby | Version 1.8.x is required |
| RubyGems | rmmseg-cpp is released as a gem |
| g++ | Used to build the native extension |
2.2 Installation
2.2.2 From Git
To build the gem manually from the latest source code. You’ll
need to have git and rake installed.
Warning 1. The latest source code may be unstable
code from the repository might still be broken sometimes.
It is generally not recommended to follow the source code.
The source code of rmmseg-cpp is hosted at
GitHub. You can get the
source code by git clone: git clone git://github.com/pluskid/rmmseg-cpp.git
then you can use Rake to build and install the gem:
cd rmmseg-cpp rake gem:install
Chapter
3
Usage
3.1 Stand Alone rmmseg
rmmseg-cpp comes with a script rmmseg. To get the basic usage, just execute it with -h option:
rmmseg -h
It reads from STDIN and print result to STDOUT. Here is a real
example:
3.2 Use in Ruby program
3.2.1 Initialize
To use rmmseg-cpp in Ruby program, you’ll first load it with RubyGems:
require 'rubygems' require 'rmmseg'
Then you may customize the dictionaries used by rmmseg-cpp
(see the rdoc on
how to add your own dictionaries) and load all dictionaries:
RMMSeg::Dictionary.load_dictionaries
Now rmmseg-cpp will be ready to do segmenting.
3.2.2 Ferret Integration
To use rmmseg-cpp with Ferret, you’ll need to require the
Ferret support of rmmseg-cpp (Of course you’ll also have to
got Ferret installed. If you have problems running the belowing
example, please try to update to the latest version of both
Ferret and rmmseg-cpp first):
require 'rmmseg/ferret'
rmmseg-cpp comes with a ready to use Ferret analyzer:
analyzer = RMMSeg::Ferret::Analyzer.new { |tokenizer| Ferret::Analysis::LowerCaseFilter.new(tokenizer) } index = Ferret::Index::Index.new(:analyzer => analyzer)
A complete example can be found in misc/ferret_example.rb. The result
of running that example is shown in Figure 1. Ferret Example Screenshot.
3.2.3 Normal Ruby program
rmmseg-cpp can also be used in normal Ruby programs. Just create
an Algorithm object and call next_token until a nil is returned:
algor = RMMSeg::Algorithm.new(text) loop do tok = algor.next_token break if tok.nil? puts "#{tok.text} [#{tok.start}..#{tok.end}]" end
Chapter
4
Who use it
appear in this list, please contact me.
- JavaEye: One of the biggest software developper
community in China.
Chapter
5
Resources
- Project Home: The Project page at RubyForge.
- RDoc of rmmseg-cpp: The auto generated rdoc of RMMSeg.
- Free Mind: The author’s blog.
- Author’s Email: Contact me if you have any problem.
