Simple Sentence Detector and Tokenizer Using OpenNLP
May 9, 2013 15 Comments
Machine learning is a branch of artificial intelligence. In this we create and study about systems that can learn from data. We all learn from our experience or others experience. In machine learning, the system is also getting learned from some experience, which we feed as data.
So for getting an inference about something, first we train the system with some set of data. With that data, the system learns and will become capable to give inference for new data. This is the basic principal behind machine learning.
There are a lot of machine learning toolkits available. Here I am explaining a simple program by using Apache OpenNLP. OpenNLP library is a machine learning based toolkit which is made for text processing. A lot of components are available in this toolkit. Here I am explaining a simple sentence detector and a tokenizer using OpenNLP.
Sentence Detector
Download the en-sent.bin from the Apache OpenNLP website and add this to the class path.
public void SentenceSplitter()
{
SentenceDetector sentenceDetector = null;
InputStream modelIn = null;
try {
modelIn = getClass().getResourceAsStream("en-sent.bin");
final SentenceModel sentenceModel = new SentenceModel(modelIn);
modelIn.close();
sentenceDetector = new SentenceDetectorME(sentenceModel);
}
catch (final IOException ioe) {
ioe.printStackTrace();
}
finally {
if (modelIn != null) {
try {
modelIn.close();
} catch (final IOException e) {}
}
}
String sentences[]=(sentenceDetector.sentDetect("I am Amal. I am engineer. I like travelling and driving"));
for(int i=0; i<sentences.length;i++)
{
System.out.println(sentences[i]);
}
}
Instead of giving sentence inside the program, you can give it as an input file.
Tokenizer
Download the en-token.bin from the Apache OpenNLP website and add this to the class path.
public void Tokenizer() throws FileNotFoundException
{
//InputStream modelIn = new FileInputStream("en-token.bin");
InputStream modelIn=getClass().getResourceAsStream("en-token.bin");
try {
TokenizerModel model = new TokenizerModel(modelIn);
Tokenizer tokenizer = new TokenizerME(model);
String tokens[] = tokenizer.tokenize("Sample tokenizer program using java");
for(int i=0; i<tokens.length;i++)
{
System.out.println(tokens[i]);
}
}
catch (IOException e) {
e.printStackTrace();
}
finally {
if (modelIn != null) {
try {
modelIn.close();
}
catch (IOException e) {
}
}
}
}
