Let\u2019s try to explore what NLTK could do. First, we would need to install the library with the following code.<\/p>\n
\n
pip install -U nltk\r\n<\/code><\/pre>\n<\/div>\n <\/p>\n
Next, we would see what NLTK could do. First, NLTK can perform the tokenization process using the following code:<\/p>\n
\n
import nltk from nltk.tokenize\r\nimport word_tokenize\r\n\r\n# Download the necessary resources\r\nnltk.download('punkt')\r\n\r\ntext = \"The fruit in the table is a banana\"\r\ntokens = word_tokenize(text)\r\n\r\nprint(tokens)\r\n<\/code><\/pre>\n<\/div>\n <\/p>\n
\n
Output>> \r\n['The', 'fruit', 'in', 'the', 'table', 'is', 'a', 'banana']\r\n<\/code><\/pre>\n<\/div>\n <\/p>\n
Tokenization basically would divide each word in a sentence into individual data.<\/p>\n
With NLTK, we can also perform Part-of-Speech (POS) Tags on the text sample.<\/p>\n
\n
from nltk.tag import pos_tag\r\n\r\nnltk.download('averaged_perceptron_tagger')\r\n\r\ntext = \"The fruit in the table is a banana\"\r\npos_tags = pos_tag(tokens)\r\n\r\nprint(pos_tags)\r\n<\/code><\/pre>\n<\/div>\n <\/p>\n
\n
Output>>\r\n[('The', 'DT'), ('fruit', 'NN'), ('in', 'IN'), ('the', 'DT'), ('table', 'NN'), ('is', 'VBZ'), ('a', 'DT'), ('banana', 'NN')]\r\n<\/code><\/pre>\n<\/div>\n <\/p>\n
The output of the POS tagger with NLTK is each token and its intended POS tags. For example, the word Fruit is Noun (NN), and the word \u2018a\u2019 is Determinant (DT).<\/p>\n
It\u2019s also possible to perform Stemming and Lemmatization with NLTK. Stemming is reducing a word to its base form by cutting its prefixes and suffixes, while Lemmatization also transforms to the base form by considering the words’ POS and morphological analysis.<\/p>\n
\n
from nltk.stem import PorterStemmer, WordNetLemmatizer\r\nnltk.download('wordnet')\r\nnltk.download('punkt')\r\n\r\ntext = \"The striped bats are hanging on their feet for best\"\r\ntokens = word_tokenize(text)\r\n\r\n# Stemming\r\nstemmer = PorterStemmer()\r\nstems = [stemmer.stem(token) for token in tokens]\r\nprint(\"Stems:\", stems)\r\n\r\n# Lemmatization\r\nlemmatizer = WordNetLemmatizer()\r\nlemmas = [lemmatizer.lemmatize(token) for token in tokens]\r\nprint(\"Lemmas:\", lemmas)\r\n<\/code><\/pre>\n<\/div>\n <\/p>\n
\n
Output>> \r\nStems: ['the', 'stripe', 'bat', 'are', 'hang', 'on', 'their', 'feet', 'for', 'best']\r\nLemmas: ['The', 'striped', 'bat', 'are', 'hanging', 'on', 'their', 'foot', 'for', 'best']\r\n<\/code><\/pre>\n<\/div>\n <\/p>\n
You can see that the stemming and lentmatization processes have slightly different results from the words.<\/p>\n
That\u2019s the simple usage of NLTK. You can still do many things with them, but the above APIs are the most commonly used.<\/p>\n