Lemmatisation
Encyclopedia
Lemmatisation in linguistics
, is the process of grouping together the different inflected forms of a word so they can be analysed as a single item.
In computational linguistics
, lemmatisation is the algorithmic process of determining the lemma for a given word. Since the process may involve complex tasks such as understanding context and determining the part of speech of a word in a sentence (requiring, for example, knowledge of the grammar
of a language) it can be a hard task to implement a lemmatiser for a new language.
In many languages, words appear in several inflected forms. For example, in English, the verb 'to walk' may appear as 'walk', 'walked', 'walks', 'walking'. The base form, 'walk', that one might look up in a dictionary, is called the lemma for the word. The combination of the base form with the part of speech is often called the lexeme
of the word.
Lemmatisation is closely related to stemming
. The difference is that a stemmer operates on a single word without knowledge of the context, and therefore cannot discriminate between words which have different meanings depending on part of speech. However, stemmers are typically easier to implement and run faster, and the reduced accuracy may not matter for some applications.
For instance:
Analysers like Lucene Snowball store the base stemmed format of the word without the knowledge of meaning, but taking into account the semantics of the word formation only. The stemmed word itself might not be a valid word: 'lazy', as seen in the example below, is stemmed by many stemmers to 'lazi'. This is because the purpose of stemming is not to produce the appropriate lemma - that is a more challenging task that requires knowledge of context. The main purpose of stemming is to map different forms of a word to a single form, and as a relatively simple, rules-based algorithm, it makes the above-mentioned sacrifice to ensure that, for example, when 'laziness' is stemmed to 'lazi', it has the same stem as 'lazy'.
The following is an example of lemmatisation and stemming. Given the following sentence:
[The] [quick] [brown] [fox] [jumps] [over] [the] [lazy] [dog]
, gives the following stems when set to perform stemming:
[quick] [brown] [fox] [jump] [over] [lazy] [dog]
the lemmas from the words in the sentence would be as follows:
[the] [quick] [brown] [fox] [jump] [over] [the] [lazy] [dog]
Linguistics
Linguistics is the scientific study of human language. Linguistics can be broadly broken into three categories or subfields of study: language form, language meaning, and language in context....
, is the process of grouping together the different inflected forms of a word so they can be analysed as a single item.
In computational linguistics
Computational linguistics
Computational linguistics is an interdisciplinary field dealing with the statistical or rule-based modeling of natural language from a computational perspective....
, lemmatisation is the algorithmic process of determining the lemma for a given word. Since the process may involve complex tasks such as understanding context and determining the part of speech of a word in a sentence (requiring, for example, knowledge of the grammar
Grammar
In linguistics, grammar is the set of structural rules that govern the composition of clauses, phrases, and words in any given natural language. The term refers also to the study of such rules, and this field includes morphology, syntax, and phonology, often complemented by phonetics, semantics,...
of a language) it can be a hard task to implement a lemmatiser for a new language.
In many languages, words appear in several inflected forms. For example, in English, the verb 'to walk' may appear as 'walk', 'walked', 'walks', 'walking'. The base form, 'walk', that one might look up in a dictionary, is called the lemma for the word. The combination of the base form with the part of speech is often called the lexeme
Lexeme
A lexeme is an abstract unit of morphological analysis in linguistics, that roughly corresponds to a set of forms taken by a single word. For example, in the English language, run, runs, ran and running are forms of the same lexeme, conventionally written as RUN...
of the word.
Lemmatisation is closely related to stemming
Stemming
In linguistic morphology and information retrieval, stemming is the process for reducing inflected words to their stem, base or root form—generally a written word form. The stem need not be identical to the morphological root of the word; it is usually sufficient that related words map to the same...
. The difference is that a stemmer operates on a single word without knowledge of the context, and therefore cannot discriminate between words which have different meanings depending on part of speech. However, stemmers are typically easier to implement and run faster, and the reduced accuracy may not matter for some applications.
For instance:
- The word "better" has "good" as its lemma. This link is missed by stemming, as it requires a dictionary look-up.
- The word "walk" is the base form for word "walking", and hence this is matched in both stemming and lemmatisation.
- The word "meeting" can be either the base form of a noun or a form of a verb ("to meet") depending on the context, e.g., "in our last meeting" or "We are meeting again tomorrow". Unlike stemming, lemmatisation does select the right lemma depending on the context.
Analysers like Lucene Snowball store the base stemmed format of the word without the knowledge of meaning, but taking into account the semantics of the word formation only. The stemmed word itself might not be a valid word: 'lazy', as seen in the example below, is stemmed by many stemmers to 'lazi'. This is because the purpose of stemming is not to produce the appropriate lemma - that is a more challenging task that requires knowledge of context. The main purpose of stemming is to map different forms of a word to a single form, and as a relatively simple, rules-based algorithm, it makes the above-mentioned sacrifice to ensure that, for example, when 'laziness' is stemmed to 'lazi', it has the same stem as 'lazy'.
The following is an example of lemmatisation and stemming. Given the following sentence:
org.apache.lucene.analysis.snowball.SnowballAnalyzer, which also filters out stop wordsStop words
In computing, stop words are words which are filtered out prior to, or after, processing of natural language data . It is controlled by human input and not automated. There is not one definite list of stop words which all tools use, if even used...
, gives the following stems when set to perform stemming:
the lemmas from the words in the sentence would be as follows: