词形归并器需要正确的词性标记才能准确处理,如果使用默认设置的WordNetLemmatizer.lemmatize()
方法,则默认标记为名词。请参见https://github.com/nltk/nltk/blob/develop/nltk/stem/wordnet.py#L39。
为解决此问题,在执行词形归并之前,请始终对数据进行词性标注,例如:
>>> from nltk.stem import WordNetLemmatizer
>>> from nltk import pos_tag, word_tokenize
>>> wnl = WordNetLemmatizer()
>>> sent = 'This is a foo bar sentence'
>>> pos_tag(word_tokenize(sent))
[('This', 'DT'), ('is', 'VBZ'), ('a', 'DT'), ('foo', 'NN'), ('bar', 'NN'), ('sentence', 'NN')]
>>> for word, tag in pos_tag(word_tokenize(sent)):
... wntag = tag[0].lower()
... wntag = wntag if wntag in ['a', 'r', 'n', 'v'] else None
... if not wntag:
... lemma = word
... else:
... lemma = wnl.lemmatize(word, wntag)
... print lemma
...
This
be
a
foo
bar
sentence
请注意,'is -> be'的意思是。
>>> wnl.lemmatize('is')
'is'
>>> wnl.lemmatize('is', 'v')
u'be'
回答这个问题,可以用你提供的例子中的文字:
>>> sent = 'These sentences involves some horsing around'
>>> for word, tag in pos_tag(word_tokenize(sent)):
... wntag = tag[0].lower()
... wntag = wntag if wntag in ['a', 'r', 'n', 'v'] else None
... lemma = wnl.lemmatize(word, wntag) if wntag else word
... print lemma
...
These
sentence
involve
some
horse
around
请注意,WordNetLemmatizer存在一些小问题:
此外,NLTK的默认POS标记器正在经历一些重大变化以提高准确性:
如果你需要一个开箱即用的词形还原解决方案,可以看看https://github.com/alvations/pywsd,我已经添加了一些try-except语句来捕获WordNet中不存在的单词,详情请见https://github.com/alvations/pywsd/blob/master/pywsd/utils.py#L66