wordpunct_tokenize
是基于简单正则表达式的分词方法。其定义如下:
wordpunct_tokenize = WordPunctTokenizer().tokenize
\w+|[^\w\s]+
来分割输入内容。word_tokenize
基于TreebankWordTokenizer
,请参见文档此处。它基本上像Penn Treebank一样对文本进行标记化。这里有一个愚蠢的例子,应该说明两者的区别。sent = "I'm a dog and it's great! You're cool and Sandy's book is big. Don't tell her, you'll regret it! 'Hey', she'll say!"
>>> word_tokenize(sent)
['I', "'m", 'a', 'dog', 'and', 'it', "'s", 'great', '!', 'You', "'re",
'cool', 'and', 'Sandy', "'s", 'book', 'is', 'big', '.', 'Do', "n't", 'tell',
'her', ',', 'you', "'ll", 'regret', 'it', '!', "'Hey", "'", ',', 'she', "'ll", 'say', '!']
>>> wordpunct_tokenize(sent)
['I', "'", 'm', 'a', 'dog', 'and', 'it', "'", 's', 'great', '!', 'You', "'",
're', 'cool', 'and', 'Sandy', "'", 's', 'book', 'is', 'big', '.', 'Don',
"'", 't', 'tell', 'her', ',', 'you', "'", 'll', 'regret', 'it', '!', "'",
'Hey', "',", 'she', "'", 'll', 'say', '!']
wordpunct_tokenize
函数会在所有特殊符号处进行分割,并将它们作为独立的单元处理。而 word_tokenize
则会保留像 're
这样的内容。但是它似乎并不太智能,因为我们可以看到它未能将初始单引号与 'Hey'
分开。sent = 'I\'m a dog and it\'s great! You\'re cool and Sandy\'s book is big. Don\'t tell her, you\'ll regret it! "Hey", she\'ll say!'
>>> word_tokenize(sent)
['I', "'m", 'a', 'dog', 'and', 'it', "'s", 'great', '!', 'You', "'re",
'cool', 'and', 'Sandy', "'s", 'book', 'is', 'big', '.', 'Do', "n't",
'tell', 'her', ',', 'you', "'ll", 'regret', 'it', '!', '``', 'Hey', "''",
',', 'she', "'ll", 'say', '!']
所以,word_tokenize
会分离双引号,但它也将它们转换为 ``
和 ''
。而 wordpunct_tokenize
则不会这样做:
>>> wordpunct_tokenize(sent)
['I', "'", 'm', 'a', 'dog', 'and', 'it', "'", 's', 'great', '!', 'You', "'",
're', 'cool', 'and', 'Sandy', "'", 's', 'book', 'is', 'big', '.', 'Don',
"'", 't', 'tell', 'her', ',', 'you', "'", 'll', 'regret', 'it', '!', '"',
'Hey', '",', 'she', "'", 'll', 'say', '!']