我正在尝试从特定的单词列中计算单词频率。
我想从字典中删除停用词。
以下是代码:
代码
import unicodecsv as csv
import nltk
import pandas as pd
import chardet
from nltk.corpus import stopwords
from nltk.tokenize import sent_tokenize
with open('data.csv','rb') as f:
result = chardet.detect(f.read())
file_band = file[file['article'].str.contains("first time")]
file.loc[:,'extracted'] = file_band['article']
top_N = 200
a = file_band['extracted'].str.lower().replace(r'\|', ' ').str.cat(sep=' ')
words = nltk.tokenize.word_tokenize(a)
word_dist = nltk.FreqDist(words)
print (word_dist)
stop_words = set(stopwords.words('english'))
word_tokens = word_tokenize(word_dist)
filtered_sentence = [w for w in word_tokens if not w in stop_words]
filtered_sentence = []
for w in word_tokens:
if w not in stop_words:
filtered_sentence.append(w)
print(word_tokens)
print(filtered_sentence)
错误
问题出现在以下错误上:
问题出在第29行,因为“word_tokenize”未定义。请确保您已正确导入所需的库或模块,并且拼写正确。NameError Traceback (most recent call last) in () 27 #过滤单词 28 stop_words = set(stopwords.words('english')) ---> 29 word_tokens = word_tokenize(word_dist) 30 31 filtered_sentence = [w for w in word_tokens if not w in stop_words]
NameError:名称 'word_tokenize' 未定义
word_tokenize
函数。 - Aran-Feyword_tokenize
时,为什么要导入sent_tokenize
? - cs95nltk.tokenize.word_tokenize(a)
之前,之后你忘记了它,直接使用word_tokenize(word_dist)
是行不通的。 - Sqoshu