我正在尝试对一些文档进行标记化,但是出现了以下错误:
UnicodeDecodeError: 'ascii' 编解码器无法解码第6位的0xef字节:序数不在128的范围内
请注意,保留了HTML标签。
import nltk
import pandas as pd
df = pd.DataFrame(pd.read_csv('status2.csv'))
documents = df['status']
result = [nltk.word_tokenize(sent) for sent in documents]
我认为这是一个Unicode问题,所以我添加了。
documents = unicode(documents, 'utf-8')
另一个错误
类型错误:需要字符串或缓冲区,但发现Series
print documents
1 Brandon Cachia ,All I know is that,you're so n...
2 Melissa Zejtunija:HAM AND CHEESE BIEX INI??? *...
3 .........Where is my mind?????
4 Having a philosophical discussion with Trudy D...