我正在将一堆RTF文件读入Python字符串中。在某些文本中,我会遇到以下错误:
Traceback (most recent call last):
File "11.08.py", line 47, in <module>
X = vectorizer.fit_transform(texts)
File "C:\Python27\lib\site-packages\sklearn\feature_extraction\text.py", line
716, in fit_transform
X = super(TfidfVectorizer, self).fit_transform(raw_documents)
File "C:\Python27\lib\site-packages\sklearn\feature_extraction\text.py", line
398, in fit_transform
term_count_current = Counter(analyze(doc))
File "C:\Python27\lib\site-packages\sklearn\feature_extraction\text.py", line
313, in <lambda>
tokenize(preprocess(self.decode(doc))), stop_words)
File "C:\Python27\lib\site-packages\sklearn\feature_extraction\text.py", line
224, in decode
doc = doc.decode(self.charset, self.charset_error)
File "C:\Python27\lib\encodings\utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x92 in position 462: invalid
start byte
我已经尝试过以下方法:
- 将文件的文本复制并粘贴到新文件中
- 将rtf文件保存为txt文件
- 使用Notepad++打开txt文件,选择“转换为utf-8”,并设置编码为utf-8
- 使用Microsoft Word打开文件并另存为新文件
可能与此无关,但以下是代码:
f = open(dir+location, "r")
doc = Rtf15Reader.read(f)
t = PlaintextWriter.write(doc).getvalue()
texts.append(t)
f.close()
vectorizer = TfidfVectorizer(sublinear_tf=True, max_df=0.5, stop_words='english')
X = vectorizer.fit_transform(texts)
X = vectorizer.fit_transform(texts.encode('utf-8'))
,如果我没记错的话,我总是搞不清何时使用.encode()
和.decode()
,只需尝试其中一个并查看发生了什么... - BrtH