我正在编写一个用于对推文进行词干提取的代码,但在编码时遇到了编码问题。当我尝试应用Porter词干提取器时,它显示出错。可能是我没有正确对其进行分词处理。
我的代码如下...
import sys
import pandas as pd
import nltk
import scipy as sp
from nltk.classify import NaiveBayesClassifier
from nltk.stem import PorterStemmer
reload(sys)
sys.setdefaultencoding('utf8')
stemmer=nltk.stem.PorterStemmer()
p_test = pd.read_csv('TestSA.csv')
train = pd.read_csv('TrainSA.csv')
def word_feats(words):
return dict([(word, True) for word in words])
for i in range(len(train)-1):
t = []
#train.SentimentText[i] = " ".join(t)
for word in nltk.word_tokenize(train.SentimentText[i]):
t.append(stemmer.stem(word))
train.SentimentText[i] = ' '.join(t)
当我尝试执行时,返回错误:
UnicodeDecodeError Traceback (most recent call last)
<ipython-input-10-5aa856d0307f> in <module>()
23 #train.SentimentText[i] = " ".join(t)
24 for word in nltk.word_tokenize(train.SentimentText[i]):
---> 25 t.append(stemmer.stem(word))
26 train.SentimentText[i] = ' '.join(t)
27
/usr/lib/python2.7/site-packages/nltk/stem/porter.pyc in stem(self, word)
631 def stem(self, word):
632 stem = self.stem_word(word.lower(), 0, len(word) - 1)
--> 633 return self._adjust_case(word, stem)
634
635 ## --NLTK--
/usr/lib/python2.7/site-packages/nltk/stem/porter.pyc in _adjust_case(self, word, stem)
602 for x in range(len(stem)):
603 if lower[x] == stem[x]:
--> 604 ret += word[x]
605 else:
606 ret += stem[x]
/usr/lib64/python2.7/encodings/utf_8.pyc in decode(input, errors)
14
15 def decode(input, errors='strict'):
---> 16 return codecs.utf_8_decode(input, errors, True)
17
18 class IncrementalEncoder(codecs.IncrementalEncoder):
UnicodeDecodeError: 'utf8' codec can't decode byte 0xc3 in position 0: unexpected end of data
有人知道我的代码哪里出了问题吗?我被卡在这个错误上了。有什么建议吗?
sys.setdefaultencoding()
的“罪恶”: https://dev59.com/TW865IYBdhLWcg3wat2l - alvas