我开始试验Python和NLTK。我遇到了一个冗长的错误信息,无法找到解决方案,希望您能提供任何见解。
import nltk,csv,numpy
from nltk import sent_tokenize, word_tokenize, pos_tag
reader = csv.reader(open('Medium_Edited.csv', 'rU'), delimiter= ",",quotechar='|')
tokenData = nltk.word_tokenize(reader)
我正在运行Python 2.7和最新的nltk软件包,操作系统为OSX Yosemite。 这是我尝试过的两行代码,但结果并没有任何区别:
with open("Medium_Edited.csv", "rU") as csvfile:
tokenData = nltk.word_tokenize(reader)
我看到的错误信息如下:
Traceback (most recent call last):
File "nltk_text.py", line 11, in <module>
tokenData = nltk.word_tokenize(reader)
File "/Library/Python/2.7/site-packages/nltk/tokenize/__init__.py", line 101, in word_tokenize
return [token for sent in sent_tokenize(text, language)
File "/Library/Python/2.7/site-packages/nltk/tokenize/__init__.py", line 86, in sent_tokenize
return tokenizer.tokenize(text)
File "/Library/Python/2.7/site-packages/nltk/tokenize/punkt.py", line 1226, in tokenize
return list(self.sentences_from_text(text, realign_boundaries))
File "/Library/Python/2.7/site-packages/nltk/tokenize/punkt.py", line 1274, in sentences_from_text
return [text[s:e] for s, e in self.span_tokenize(text, realign_boundaries)]
File "/Library/Python/2.7/site-packages/nltk/tokenize/punkt.py", line 1265, in span_tokenize
return [(sl.start, sl.stop) for sl in slices]
File "/Library/Python/2.7/site-packages/nltk/tokenize/punkt.py", line 1304, in _realign_boundaries
for sl1, sl2 in _pair_iter(slices):
File "/Library/Python/2.7/site-packages/nltk/tokenize/punkt.py", line 310, in _pair_iter
prev = next(it)
File "/Library/Python/2.7/site-packages/nltk/tokenize/punkt.py", line 1278, in _slices_from_text
for match in self._lang_vars.period_context_re().finditer(text):
TypeError: expected string or buffer
感谢您的提前帮助。
code import csv import numpy as np from nltk import sent_tokenize, word_tokenize as word_tokenize, pos_tag reader = csv.reader(open('Milling_Final_Edited.csv', 'rU'), delimiter=',', quotechar='"') for line in reader: for field in line: tokens = word_tokenize(field) code
。</br> 我是Python和NLTK的新手,所以有些东西需要学习。现在我遇到了一个编码错误-code UnicodeDecodeError: 'ascii' codec can't decode byte 0xea in position 1: ordinal not in range(128). code
</br>但该文件是utf-8编码的。 - OAKimport codecs
并以codecs.open('Milling_Final_Edited.csv', 'rU', encoding="utf-8")
的方式打开文件。 - yvespeirsmanimport csv, codecs import numpy as np from nltk import sent_tokenize, word_tokenize as word_tokenize, pos_tag as pos_tag reader =codecs.open('Milling_Final_Edited.csv', 'r', encoding="utf-8", errors="ignore") for line in reader: for field in line: tokens = word_tokenize(field) posData = pos_tag(tokens) print(posData)
<br/>我不得不在codecs.open
中添加errors="ignore"
来解决编码错误,但现在我又遇到了另一个问题。输出是[(u'3', 'LS')]。实际文件字符前面的u字符是什么意思('3')?此外,输出只有25行中的1行。 - OAKu
表示这是一个 Unicode 字符串。我认为脚本只输出一行是因为缩进有问题。请确保print
语句与word_tokenize
调用处于相同的缩进级别。 - yvespeirsman