我是Python的新手,目前正在使用Python 2。我有一些源文件,每个文件都包含大量数据(约1900万行)。它看起来像下面这样:
apple \t N \t apple
n&apos
garden \t N \t garden
b\ta\md
great \t Adj \t great
nice \t Adj \t (unknown)
etc
我的任务是搜索每个文件的第三列,以查找目标词,并且每次在语料库中找到目标词时,都需要将该词前后10个单词添加到多维字典中。
编辑:应排除包含“&”、“\”或字符串“(unknown)”的行。
我尝试使用readlines()和enumerate()解决此问题,如下面的代码所示。该代码确实完成了它的任务,但显然对于提供的源文件中的数据量而言不够高效。
我知道不应该使用readlines()或read()处理大型数据集,因为它会将整个文件加载到内存中。尽管如此,逐行读取文件时,我无法使用枚举方法获取目标词前后的10个单词。我也不能使用mmap,因为我没有权限在该文件上使用它。
因此,我想readlines方法加上一些大小限制可能是最有效的解决方案。然而,这样做的话,每次达到大小限制时,目标词后面的10个单词都不会被捕获,代码就会停止。
def get_target_to_dict(file):
targets_dict = {}
with open(file) as f:
for line in f:
targets_dict[line.strip()] = {}
return targets_dict
targets_dict = get_target_to_dict('targets_uniq.txt')
# browse directory and process each file
# find the target words to include the 10 words before and after to the dictionary
# exclude lines starting with <,-,; to just have raw text
def get_co_occurence(path_file_dir, targets, results):
lines = []
for file in os.listdir(path_file_dir):
if file.startswith('corpus'):
path_file = os.path.join(path_file_dir, file)
with gzip.open(path_file) as corpusfile:
# PROBLEMATIC CODE HERE
# lines = corpusfile.readlines()
for line in corpusfile:
if re.match('[A-Z]|[a-z]', line):
if '(unknown)' in line:
continue
elif '\\' in line:
continue
elif '&' in line:
continue
lines.append(line)
for i, line in enumerate(lines):
line = line.strip()
if re.match('[A-Z][a-z]', line):
parts = line.split('\t')
lemma = parts[2]
if lemma in targets:
pos = parts[1]
if pos not in targets[lemma]:
targets[lemma][pos] = {}
counts = targets[lemma][pos]
context = []
# look at 10 previous lines
for j in range(max(0, i-10), i):
context.append(lines[j])
# look at the next 10 lines
for j in range(i+1, min(i+11, len(lines))):
context.append(lines[j])
# END OF PROBLEMATIC CODE
for context_line in context:
context_line = context_line.strip()
parts_context = context_line.split('\t')
context_lemma = parts_context[2]
if context_lemma not in counts:
counts[context_lemma] = {}
context_pos = parts_context[1]
if context_pos not in counts[context_lemma]:
counts[context_lemma][context_pos] = 0
counts[context_lemma][context_pos] += 1
csvwriter = csv.writer(results, delimiter='\t')
for k,v in targets.iteritems():
for k2,v2 in v.iteritems():
for k3,v3 in v2.iteritems():
for k4,v4 in v3.iteritems():
csvwriter.writerow([str(k), str(k2), str(k3), str(k4), str(v4)])
#print(str(k) + "\t" + str(k2) + "\t" + str(k3) + "\t" + str(k4) + "\t" + str(v4))
results = open('results_corpus.csv', 'wb')
word_occurrence = get_co_occurence(path_file_dir, targets_dict, results)
出于完整性的原因,我复制了代码的整个部分,因为它是创建多维字典的一个函数的一部分,该函数从提取的所有信息中创建多维字典并将其写入csv文件中。
我非常希望能得到任何提示或建议,使这段代码更加高效。
编辑 我已经更正了代码,以便考虑到目标单词前后确切的10个单词。
map
、filter
、groupby
和islice
高效地完成这个任务。 - Eli Korvigomap
,我肯定需要将corpusfile转换为列表,对吧? - dani_anyman