我制作了这个脚本，让所有人都能理解如何进行分词，这样他们就可以自己构建自然语言处理引擎。
import re
from contextlib import redirect_stdout
from io import StringIO
example = 'Mary had a little lamb, Jack went up the hill, Jill followed suit, i woke up suddenly, it was a really bad dream...'
def token_to_sentence(str):
f = StringIO()
with redirect_stdout(f):
regex_of_sentence = re.findall('([\w\s]{0,})[^\w\s]', str)
regex_of_sentence = [x for x in regex_of_sentence if x is not '']
for i in regex_of_sentence:
print(i)
first_step_to_sentence = (f.getvalue()).split('\n')
g = StringIO()
with redirect_stdout(g):
for i in first_step_to_sentence:
try:
regex_to_clear_sentence = re.search('\s([\w\s]{0,})', i)
print(regex_to_clear_sentence.group(1))
except:
print(i)
sentence = (g.getvalue()).split('\n')
return sentence
def token_to_words(str):
f = StringIO()
with redirect_stdout(f):
for i in str:
regex_of_word = re.findall('([\w]{0,})', i)
regex_of_word = [x for x in regex_of_word if x is not '']
for word in regex_of_word:
print(regex_of_word)
words = (f.getvalue()).split('\n')
我做了一个不同的过程，我重新从段落开始处理，以使每个人更加理解文字处理。 段落到过程如下：
example = 'Mary had a little lamb, Jack went up the hill, Jill followed suit, i woke up suddenly, it was a really bad dream...'
将段落分割为句子。
sentence = token_to_sentence(example)
英译中：
将产生结果：
['Mary had a little lamb', 'Jack went up the hill', 'Jill followed suit', 'i woke up suddenly', 'it was a really bad dream']
将其分词：
words = token_to_words(sentence)
将会导致：
['Mary', 'had', 'a', 'little', 'lamb', 'Jack', 'went, 'up', 'the', 'hill', 'Jill', 'followed', 'suit', 'i', 'woke', 'up', 'suddenly', 'it', 'was', 'a', 'really', 'bad', 'dream']
我会解释这是如何工作的。
首先，我使用正则表达式搜索所有单词和空格，这些空格分隔单词并在找到标点符号之前停止。 正则表达式是：
([\w\s]{0,})[^\w\s]{0,}
因此，计算将考虑括号中的单词和空格。
'(Mary had a little lamb),( Jack went up the hill, Jill followed suit),( i woke up suddenly),( it was a really bad dream)...'
结果仍不明确，包含一些“None”字符。因此我使用了这个脚本来删除“None”字符：
[x for x in regex_of_sentence if x is not '']
所以该段落将被分解成句子，但是不是清晰的句子，结果为：
['Mary had a little lamb', ' Jack went up the hill', ' Jill followed suit', ' i woke up suddenly', ' it was a really bad dream']
如您所见，结果显示有些句子以空格开头。因此，为了使段落更清晰而不需要以空格开头，我使用了这个正则表达式：
\s([\w\s]{0,})
它将会生成一个清晰的句子，例如：
['Mary had a little lamb', 'Jack went up the hill', 'Jill followed suit', 'i woke up suddenly', 'it was a really bad dream']
所以，我们必须进行两个步骤才能得到好的结果。
你问题的答案从这里开始...
为了将句子分词，我遍历整段话并使用正则表达式来捕获单词。正则表达式如下：
([\w]{0,})
再次使用以下方式清除空白字符：
[x for x in regex_of_word if x is not '']
因此，结果非常清晰，只有单词列表。
['Mary', 'had', 'a', 'little', 'lamb', 'Jack', 'went, 'up', 'the', 'hill', 'Jill', 'followed', 'suit', 'i', 'woke', 'up', 'suddenly', 'it', 'was', 'a', 'really', 'bad', 'dream']
在未来，要制作一个好的自然语言处理系统，你需要拥有自己的短语数据库，并搜索该短语是否出现在句子中，然后列出短语列表，剩下的单词就是普通单词。
通过这种方法，我可以建立我的本地语言（印度尼西亚语）的自然语言处理系统，因为它缺乏很多模块。
修改后：
我没有看到您想要比较单词的问题。所以您还有另一个句子可以比较…… 我会给你额外的奖励，不仅如此，我还会告诉你如何计算它。
mod_example = ["'Mary' 'had' 'a' 'little' 'lamb'" , 'Jack' 'went' 'up' 'the' 'hill']
在这种情况下，你必须执行以下步骤：
1. 迭代mod_example
2. 将第一句话与mod_example中的单词进行比较。
3. 进行一些计算。
因此，脚本将是：
import re
from contextlib import redirect_stdout
from io import StringIO
example = 'Mary had a little lamb, Jack went up the hill, Jill followed suit, i woke up suddenly, it was a really bad dream...'
mod_example = ["'Mary' 'had' 'a' 'little' 'lamb'" , 'Jack' 'went' 'up' 'the' 'hill']
def token_to_sentence(str):
f = StringIO()
with redirect_stdout(f):
regex_of_sentence = re.findall('([\w\s]{0,})[^\w\s]', str)
regex_of_sentence = [x for x in regex_of_sentence if x is not '']
for i in regex_of_sentence:
print(i)
first_step_to_sentence = (f.getvalue()).split('\n')
g = StringIO()
with redirect_stdout(g):
for i in first_step_to_sentence:
try:
regex_to_clear_sentence = re.search('\s([\w\s]{0,})', i)
print(regex_to_clear_sentence.group(1))
except:
print(i)
sentence = (g.getvalue()).split('\n')
return sentence
def token_to_words(str):
f = StringIO()
with redirect_stdout(f):
for i in str:
regex_of_word = re.findall('([\w]{0,})', i)
regex_of_word = [x for x in regex_of_word if x is not '']
for word in regex_of_word:
print(regex_of_word)
words = (f.getvalue()).split('\n')
def convert_to_words(str):
sentences = token_to_sentence(str)
for i in sentences:
word = token_to_words(i)
return word
def compare_list_of_words__to_another_list_of_words(from_strA, to_strB):
fromA = list(set(from_strA))
for word_to_match in fromA:
totalB = len(to_strB)
number_of_match = (to_strB).count(word_to_match)
data = str((((to_strB).count(word_to_match))/totalB)*100)
print('words: -- ' + word_to_match + ' --' + '\n'
' number of match : ' + number_of_match + ' from ' + str(totalB) + '\n'
' percent of match : ' + data + ' percent')
if __name__ == '__main__':
getsentences = token_to_sentence(example)
getwords = token_to_words(getsentences)
compare_list_of_words__to_another_list_of_words(getwords, mod_example)
