如何从句子中提取字符ngram？- Python

Question

如何从句子中提取字符ngram？- Python

6

下面的word2ngrams函数从一个单词中提取字符3元组：

>>> x = 'foobar'
>>> n = 3
>>> [x[i:i+n] for i in range(len(x)-n+1)]
['foo', 'oob', 'oba', 'bar']

这篇文章展示了单个单词的字符n-gram提取，具体内容可参考Quick implementation of character n-grams using python。但如果我有句子并想提取字符n-gram，除了反复调用word2ngram()之外，是否有更快的方法？

如何使用正则表达式实现相同的word2ngram和sent2ngram输出？速度会更快吗？

我尝试过：

import string, random, time
from itertools import chain

def word2ngrams(text, n=3):
  """ Convert word into character ngrams. """
  return [text[i:i+n] for i in range(len(text)-n+1)]

def sent2ngrams(text, n=3):
    return list(chain(*[word2ngrams(i,n) for i in text.lower().split()]))

def sent2ngrams_simple(text, n=3):
    text = text.lower()
    return [text[i:i+n] for i in range(len(text)-n+1) if not " " in text[i:i+n]]

# Generate 10000 random strings of length 100.
sents = [" ".join([''.join(random.choice(string.ascii_uppercase) for j in range(10)) for i in range(100)]) for k in range(100)]

start = time.time()
x = [sent2ngrams(i) for i in sents]
print time.time() - start        

start = time.time()
y = [sent2ngrams_simple(i) for i in sents]
print time.time() - start        

print x==y

[out]:

0.0205280780792
0.0271739959717
True

编辑

正则表达式方法看起来很优雅，但它的性能比迭代调用 word2ngram() 慢：

import string, random, time, re
from itertools import chain

def word2ngrams(text, n=3):
  """ Convert word into character ngrams. """
  return [text[i:i+n] for i in range(len(text)-n+1)]

def sent2ngrams(text, n=3):
    return list(chain(*[word2ngrams(i,n) for i in text.lower().split()]))

def sent2ngrams_simple(text, n=3):
    text = text.lower()
    return [text[i:i+n] for i in range(len(text)-n+1) if not " " in text[i:i+n]]

def sent2ngrams_regex(text, n=3):
    rgx = '(?=('+'\S'*n+'))'
    return re.findall(rgx,text)

# Generate 10000 random strings of length 100.
sents = [" ".join([''.join(random.choice(string.ascii_uppercase) for j in range(10)) for i in range(100)]) for k in range(100)]

start = time.time()
x = [sent2ngrams(i) for i in sents]
print time.time() - start        

start = time.time()
y = [sent2ngrams_simple(i) for i in sents]
print time.time() - start        

start = time.time()
z = [sent2ngrams_regex(i) for i in sents]
print time.time() - start  

print x==y==z

[I]:

0.0211708545685
0.0284190177917
0.0303599834442
True

- alvas

1个回答

网页内容由stack overflow 提供, 点击上面的

可以查看英文原文，
原文链接

- user557597 · Accepted Answer

为什么不直接使用 (?=(...))？

编辑同样的事情，但不包括空格 (?=(\S\S\S))
编辑2 你也可以只使用你想要的。例如：只使用字母数字 (?=([^\W_]{3}))

使用前瞻来捕获三个字符。然后引擎每次将位置向上移动1次进行匹配。然后捕获下一个3个字符。

foobar 的结果是
foo
oob
oba
bar

 # Compressed regex
 #  (?=(...))

 # Expanded regex
 (?=                   # Start Lookahead assertion
      (                     # Capture group 1 start
           .                     # dot - metachar, matches any character except newline
           .                     # dot - metachar
           .                     # dot - metachar
      )                     # Capture group 1 end
 )                     # End Lookahead assertion