有一个测试句子和一个参考句子。我如何编写Python脚本,以BLEU度量衡的形式衡量这两个句子之间的相似性?BLEU度量衡被用于自动机器翻译评估。
BLEU分数由两部分组成,修改后的精确度和长度惩罚。有关详细信息,请参见论文。
您可以在NLTK中使用nltk.align.bleu_score
模块。
以下是一个代码示例:
import nltk
hypothesis = ['It', 'is', 'a', 'cat', 'at', 'room']
reference = ['It', 'is', 'a', 'cat', 'inside', 'the', 'room']
#there may be several references
BLEUscore = nltk.translate.bleu_score.sentence_bleu([reference], hypothesis)
print(BLEUscore)
请注意,默认的BLEU分数使用n=4,其中包括从unigram到4-gram。如果您的句子小于4,则需要重置N值,否则将返回ZeroDivisionError: Fraction(0, 0)
错误。
因此,您应该像这样重置权重:
import nltk
hypothesis = ["open", "the", "file"]
reference = ["open", "file"]
#the maximum is bigram, so assign the weight into 2 half.
BLEUscore = nltk.translate.bleu_score.sentence_bleu([reference], hypothesis, weights = (0.5, 0.5))
print(BLEUscore)
nltk
下的BLEU模块来计算BLEU分数。请参见这里。pip install sacrebleu
from nltk.translate.bleu_score import sentence_bleu
import argparse
def argparser():
Argparser = argparse.ArgumentParser()
Argparser.add_argument('--reference', type=str, default='summaries.txt', help='Reference File')
Argparser.add_argument('--candidate', type=str, default='candidates.txt', help='Candidate file')
args = Argparser.parse_args()
return args
args = argparser()
reference = open(args.reference, 'r').readlines()
candidate = open(args.candidate, 'r').readlines()
if len(reference) != len(candidate):
raise ValueError('The number of sentences in both files do not match.')
score = 0.
for i in range(len(reference)):
score += sentence_bleu([reference[i].strip().split()], candidate[i].strip().split())
score /= len(reference)
print("The bleu score is: "+str(score))
使用命令 python file_name.py --reference file1.txt --candidate file2.txt
如果已知测试和参考句子,我可以展示如何计算BLEU分数的一些例子。
你甚至可以将这两个句子作为字符串输入并转换为列表。
from nltk.translate.bleu_score import sentence_bleu
reference = [['the', 'cat',"is","sitting","on","the","mat"]]
test = ["on",'the',"mat","is","a","cat"]
score = sentence_bleu( reference, test)
print(score)
from nltk.translate.bleu_score import sentence_bleu
reference = [['the', 'cat',"is","sitting","on","the","mat"]]
test = ["there",'is',"cat","sitting","cat"]
score = sentence_bleu( reference, test)
print(score)
class BLEU(tf.keras.metrics.Metric):
def __init__(self, name='bleu_score'):
super(BLEU, self).__init__()
self.bleu_score = 0
def update_state(self, y_true, y_pred, sample_weight=None):
y_pred = tf.argmax(y_pred, -1)
self.bleu_score = 0
for i, j in zip(y_pred, y_true):
tf.autograph.experimental.set_loop_options()
total_words = tf.math.count_nonzero(i)
total_matches = 0
for word in i:
if word == 0:
break
for q in range(len(j)):
if j[q] == 0:
break
if word == j[q]:
total_matches += 1
j = tf.boolean_mask(j, [False if y == q else True for y in range(len(j))])
break
self.bleu_score += total_matches / total_words
def result(self):
return self.bleu_score / BATCH_SIZE