在这个链接中,它说截断的MD5是均匀分布的。我想用PySpark来验证它,并首先在Python中创建了1,000,000个UUID,如下所示。然后从MD5中截取了前三个字符。但是,我得到的图形与均匀分布的累积分布函数不相似。我尝试了UUID1和UUID4,结果也类似。如何正确地确认截断MD5的均匀分布?
import uuid
import numpy as np
import matplotlib.pyplot as plt
from statsmodels.distributions.empirical_distribution import ECDF
import pandas as pd
import pyspark.sql.functions as f
%matplotlib inline
### Generate 1,000,000 UUID1
uuid1 = [str(uuid.uuid1()) for i in range(1000000)] # make a UUID based on the host ID and current time
uuid1_df = pd.DataFrame({'uuid1':uuid1})
uuid1_spark_df = spark.createDataFrame(uuid1_df)
uuid1_spark_df = uuid1_spark_df.withColumn('hash', f.md5(f.col('uuid1')))\
.withColumn('truncated_hash3', f.substring(f.col('hash'), 1, 3))
count_by_truncated_hash3_uuid1 = uuid1_spark_df.groupBy('truncated_hash3').count()
uuid1_count_list = [row[1] for row in count_by_truncated_hash3_uuid1.collect()]
ecdf = ECDF(np.array(uuid1_count_list))
plt.figure(figsize = (14, 8))
plt.plot(ecdf.x,ecdf.y)
plt.show()
编辑: 我添加了直方图。如下图所示,它看起来更像正态分布。
plt.figure(figsize = (14, 8))
plt.hist(uuid1_count_list)
plt.title('Histogram of counts in each truncated hash')
plt.show()