如何“规范化”Python 3的Unicode字符串

Question

我需要比较两个字符串。其中aa是从PDF文件中提取的（使用pdfminer/chardet），而bb是键盘输入。如何规范化第一个字符串以进行比较？

>>> aa = "ā"
>>> bb = "ā"
>>> aa == bb
False
>>> 
>>> aa.encode('utf-8')
b'\xc4\x81'
>>> bb.encode('utf-8')
b'a\xcc\x84'

- rudensm

Answer 1

你可以使用unicodedata.normalize函数进行规范化：

>>> aa = b'\xc4\x81'.decode('utf8')   # composed form
>>> bb = b'a\xcc\x84'.decode('utf8')  # decomposed form
>>> aa
'ā'
>>> bb
'ā'
>>> aa == bb
False
>>> import unicodedata as ud
>>> aa == ud.normalize('NFC',bb)  # compare composed
True
>>> ud.normalize('NFD',aa) == bb  # compare decomposed
True

- Mark Tolonen