关于 ASCII_8BIT
Encoding::ASCII_8BIT 是一种特殊的编码方式,通常用于字节串而不是字符串。但正如名称所示,它的ASCII字符范围内的字符被认为是ASCII字符。当您使用ASCII-8BIT字符与其他兼容ASCII字符时,这将非常有用。
背景
我想使用 ASCII_8BIT,因为我需要编码介于0x00(0d00)和0xff(0d255)之间的所有字符,即ASCII(0-127)加扩展ASCII(128-255)。 ASCII(编码方式,US-ASCII)是一种7位编码方式,仅识别ASCII(字符集)字符(0-127)。正如名称所示,我原本期望 ASCII_8BIT 将其扩展为 8 位以添加对 128-255 的支持。
问题
当我使用 chr 时,编码会自动设置为 ASCII_8BIT,但当我直接在字符串中放置一个介于128-255(0x80-0xff)之间的字符,然后询问编码是什么时,我得到的是 UTF-8,如果我尝试将其转换为 ASCII_8BIT 则会出现错误。
irb(main):014:0> 0x8f.chr
=> "\x8F"
irb(main):015:0> 0x8f.chr.encoding
=> #<Encoding:ASCII-8BIT>
irb(main):016:0> "\x8f".encode(Encoding::ASCII_8BIT)
Traceback (most recent call last):
5: from /usr/bin/irb:23:in `<main>'
4: from /usr/bin/irb:23:in `load'
3: from /usr/lib/ruby/gems/2.6.0/gems/irb-1.0.0/exe/irb:11:in `<top (required)>'
2: from (irb):16
1: from (irb):16:in `encode'
Encoding::InvalidByteSequenceError ("\x8F" on UTF-8)
irb(main):021:0> "\x8F".encoding
=> #<Encoding:UTF-8>
Ruby核心有bug吗?我需要能够对8之间的所有内容进行编码。
ASCII 8BIT的另一个名称是BINARY,因为正如前面的引用所述,它应该能够编码任何字节。
irb(main):035:0> Encoding::ASCII_8BIT.names
=> ["ASCII-8BIT", "BINARY"]
其他编码方式
只是告诉我使用其他编码方式并不是问题的答案,除非它能真正映射所有255个扩展ASCII字符。
- 我不想使用UTF-8,因为它是多字节而不是单字节编码。
- ISO/IEC 8859-1(Latin1,8位)仅包含191个字符(ASCII + 63个字符)
ISO字符集与代码页不同的一个显著区别在于,ISO标准中128至159号字符位置,即高阶位设置为ASCII控制字符的位置明确未使用和未定义,虽然它们经常被用于专有代码页中可打印的字符,这是违反ISO标准的行为几乎是普遍的。参考文献:扩展ASCII - ISO 8859和专有适配
- Windows-1252(CP-1252,8位)并不包含所有255个字符,而且与扩展ASCII有不同的映射。
Ruby中提供的编码方式:
irb(main):036:0> Encoding.name_list
=> ["ASCII-8BIT", "UTF-8", "US-ASCII", "UTF-16BE", "UTF-16LE", "UTF-32BE", "UTF-32LE", "UTF-16", "UTF-32", "UTF8-MAC", "EUC-JP", "Windows-31J", "Big5", "Big5-HKSCS", "Big5-UAO", "CP949", "Emacs-Mule", "EUC-KR", "EUC-TW", "GB2312", "GB18030", "GBK", "ISO-8859-1", "ISO-8859-2", "ISO-8859-3", "ISO-8859-4", "ISO-8859-5", "ISO-8859-6", "ISO-8859-7", "ISO-8859-8", "ISO-8859-9", "ISO-8859-10", "ISO-8859-11", "ISO-8859-13", "ISO-8859-14", "ISO-8859-15", "ISO-8859-16", "KOI8-R", "KOI8-U", "Shift_JIS", "Windows-1250", "Windows-1251", "Windows-1252", "Windows-1253", "Windows-1254", "Windows-1257", "BINARY", "IBM437", "CP437", "IBM737", "CP737", "IBM775", "CP775", "CP850", "IBM850", "IBM852", "CP852", "IBM855", "CP855", "IBM857", "CP857", "IBM860", "CP860", "IBM861", "CP861", "IBM862", "CP862", "IBM863", "CP863", "IBM864", "CP864", "IBM865", "CP865", "IBM866", "CP866", "IBM869", "CP869", "Windows-1258", "CP1258", "GB1988", "macCentEuro", "macCroatian", "macCyrillic", "macGreek", "macIceland", "macRoman", "macRomania", "macThai", "macTurkish", "macUkraine", "CP950", "Big5-HKSCS:2008", "CP951", "IBM037", "ebcdic-cp-us", "stateless-ISO-2022-JP", "eucJP", "eucJP-ms", "euc-jp-ms", "CP51932", "EUC-JIS-2004", "EUC-JISX0213", "eucKR", "eucTW", "EUC-CN", "eucCN", "GB12345", "CP936", "ISO-2022-JP", "ISO2022-JP", "ISO-2022-JP-2", "ISO2022-JP2", "CP50220", "CP50221", "ISO8859-1", "ISO8859-2", "ISO8859-3", "ISO8859-4", "ISO8859-5", "ISO8859-6", "Windows-1256", "CP1256", "ISO8859-7", "ISO8859-8", "Windows-1255", "CP1255", "ISO8859-9", "ISO8859-10", "ISO8859-11", "TIS-620", "Windows-874", "CP874", "ISO8859-13", "ISO8859-14", "ISO8859-15", "ISO8859-16", "CP878", "MacJapanese", "MacJapan", "ASCII", "ANSI_X3.4-1968", "646", "UTF-7", "CP65000", "CP65001", "UTF-8-MAC", "UTF-8-HFS", "UCS-2BE", "UCS-4BE", "UCS-4LE", "CP932", "csWindows31J", "SJIS", "PCK", "CP1250", "CP1251", "CP1252", "CP1253", "CP1254", "CP1257", "UTF8-DoCoMo", "SJIS-DoCoMo", "UTF8-KDDI", "SJIS-KDDI", "ISO-2022-JP-KDDI", "stateless-ISO-2022-JP-KDDI", "UTF8-SoftBank", "SJIS-SoftBank", "locale", "external", "filesystem", "internal"]
要进行比较的是Python编码,参考文献为https://docs.python.org/3/library/codecs.html#standard-encodings
考虑因素
通过阅读扩展ASCII-多字节字符编码,似乎唯一真正扩展ASCII编码是UTF-8,但它是多字节的。看起来也没有真正的单字节扩展ASCII编码存在。
从字节角度来看,可以使用任何8位(单字节)编码,如扩展ASCII-可用于计算机可读语言的用途中所说:
所有ASCII字节(0x00至0x7F)在所有扩展ASCII变体中具有相同的含义。
但问题在于像ISO-8859-1这样的实现特别未定义了某些字符范围,因此会导致错误。
irb(main):009:0> (0..255).map { |c| c.chr}.join.encode(Encoding::ISO_8859_1)
Traceback (most recent call last):
6: from /usr/bin/irb:23:in `<main>'
5: from /usr/bin/irb:23:in `load'
4: from /usr/lib/ruby/gems/2.6.0/gems/irb-1.0.0/exe/irb:11:in `<top (required)>'
3: from (irb):9
2: from (irb):9:in `rescue in irb_binding'
1: from (irb):9:in `encode'
Encoding::UndefinedConversionError ("\x80" to UTF-8 in conversion from ASCII-8BIT to UTF-8 to ISO-8859-1)
更新 - force_encoding
我发现了字符串方法force_encoding。
irb(main)> a = "\x8f"
=> "\x8F"
irb(main)> a.encoding
=> #<Encoding:UTF-8>
irb(main)> a.encode(Encoding::ASCII_8BIT)
Traceback (most recent call last):
5: from /usr/bin/irb:23:in `<main>'
4: from /usr/bin/irb:23:in `load'
3: from /usr/lib/ruby/gems/2.6.0/gems/irb-1.0.0/exe/irb:11:in `<top (required)>'
2: from (irb):42
1: from (irb):42:in `encode'
Encoding::InvalidByteSequenceError ("\x8F" on UTF-8)
irb(main)> a.force_encoding(Encoding::ASCII_8BIT)
=> "\x8F"
irb(main):040:0> a.encoding
=> #<Encoding:ASCII-8BIT>
使用
force_encoding
而不是encode
的危险在哪里?如果我意外传递了多字节字符,它只会转换为多个单字节字符吗?因此,如果确保所有传递给应用程序的字符都在扩展ASCII范围内(单字节),则不会有危险,但如果传递了一些UTF-8字符,则不安全并且会导致静默转换问题。irb(main):044:0> "\ud087".force_encoding(Encoding::ASCII_8BIT)
=> "\xED\x82\x87"
irb(main):045:0> "\ud087".bytes
=> [237, 130, 135]
更新 - 回答
@mu-is-too-short的回答和@ForeverZer0的评论建议我应该使用pack
和unpack
来处理原始字节,而不是使用编码并通过它来解决问题。
因此,与其使用编码并绕过它,
pattern = 'A' * 2606 + "\x8F\x35\x4A\x5F" + 'C' * 390
pattern.force_encoding(Encoding::ASCII_8BIT)
我应该直接使用字节
pattern = ['A'.ord] * 2606 + [0x8F, 0x35, 0x4A, 0x5F] + ['C'.ord] * 390
pattern = pattern.pack('C*')
或者这种更易读的语法
pattern = 'A'.bytes * 2606 + "\x8F\x35\x4A\x5F".bytes + 'C'.bytes * 390
pattern = pattern.pack('C*')
ASCII_8BIT
实际上不是一种编码,它更像是一种非编码,也没有“扩展ASCII字符”,它们未被正式定义。DOS ANSI(代码页437)是众多8位编码之一,还有Latin-1、Windows-1252等。你的源数据是哪种格式?如果你正在处理原始二进制数据,答案是BINARY
,默认情况下转换为ASCII_8BIT
,或者换句话说,它保留字节并不进行任何转换。 - tadmanforce_encoding
只是强制 Ruby 以不同的方式解释相同的数据,数据仍然保持不变,只是被以不同的方式查看。encode
实际上会转换数据并返回不同的数据。 - ForeverZer0.force_encoding(Encoding::ASCII)
或force_encoding(Encoding::ASCII_8BIT)
来确保发送原始字节而不是转换多字节,例如如果编码本来会自动设置为UTF-8。 - norajpack
和unpack
函数来确保获取的是原始二进制数据,而不是尝试使用编码。Ruby通常使用字符串表示原始数据很方便,但会使某些用例(比如你的情况)变得更加微妙。 - ForeverZer0