Golang：读取Gzip Reader时出现意外的EOF错误

Question

我目前正在一个项目中工作，在这个项目中，客户端python脚本使用gzip包压缩输入，然后根据AES加密压缩的数据，最终将加密数据编码为base64编码方案。在服务器端，使用golang编写，我按照相反的顺序执行相同的步骤以达到原始数据。

从输出结果可以看出，我可以对来自客户端的数据进行解码、解密和解压缩，但是读取gzip.NewReader类型时会抛出“unexpected EOF error”的错误（请参见问题底部的输出）。

到目前为止，我没有去除填充就完成了以上操作，但是它给我带来了“invalid gzip header error”的错误。为了验证有效的gzip Reader与脚本创建的Reader之间是否有任何显着差异，我将原始数据压缩成压缩形式。显然，从输出中我无法看出任何值得注意的差异，但可能由于读取器创建的时间变化而导致字节上的差异。

Original data: {"i": "MB88", "p": [{"d": [54.3, 0, 99, 49.35, 3, 99, 51.533, 1, 98, 28964, 7348, 43590, 97107, 10350, 55200, 49.7], "t": 1499089212.136789, "dt": "p"}]}

Gzipped data: "\x1f\x8b\x08\x00\xf9w[Y\x02\xff%\x8e=\x0e\xc30\x08F\xaf\x82\x98\x91\x05\xe6\xc7\xa6c\xf7\x9e\xa0\xca\x96\xa5[\x86lQ\xee^\xdcN\xf0\xf4\xc1\x83\x0b?\xf8\x00|=\xe7D\x02<\n\xde\x17\xee\xab\xb85%\x82L\x02\xcb\xa6N\xa0\x7fri\xae\xd5K\xe1$\xe83\xc3\x08\x86Z\x81\xa9g-y\x88\xf6\x9a\xf5E\xde\x99\x7f\x96\xb1\xd5\x99\xb3\xfcb\x99\x121D\x1bG\xe7^.\xdcWPO\xdc\xdb\xfd\x05\x0ev\x15\x1d\x99\x00\x00\x00"

Gzip Reader created from the request by client

GzReader: &{{ [] 2017-07-03 15:40:12 +0200 CEST 255} 0xc42007e150 0xc4200a8000 0 0 [31 139 8 0 60 73 90 89 2 255 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] true}

Gzip Reader created by directly reading the original data

GzReader2: &{{ [] 2017-07-04 13:11:53 +0200 CEST 255} 0xc4200741e0 0xc4200a9300 0 0 [31 139 8 0 249 119 91 89 2 255 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] true}

现在，我很好奇是什么导致了这个错误，以及如何解决这个问题。由于某些限制，我无法在此处发布原始脚本，但这些是我目前正在测试代码的实际测试脚本。我试图提供尽可能多的信息，但如果您需要更多解释，请随时问我。如果你能帮我解决这个问题，我会非常感激！

干杯！

客户端代码（测试人员）：

import time
import json
import requests
import random
import gzip
import base64

from Crypto.Cipher import AES

baseurl = 'http://0.0.0.0:80'
key = b'TfvY7I358yospfWKcoviZizOShpm5hyH'
iv = b'mb13KcoviZizvYhp'

MODE = AES.MODE_CFB
BLOCK_SIZE = 16
SEGMENT_SIZE = 128


def http_post(url, data):
    print('Going to make a request to {} with the following data: {}'.format(url, data))
    r = requests.post(url,
                      data=data,
                      headers={'Content-type': 'application/json; charset=utf-8',
                               'Connection': 'keep-alive'}, )
    if r.status_code != 200:
        print('Server returned unexpected response code {}, and content: {}'.format(r.status_code, r.content))
        return False
    else:
        data = r.json()
        return data


def post_sample_data(key, iv):
    fake_device_id = "MB88"
    Load_VA_Total_Mean = random.randint(1000, 100000)
    print('Data that should come back: {}'.format(Load_VA_Total_Mean))
    data = {"i": fake_device_id, "p": [
        {"d": [54.3, 0, 99, 49.35, 3, 99, 51.533, 1, 98, 28964, 7348, 43590, Load_VA_Total_Mean, 10350, 55200, 49.7],
         "t": time.time(), "dt": "p"}]}
    url = baseurl + '/realtimedata'
    encryption_key_reference = 1
    payload = '{}\n{}'.format(convert_pack(data, key, iv), encryption_key_reference)
    return http_post(url, payload)


def check_sample_request():
    fake_device_id = "MB88"
    url = baseurl + '/datapoints/powerzones/status'
    data = [{"name": "P1", "field": "Load_VA_Total_Mean", "devices": [{"id": fake_device_id, "multiplier": 1}]}]
    return http_post(url, json.dumps(data))


def convert_pack(inputdict, key, iv):
    print('input: {}'.format(inputdict))
    print('key: {}'.format(key))

    jsonpayload = json.dumps(inputdict)  # encode dict to json string
    print('json: {}'.format(jsonpayload))

    gzippayload = gzip.compress(jsonpayload.encode('utf-8'))  # compress with gzip
    print('gzip: {}'.format(gzippayload))

    encryptedpayload = base64.b64encode(encrypt(key, iv, gzippayload))
    print('encrypted: {}'.format(encryptedpayload))
    return str(encryptedpayload, encoding='utf-8')    


def _pad_string(value):
    length = len(value)
    pad_size = BLOCK_SIZE - (length % BLOCK_SIZE)
    return value.ljust(length + pad_size, b'\x00')


def encrypt(key, iv, plaintext):
    aes = AES.new(key, MODE, iv, segment_size=SEGMENT_SIZE)
    plaintext = _pad_string(plaintext)
    encrypted_text = aes.encrypt(plaintext)
    return encrypted_text


post_sample_data(key, iv)
check_sample_request()

服务器代码（测试人员）：

package main

import (
    "crypto/aes"
    "crypto/cipher"
    "encoding/base64"
    "fmt"
    "compress/gzip"
    "io/ioutil"
    "bytes"
    "strings"
)

func main() {
    //a sample input text generated by the client script
    originalText := "xhhpLqMJs4g8+8Hb751bcalljkeHDJfbGdMqr9E9im1OkiJzmdLr7pWv/Grh0+uzdeMQBMzefqzTrzetf1fZcBm93IEF6wl8WYE5x2uAZ5XY39bxlRWDt4cTNk/wWisbAMSzNJqQUq2UmqBLaWcI0LU1NGPZd8EKHK36Iwlp8NuP+YsQIJSmQdN2cvNFOLGv"
    fmt.Println("Original request: ", originalText)
    key := []byte("TfvY7I358yospfWKcoviZizOShpm5hyH")
    text := decrypt(key, originalText)

    gzReader, err1 := gzip.NewReader(bytes.NewReader(text))
    fmt.Println("GzReader: ", gzReader) 
    if err1 != nil {
        fmt.Println("error1: ", err1)
    }

    content, err2 := ioutil.ReadAll(gzReader)   
    if err2 != nil {
        fmt.Println("error2: ",err2)
    }

    fmt.Printf("\nReading: \n -> Content in bytes: %v \n -> Content in string: %s\n", content, string(content)) 
}

func decrypt(key []byte, cryptoText string) []byte {
    ciphertext, _ := base64.StdEncoding.DecodeString(cryptoText)
    fmt.Printf("\nDecoding: \n -> Original: %v \n -> Decoded in bytes: %v \n -> Decoded in string: %s\n", cryptoText, ciphertext, ciphertext)
    block, err := aes.NewCipher(key)
    if err != nil {
        panic(err)
    }

    iv := []byte("mb13KcoviZizvYhp")    
    stream := cipher.NewCFBDecrypter(block, iv)
    stream.XORKeyStream(ciphertext, ciphertext)
    fmt.Printf("\nDecrypting: \n -> Decrypted in bytes: %v \n -> Decrypted in string: %s\n", ciphertext, ciphertext)

    ciphertext = bytes.Trim(ciphertext, "\x00")
    fmt.Printf("\nTrimming the padding: \n -> Decrypted in bytes: %v \n -> Decrypted in string: %s\n\n", ciphertext, ciphertext)    
    return ciphertext
}

终端输出：

Original request:

xhhpLqMJs4g8+8Hb751bcalljkeHDJfbGdMqr9E9im1OkiJzmdLr7pWv/Grh0+uzdeMQBMzefqzTrzetf1fZcBm93IEF6wl8WYE5x2uAZ5XY39bxlRWDt4cTNk/wWisbAMSzNJqQUq2UmqBLaWcI0LU1NGPZd8EKHK36Iwlp8NuP+YsQIJSmQdN2cvNFOLGv

Decoding:

-> Original: xhhpLqMJs4g8+8Hb751bcalljkeHDJfbGdMqr9E9im1OkiJzmdLr7pWv/Grh0+uzdeMQBMzefqzTrzetf1fZcBm93IEF6wl8WYE5x2uAZ5XY39bxlRWDt4cTNk/wWisbAMSzNJqQUq2UmqBLaWcI0LU1NGPZd8EKHK36Iwlp8NuP+YsQIJSmQdN2cvNFOLGv

-> Decoded in bytes: [198 24 105 46 163 9 179 136 60 251 193 219 239 157 91 113 169 101 142 71 135 12 151 219 25 211 42 175 209 61 138 109 78 146 34 115 153 210 235 238 149 175 252 106 225 211 235 179 117 227 16 4 204 222 126 172 211 175 55 173 127 87 217 112 25 189 220 129 5 235 9 124 89 129 57 199 107 128 103 149 216 223 214 241 149 21 131 183 135 19 54 79 240 90 43 27 0 196 179 52 154 144 82 173 148 154 160 75 105 103 8 208 181 53 52 99 217 119 193 10 28 173 250 35 9 105 240 219 143 249 139 16 32 148 166 65 211 118 114 243 69 56 177 175]

-> Decoded in string: �i.� ��<����[q�e�G� ���*��=�mN�"s����j���u���~�ӯ7�W�p�܁� |Y�9�k�g��������6O�Z+ij4��R����Kiе54c�w� ��# i�ۏ�� ��A�vr�E8��

Decrypting:

-> Decrypted in bytes: [31 139 8 0 60 73 90 89 2 255 37 142 49 14 2 65 8 69 175 66 168 201 4 6 88 192 210 222 19 152 237 182 177 179 176 219 236 221 101 180 130 151 15 15 78 124 225 13 240 113 207 68 2 124 55 60 79 60 86 113 27 74 192 4 85 4 86 67 157 64 255 228 50 92 187 151 198 36 152 89 155 17 132 90 131 169 215 90 10 225 232 9 86 111 114 159 204 63 75 236 125 230 211 126 177 42 206 154 50 135 232 22 217 90 60 86 208 79 92 251 245 5 234 198 128 61 153 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] �q�D|7

Trimming the padding:

-> Decrypted in bytes: [31 139 8 0 60 73 90 89 2 255 37 142 49 14 2 65 8 69 175 66 168 201 4 6 88 192 210 222 19 152 237 182 177 179 176 219 236 221 101 180 130 151 15 15 78 124 225 13 240 113 207 68 2 124 55 60 79 60 86 113 27 74 192 4 85 4 86 67 157 64 255 228 50 92 187 151 198 36 152 89 155 17 132 90 131 169 215 90 10 225 232 9 86 111 114 159 204 63 75 236 125 230 211 126 177 42 206 154 50 135 232 22 217 90 60 86 208 79 92 251 245 5 234 198 128 61 153] �q�D|7

GzReader: &{{ [] 2017-07-03 15:40:12 +0200 CEST 255} 0xc42007e150 0xc4200a8000 0 0 [31 139 8 0 60 73 90 89 2 255 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] true}

error2: unexpected EOF

Reading:

-> Content in bytes: [123 34 105 34 58 32 34 77 66 56 56 34 44 32 34 112 34 58 32 91 123 34 100 34 58 32 91 53 52 46 51 44 32 48 44 32 57 57 44 32 52 57 46 51 53 44 32 51 44 32 57 57 44 32 53 49 46 53 51 51 44 32 49 44 32 57 56 44 32 50 56 57 54 52 44 32 55 51 52 56 44 32 52 51 53 57 48 44 32 57 55 49 48 55 44 32 49 48 51 53 48 44 32 53 53 50 48 48 44 32 52 57 46 55 93 44 32 34 116 34 58 32 49 52 57 57 48 56 57 50 49 50 46 49 51 54 55 56 57 44 32 34 100 116 34 58 32 34 112 34 125 93 125]

-> Content in string: {"i": "MB88", "p": [{"d": [54.3, 0, 99, 49.35, 3, 99, 51.533, 1, 98, 28964, 7348, 43590, 97107, 10350, 55200, 49.7], "t": 1499089212.136789, "dt": "p"}]}

- bkaankuguoglu

Answer 1

当你认为在“修剪填充”时，你实际上是删除了属于gzip流结尾的三个零。仅仅因为它是一个零并不意味着它是填充。请保留153后面的三个零。

当你不修剪填充时，gzip解码器将寻找成功解压第一个之后的另一个gzip流。如果看到零而不是gzip头，则会导致无效的gzip头错误。

您需要跟踪gzip流的大小，以便知道哪些是填充物，哪些不是。或者您可以一次解码一个gzip流，当您看到以下零时，您可以得出结论那就是填充。

- Mark Adler

---

感谢您的反馈！在阅读您的评论之前，我尝试了另一种方法，基本上是避免错误，但实际上似乎有效。我修改了代码中的错误处理程序如下： if err2 != nil && err2.Error() != "gzip: invalid header" { fmt.Println("error2: ",err2) } 然后我删除了删除填充行的代码： ciphertext = bytes.Trim(ciphertext, "\x00") 老实说，我仍然很困惑这是如何工作的。我的方法在长期运行中可能会引起问题吗？ - bkaankuguoglu

这是一个不好的想法，因为它会忽略一个真正的无效头错误，其中提供的数据根本不是gzip流。 - Mark Adler