logo标签列表

将浮点数向量编码为十六进制或Base64二进制的最快方法

c++vectorfloating-pointhexbyte
4

现在，我是这样做的：

//encoding
std::vector<float> data = ...
const unsigned char* bytes = reinterpret_cast<const unsigned char*>(&data[0]);
std::vector<unsigned char> byteVec(bytes, bytes + sizeof(float) * data.size());
std::string newData = base64_encode(&byteVec[0], byteVec.size());



//decoding
std::vector<unsigned char> decodedData = base64_decode(newData);
unsigned char* bytes = &(decodedData[0]);    // point to beginning of memory
float* floatArray = reinterpret_cast<float*>(bytes);
std::vector<float> floatVec(floatArray, floatArray + decodedData.size() / sizeof(float));

编码需要0.04秒，解码需要0.08秒。这太长了。是否有更快的方法？
我正在使用一个在线找到的base64库，但如果使用十六进制有更快的方法，我绝对愿意尝试！
我使用的函数是在this answer中找到的。
编辑：我也找不到一种将此向量转换为/从十六进制的方法，因此任何解决方案都将受到赞赏。
编辑：所有时间都在编码/解码函数中。其中没有矢量/数组或字节/浮点数的转换。
@Holt 是的，这个编码/解码是有效的。为什么看起来不对？ - anc
2
@anc，你处理了多少数据？你使用什么样的硬件？时间实际上花在哪里？分配内存？复制？转换？验证？如果你真的想要提高性能，我建议你编写一个处理类似于你实际工作负载的小型完整程序，并开始进行分析-修改-再次分析，直到达到可接受的性能水平。 - yzt
1
除非您提供一个最小化、完整、可重现的示例（MCVE），否则这样的问题很难得到答案。我们可以进行各种猜测，但我们无法说出任何具体的东西。 - Passer By
@anc，我无法使用您提供的代码并查看您所声称的结果。那是不完整且不可验证的。 - Passer By
你可以在GitHub上搜索任何开源的base64库。https://github.com/search?l=C%2B%2B&o=desc&q=base64&s=stars&type=Repositories&utf8=%E2%9C%93 - Sahib Yar
显示剩余7条评论
2个回答
2
这里是将十六进制转换为字符串的代码。
在我的机器上，编码比你提供的链接快12倍，解码快3.3倍。
如果您愿意放宽一些要求，可以使解码速度快30％，但需要使用字符[a-p]而不是十六进制格式。
解码程序中没有错误检查，可以通过降低速度来实现。
注意：您是否启用了优化编译？我问这个问题是因为我只有在禁用优化时才能复现您的计时结果。启用优化后，编码比解码慢。禁用优化后，编码比解码快2倍（与您的结果相同）。
#include <stdio.h>
#include <string.h>
#include <string>
#include <vector>

void hexEncode(void *output, const void *input, size_t size) {
    static const char table[256][2] = {
        {'0','0'},{'0','1'},{'0','2'},{'0','3'},{'0','4'},{'0','5'},{'0','6'},{'0','7'},{'0','8'},{'0','9'},{'0','a'},{'0','b'},{'0','c'},{'0','d'},{'0','e'},{'0','f'},
        {'1','0'},{'1','1'},{'1','2'},{'1','3'},{'1','4'},{'1','5'},{'1','6'},{'1','7'},{'1','8'},{'1','9'},{'1','a'},{'1','b'},{'1','c'},{'1','d'},{'1','e'},{'1','f'},
        {'2','0'},{'2','1'},{'2','2'},{'2','3'},{'2','4'},{'2','5'},{'2','6'},{'2','7'},{'2','8'},{'2','9'},{'2','a'},{'2','b'},{'2','c'},{'2','d'},{'2','e'},{'2','f'},
        {'3','0'},{'3','1'},{'3','2'},{'3','3'},{'3','4'},{'3','5'},{'3','6'},{'3','7'},{'3','8'},{'3','9'},{'3','a'},{'3','b'},{'3','c'},{'3','d'},{'3','e'},{'3','f'},
        {'4','0'},{'4','1'},{'4','2'},{'4','3'},{'4','4'},{'4','5'},{'4','6'},{'4','7'},{'4','8'},{'4','9'},{'4','a'},{'4','b'},{'4','c'},{'4','d'},{'4','e'},{'4','f'},
        {'5','0'},{'5','1'},{'5','2'},{'5','3'},{'5','4'},{'5','5'},{'5','6'},{'5','7'},{'5','8'},{'5','9'},{'5','a'},{'5','b'},{'5','c'},{'5','d'},{'5','e'},{'5','f'},
        {'6','0'},{'6','1'},{'6','2'},{'6','3'},{'6','4'},{'6','5'},{'6','6'},{'6','7'},{'6','8'},{'6','9'},{'6','a'},{'6','b'},{'6','c'},{'6','d'},{'6','e'},{'6','f'},
        {'7','0'},{'7','1'},{'7','2'},{'7','3'},{'7','4'},{'7','5'},{'7','6'},{'7','7'},{'7','8'},{'7','9'},{'7','a'},{'7','b'},{'7','c'},{'7','d'},{'7','e'},{'7','f'},
        {'8','0'},{'8','1'},{'8','2'},{'8','3'},{'8','4'},{'8','5'},{'8','6'},{'8','7'},{'8','8'},{'8','9'},{'8','a'},{'8','b'},{'8','c'},{'8','d'},{'8','e'},{'8','f'},
        {'9','0'},{'9','1'},{'9','2'},{'9','3'},{'9','4'},{'9','5'},{'9','6'},{'9','7'},{'9','8'},{'9','9'},{'9','a'},{'9','b'},{'9','c'},{'9','d'},{'9','e'},{'9','f'},
        {'a','0'},{'a','1'},{'a','2'},{'a','3'},{'a','4'},{'a','5'},{'a','6'},{'a','7'},{'a','8'},{'a','9'},{'a','a'},{'a','b'},{'a','c'},{'a','d'},{'a','e'},{'a','f'},
        {'b','0'},{'b','1'},{'b','2'},{'b','3'},{'b','4'},{'b','5'},{'b','6'},{'b','7'},{'b','8'},{'b','9'},{'b','a'},{'b','b'},{'b','c'},{'b','d'},{'b','e'},{'b','f'},
        {'c','0'},{'c','1'},{'c','2'},{'c','3'},{'c','4'},{'c','5'},{'c','6'},{'c','7'},{'c','8'},{'c','9'},{'c','a'},{'c','b'},{'c','c'},{'c','d'},{'c','e'},{'c','f'},
        {'d','0'},{'d','1'},{'d','2'},{'d','3'},{'d','4'},{'d','5'},{'d','6'},{'d','7'},{'d','8'},{'d','9'},{'d','a'},{'d','b'},{'d','c'},{'d','d'},{'d','e'},{'d','f'},
        {'e','0'},{'e','1'},{'e','2'},{'e','3'},{'e','4'},{'e','5'},{'e','6'},{'e','7'},{'e','8'},{'e','9'},{'e','a'},{'e','b'},{'e','c'},{'e','d'},{'e','e'},{'e','f'},
        {'f','0'},{'f','1'},{'f','2'},{'f','3'},{'f','4'},{'f','5'},{'f','6'},{'f','7'},{'f','8'},{'f','9'},{'f','a'},{'f','b'},{'f','c'},{'f','d'},{'f','e'},{'f','f'}
    };

    const unsigned char *s = static_cast<const unsigned char *>(input);
    char *t = static_cast<char *>(output);
    const unsigned char *e;

    e = static_cast<const unsigned char *>(input) + (size&~3);
    while (s<e) {
        int v0 = s[0];
        int v1 = s[1];
        int v2 = s[2];
        int v3 = s[3];

        memcpy(t+0, table[v0], 2);
        memcpy(t+2, table[v1], 2);
        memcpy(t+4, table[v2], 2);
        memcpy(t+6, table[v3], 2);
        s += 4;
        t += 8;
    }

    e = static_cast<const unsigned char *>(input) + size;
    while (s<e) {
        memcpy(t, table[*s], 2);
        s++;
        t += 2;
    }
}

void hexDecode(void *output, const void *input, size_t size) {
    static const signed char table[128] = {
        -1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,
        -1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,
        -1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,
         0, 1, 2, 3, 4, 5, 6, 7, 8, 9,-1,-1,-1,-1,-1,-1,
        -1,10,11,12,13,14,15,-1,-1,-1,-1,-1,-1,-1,-1,-1,
        -1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,
        -1,10,11,12,13,14,15,-1,-1,-1,-1,-1,-1,-1,-1,-1,
        -1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1
    };
    const unsigned char *s = static_cast<const unsigned char *>(input);
    char *t = static_cast<char *>(output);
    const unsigned char *e;

    e = static_cast<const unsigned char *>(input) + (size&~3);
    while (s<e) {
        int v0 = table[s[0]];
        int v1 = table[s[1]];
        int v2 = table[s[2]];
        int v3 = table[s[3]];

        t[0] = v0<<4|v1;
        t[1] = v2<<4|v3;

        s += 4;
        t += 2;
    }

    e = static_cast<const unsigned char *>(input) + size;
    while (s<e) {
        int v0 = table[s[0]];
        int v1 = table[s[1]];

        *t = v0<<4|v1;

        s += 2;
        t++;
    }
}

std::string hexEncodeToString(const void *input, size_t size) {
    std::string s;
    s.resize(size*2);
    // Note: const_cast'ing this maybe not standard conformant here
    hexEncode(const_cast<char *>(s.data()), input, size);
    return s;
}

std::vector<unsigned char> hexDecodeToVector(const std::string &input) {
    std::vector<unsigned char> s;
    s.resize(input.length()>>1);
    hexDecode(s.data(), input.data(), input.length());
    return s;
}


int main() {
    float in[2] = { 3.14f, 2.718f };

    std::vector<unsigned char> r = hexDecodeToVector(hexEncodeToString(in, sizeof(in)));
    const float *f = reinterpret_cast<float*>(r.data());

    for (size_t i=0; i<2; i++) {
        printf("%f\n", f[i]);
    }
}

这是一个[a-p]版本：
void apEncode(void *output, const void *input, size_t size) {
    static const char table[256][2] = {
        {'a','a'},{'a','b'},{'a','c'},{'a','d'},{'a','e'},{'a','f'},{'a','g'},{'a','h'},{'a','i'},{'a','j'},{'a','k'},{'a','l'},{'a','m'},{'a','n'},{'a','o'},{'a','p'},
        {'b','a'},{'b','b'},{'b','c'},{'b','d'},{'b','e'},{'b','f'},{'b','g'},{'b','h'},{'b','i'},{'b','j'},{'b','k'},{'b','l'},{'b','m'},{'b','n'},{'b','o'},{'b','p'},
        {'c','a'},{'c','b'},{'c','c'},{'c','d'},{'c','e'},{'c','f'},{'c','g'},{'c','h'},{'c','i'},{'c','j'},{'c','k'},{'c','l'},{'c','m'},{'c','n'},{'c','o'},{'c','p'},
        {'d','a'},{'d','b'},{'d','c'},{'d','d'},{'d','e'},{'d','f'},{'d','g'},{'d','h'},{'d','i'},{'d','j'},{'d','k'},{'d','l'},{'d','m'},{'d','n'},{'d','o'},{'d','p'},
        {'e','a'},{'e','b'},{'e','c'},{'e','d'},{'e','e'},{'e','f'},{'e','g'},{'e','h'},{'e','i'},{'e','j'},{'e','k'},{'e','l'},{'e','m'},{'e','n'},{'e','o'},{'e','p'},
        {'f','a'},{'f','b'},{'f','c'},{'f','d'},{'f','e'},{'f','f'},{'f','g'},{'f','h'},{'f','i'},{'f','j'},{'f','k'},{'f','l'},{'f','m'},{'f','n'},{'f','o'},{'f','p'},
        {'g','a'},{'g','b'},{'g','c'},{'g','d'},{'g','e'},{'g','f'},{'g','g'},{'g','h'},{'g','i'},{'g','j'},{'g','k'},{'g','l'},{'g','m'},{'g','n'},{'g','o'},{'g','p'},
        {'h','a'},{'h','b'},{'h','c'},{'h','d'},{'h','e'},{'h','f'},{'h','g'},{'h','h'},{'h','i'},{'h','j'},{'h','k'},{'h','l'},{'h','m'},{'h','n'},{'h','o'},{'h','p'},
        {'i','a'},{'i','b'},{'i','c'},{'i','d'},{'i','e'},{'i','f'},{'i','g'},{'i','h'},{'i','i'},{'i','j'},{'i','k'},{'i','l'},{'i','m'},{'i','n'},{'i','o'},{'i','p'},
        {'j','a'},{'j','b'},{'j','c'},{'j','d'},{'j','e'},{'j','f'},{'j','g'},{'j','h'},{'j','i'},{'j','j'},{'j','k'},{'j','l'},{'j','m'},{'j','n'},{'j','o'},{'j','p'},
        {'k','a'},{'k','b'},{'k','c'},{'k','d'},{'k','e'},{'k','f'},{'k','g'},{'k','h'},{'k','i'},{'k','j'},{'k','k'},{'k','l'},{'k','m'},{'k','n'},{'k','o'},{'k','p'},
        {'l','a'},{'l','b'},{'l','c'},{'l','d'},{'l','e'},{'l','f'},{'l','g'},{'l','h'},{'l','i'},{'l','j'},{'l','k'},{'l','l'},{'l','m'},{'l','n'},{'l','o'},{'l','p'},
        {'m','a'},{'m','b'},{'m','c'},{'m','d'},{'m','e'},{'m','f'},{'m','g'},{'m','h'},{'m','i'},{'m','j'},{'m','k'},{'m','l'},{'m','m'},{'m','n'},{'m','o'},{'m','p'},
        {'n','a'},{'n','b'},{'n','c'},{'n','d'},{'n','e'},{'n','f'},{'n','g'},{'n','h'},{'n','i'},{'n','j'},{'n','k'},{'n','l'},{'n','m'},{'n','n'},{'n','o'},{'n','p'},
        {'o','a'},{'o','b'},{'o','c'},{'o','d'},{'o','e'},{'o','f'},{'o','g'},{'o','h'},{'o','i'},{'o','j'},{'o','k'},{'o','l'},{'o','m'},{'o','n'},{'o','o'},{'o','p'},
        {'p','a'},{'p','b'},{'p','c'},{'p','d'},{'p','e'},{'p','f'},{'p','g'},{'p','h'},{'p','i'},{'p','j'},{'p','k'},{'p','l'},{'p','m'},{'p','n'},{'p','o'},{'p','p'}
    };

    const unsigned char *s = static_cast<const unsigned char *>(input);
    char *t = static_cast<char *>(output);
    const unsigned char *e;

    e = static_cast<const unsigned char *>(input) + (size&~3);
    while (s<e) {
        int v0 = s[0];
        int v1 = s[1];
        int v2 = s[2];
        int v3 = s[3];

        memcpy(t+0, table[v0], 2);
        memcpy(t+2, table[v1], 2);
        memcpy(t+4, table[v2], 2);
        memcpy(t+6, table[v3], 2);
        s += 4;
        t += 8;
    }

    e = static_cast<const unsigned char *>(input) + size;
    while (s<e) {
        memcpy(t, table[*s], 2);
        s++;
        t += 2;
    }
}

void apDecode(void *output, const void *input, size_t size) {
    const unsigned char *s = static_cast<const unsigned char *>(input);
    char *t = static_cast<char *>(output);
    const unsigned char *e;

    e = static_cast<const unsigned char *>(input) + (size&~3);
    while (s<e) {
        int v0 = s[0]-'a';
        int v1 = s[1]-'a';
        int v2 = s[2]-'a';
        int v3 = s[3]-'a';

        t[0] = v0<<4|v1;
        t[1] = v2<<4|v3;

        s += 4;
        t += 2;
    }

    e = static_cast<const unsigned char *>(input) + size;
    while (s<e) {
        int v0 = s[0]-'a';
        int v1 = s[1]-'a';

        *t = v0<<4|v1;

        s += 2;
        t++;
    }
}
非常感谢您的回答！我有这段代码来调用编码：std::vector<unsigned char> byteVec(bytes, bytes + sizeof(float) * data.size()); std::string str = ""; hexEncode(&str, &byteVec[0], byteVec.size()); - anc
但是我在encode的第一个while循环中，在memcpy部分收到了segfault。你知道这是为什么吗？ - anc
你需要预先分配字符串的空间，并使用该空间而不是&str。 str.resize(data.size()*2)，并使用const_cast<char *>(str.data())作为hexEncode的输出参数。我不确定这是否符合标准。也许你需要使用一个临时向量，然后用它来初始化字符串。 - geza
你能否为你的回答添加一个例子？虽然我不再收到分段错误，但从编码返回的是一个空字符串。 - anc
@anc：我添加了两个辅助函数和一个小例子main()。 - geza
显示剩余4条评论
0

请使用这个测试基准来测试，这是经过测试的代码。我对复制算法进行了基准测试，与移动相比，复制在一个100万元素数组上快了25毫秒。如果没有改进，那么您需要查看编码器和解码器函数。如果0.04秒太长，那么看起来您需要进行微观优化。也许制作一个接受const迭代器的编码器会产生积极的结果。

发布代码，我很乐意编辑以更好地回答。

auto data = std::vector<float>( (1024 * 1024), 0.1f );

auto i_ptr = reinterpret_cast<std::uint8_t *>( data.data() );
auto i_len = data.size() * sizeof( float );
auto str = _base64_encode( i_ptr, i_len );


auto out = _base64_decode( str );

auto o_ptr = reinterpret_cast<float *>( out.data() );
auto o_len = out.size() / sizeof( float );

auto a_dest = std::vector<float>{};
std::copy( o_ptr, o_ptr + o_len, std::back_inserter( a_dest ) );

编辑：

我已经研究了你提供的算法，发现有很多优化的空间。例如，在解码函数中，每次循环迭代都要进行12次查找和4个if语句，其中只有1个if语句没有在上一次迭代中执行。由于b4的结果会输入到b3中，因此可以有效地删除b4。

说实话，这更像是一个代码审查问题。我建议阅读算法库和Alexander Stepanov的《从数学到通用编程》。

谢谢您的回答！函数名前面的下划线是无意的吗？ - anc
没关系，有帮助吗？下划线只是我写私有函数的方式，私有变量后面加一个下划线。 - Treebeard
哦，好的！不过很遗憾，它并没有产生任何影响 :( - anc
你用什么来对此进行基准测试？你尝试过 Google Benchmark 吗？另外，你的编码和解码函数的代码是什么？ - Treebeard
我无法在问题中添加太多代码，但它与链接答案中的内容完全相同。 - anc
网页内容由stack overflow 提供, 点击上面的
可以查看英文原文，
原文链接