TBB可扩展分配器是否强调内存碎片?

4
我有一个视频检索系统,在检索过程中消耗大量内存。我知道tbb可扩展分配器将释放的内存释放到内存池中,而不是返回给操作系统。这是否意味着该池将始终在其池中具有先前分配的内存,并且当其他线程需要内存时,可能会导致内存耗尽?
我正在使用24个核心,47G内存的2台机器。我的程序有24个线程,每个线程处理一个检索任务,并使用tbb可扩展分配器进行内存分配,但仍然出现了bad alloc异常。我还使用valgrind检测内存泄漏,并找到下面的报告,似乎只有由tbb可扩展分配器引起的“仍然可达”问题,没有其他内存泄漏。有人可以向我展示如何解决这个问题吗?
==1224== HEAP SUMMARY:
==1224==     in use at exit: 147,456 bytes in 9 blocks
==1224==   total heap usage: 10 allocs, 1 frees, 148,480 bytes allocated
==1224== 
==1224== Thread 1:
==1224== 16,384 bytes in 1 blocks are still reachable in loss record 1 of 4
==1224==    at 0x4A0610C: malloc (vg_replace_malloc.c:195)
==1224==    by 0x4E285C6: rml::internal::getRawMemory(unsigned long, bool) (in /home/is_admin/tbb40_233oss/build/linux_intel64_gcc_cc4.1.2_libc2.5_kernel2.6.18_release/libtbbmalloc.so.2)
==1224==    by 0x4E2AB2B: rml::internal::BackRefMaster::findFreeBlock() (in /home/is_admin/tbb40_233oss/build/linux_intel64_gcc_cc4.1.2_libc2.5_kernel2.6.18_release/libtbbmalloc.so.2)
==1224==    by 0x4E2AE49: rml::internal::BackRefIdx::newBackRef(bool) (in /home/is_admin/tbb40_233oss/build/linux_intel64_gcc_cc4.1.2_libc2.5_kernel2.6.18_release/libtbbmalloc.so.2)
==1224==    by 0x4E26C49: rml::internal::MemoryPool::getEmptyBlock(unsigned long) (in /home/is_admin/tbb40_233oss/build/linux_intel64_gcc_cc4.1.2_libc2.5_kernel2.6.18_release/libtbbmalloc.so.2)
==1224==    by 0x4E27676: rml::internal::internalPoolMalloc(rml::MemoryPool*, unsigned long) (in /home/is_admin/tbb40_233oss/build/linux_intel64_gcc_cc4.1.2_libc2.5_kernel2.6.18_release/libtbbmalloc.so.2)
==1224==    by 0x4E27825: scalable_malloc (in /home/is_admin/tbb40_233oss/build/linux_intel64_gcc_cc4.1.2_libc2.5_kernel2.6.18_release/libtbbmalloc.so.2)
==1224==    by 0x4C21278: operator new(unsigned long) (in /home/is_admin/tbb40_233oss/build/linux_intel64_gcc_cc4.1.2_libc2.5_kernel2.6.18_release/libtbbmalloc_proxy.so.2)
==1224==    by 0x458922: __gnu_cxx::new_allocator<__gnu_cxx::_Hashtable_node<std::pair<unsigned int const, s_Keypoint*> > >::allocate(unsigned long, void const*) (new_allocator.h:88)
==1224==    by 0x458947: __gnu_cxx::hashtable<std::pair<unsigned int const, s_Keypoint*>, unsigned int, __gnu_cxx::hash<unsigned int>, std::_Select1st<std::pair<unsigned int const, s_Keypoint*> >, std::equal_to<unsigned int>, std::allocator<s_Keypoint*> >::_M_get_node() (hashtable.h:297)
==1224==    by 0x458963: __gnu_cxx::hashtable<std::pair<unsigned int const, s_Keypoint*>, unsigned int, __gnu_cxx::hash<unsigned int>, std::_Select1st<std::pair<unsigned int const, s_Keypoint*> >, std::equal_to<unsigned int>, std::allocator<s_Keypoint*> >::_M_new_node(std::pair<unsigned int const, s_Keypoint*> const&) (hashtable.h:605)
==1224==    by 0x458ABC: __gnu_cxx::hashtable<std::pair<unsigned int const, s_Keypoint*>, unsigned int, __gnu_cxx::hash<unsigned int>, std::_Select1st<std::pair<unsigned int const, s_Keypoint*> >, std::equal_to<unsigned int>, std::allocator<s_Keypoint*> >::insert_equal_noresize(std::pair<unsigned int const, s_Keypoint*> const&) (hashtable.h:783)
==1224== 
==1224== 16,384 bytes in 1 blocks are still reachable in loss record 2 of 4
==1224==    at 0x4A0610C: malloc (vg_replace_malloc.c:195)
==1224==    by 0x4E285C6: rml::internal::getRawMemory(unsigned long, bool) (in /home/is_admin/tbb40_233oss/build/linux_intel64_gcc_cc4.1.2_libc2.5_kernel2.6.18_release/libtbbmalloc.so.2)
==1224==    by 0x4E2AB2B: rml::internal::BackRefMaster::findFreeBlock() (in /home/is_admin/tbb40_233oss/build/linux_intel64_gcc_cc4.1.2_libc2.5_kernel2.6.18_release/libtbbmalloc.so.2)
==1224==    by 0x4E2AE49: rml::internal::BackRefIdx::newBackRef(bool) (in /home/is_admin/tbb40_233oss/build/linux_intel64_gcc_cc4.1.2_libc2.5_kernel2.6.18_release/libtbbmalloc.so.2)
==1224==    by 0x4E2A690: rml::internal::mallocLargeObject(rml::internal::ExtMemoryPool*, unsigned long, unsigned long, bool) (in /home/is_admin/tbb40_233oss/build/linux_intel64_gcc_cc4.1.2_libc2.5_kernel2.6.18_release/libtbbmalloc.so.2)
==1224==    by 0x4E27825: scalable_malloc (in /home/is_admin/tbb40_233oss/build/linux_intel64_gcc_cc4.1.2_libc2.5_kernel2.6.18_release/libtbbmalloc.so.2)
==1224==    by 0x4C21278: operator new(unsigned long) (in /home/is_admin/tbb40_233oss/build/linux_intel64_gcc_cc4.1.2_libc2.5_kernel2.6.18_release/libtbbmalloc_proxy.so.2)
==1224==    by 0x4553AC: __gnu_cxx::new_allocator<s_Keypoint*>::allocate(unsigned long, void const*) (new_allocator.h:88)
==1224==    by 0x4553D4: std::_Vector_base<s_Keypoint*, std::allocator<s_Keypoint*> >::_M_allocate(unsigned long) (stl_vector.h:127)
==1224==    by 0x455C33: std::vector<s_Keypoint*, std::allocator<s_Keypoint*> >::_M_insert_aux(__gnu_cxx::__normal_iterator<s_Keypoint**, std::vector<s_Keypoint*, std::allocator<s_Keypoint*> > >, s_Keypoint* const&) (vector.tcc:275)
==1224==    by 0x455E87: std::vector<s_Keypoint*, std::allocator<s_Keypoint*> >::push_back(s_Keypoint* const&) (stl_vector.h:610)
==1224==    by 0x45711C: DirectHash::getNeighbors1(std::vector<s_Keypoint*, std::allocator<s_Keypoint*> >&, unsigned int) (directhash.cpp:157)
==1224== 
==1224== 49,152 bytes in 3 blocks are still reachable in loss record 3 of 4
==1224==    at 0x4A0610C: malloc (vg_replace_malloc.c:195)
==1224==    by 0x4E285C6: rml::internal::getRawMemory(unsigned long, bool) (in /home/is_admin/tbb40_233oss/build/linux_intel64_gcc_cc4.1.2_libc2.5_kernel2.6.18_release/libtbbmalloc.so.2)
==1224==    by 0x4E2AB2B: rml::internal::BackRefMaster::findFreeBlock() (in /home/is_admin/tbb40_233oss/build/linux_intel64_gcc_cc4.1.2_libc2.5_kernel2.6.18_release/libtbbmalloc.so.2)
==1224==    by 0x4E2AE49: rml::internal::BackRefIdx::newBackRef(bool) (in /home/is_admin/tbb40_233oss/build/linux_intel64_gcc_cc4.1.2_libc2.5_kernel2.6.18_release/libtbbmalloc.so.2)
==1224==    by 0x4E26C49: rml::internal::MemoryPool::getEmptyBlock(unsigned long) (in /home/is_admin/tbb40_233oss/build/linux_intel64_gcc_cc4.1.2_libc2.5_kernel2.6.18_release/libtbbmalloc.so.2)
==1224==    by 0x4E27676: rml::internal::internalPoolMalloc(rml::MemoryPool*, unsigned long) (in /home/is_admin/tbb40_233oss/build/linux_intel64_gcc_cc4.1.2_libc2.5_kernel2.6.18_release/libtbbmalloc.so.2)
==1224==    by 0x4E27825: scalable_malloc (in /home/is_admin/tbb40_233oss/build/linux_intel64_gcc_cc4.1.2_libc2.5_kernel2.6.18_release/libtbbmalloc.so.2)
==1224==    by 0x4C21278: operator new(unsigned long) (in /home/is_admin/tbb40_233oss/build/linux_intel64_gcc_cc4.1.2_libc2.5_kernel2.6.18_release/libtbbmalloc_proxy.so.2)
==1224==    by 0x458922: __gnu_cxx::new_allocator<__gnu_cxx::_Hashtable_node<std::pair<unsigned int const, s_Keypoint*> > >::allocate(unsigned long, void const*) (new_allocator.h:88)
==1224==    by 0x458947: __gnu_cxx::hashtable<std::pair<unsigned int const, s_Keypoint*>, unsigned int, __gnu_cxx::hash<unsigned int>, std::_Select1st<std::pair<unsigned int const, s_Keypoint*> >, std::equal_to<unsigned int>, std::allocator<s_Keypoint*> >::_M_get_node() (hashtable.h:297)
==1224==    by 0x458963: __gnu_cxx::hashtable<std::pair<unsigned int const, s_Keypoint*>, unsigned int, __gnu_cxx::hash<unsigned int>, std::_Select1st<std::pair<unsigned int const, s_Keypoint*> >, std::equal_to<unsigned int>, std::allocator<s_Keypoint*> >::_M_new_node(std::pair<unsigned int const, s_Keypoint*> const&) (hashtable.h:605)
==1224==    by 0x458A42: __gnu_cxx::hashtable<std::pair<unsigned int const, s_Keypoint*>, unsigned int, __gnu_cxx::hash<unsigned int>, std::_Select1st<std::pair<unsigned int const, s_Keypoint*> >, std::equal_to<unsigned int>, std::allocator<s_Keypoint*> >::insert_equal_noresize(std::pair<unsigned int const, s_Keypoint*> const&) (hashtable.h:776)
==1224== 
==1224== 65,536 bytes in 4 blocks are still reachable in loss record 4 of 4
==1224==    at 0x4A0610C: malloc (vg_replace_malloc.c:195)
==1224==    by 0x4E285C6: rml::internal::getRawMemory(unsigned long, bool) (in /home/is_admin/tbb40_233oss/build/linux_intel64_gcc_cc4.1.2_libc2.5_kernel2.6.18_release/libtbbmalloc.so.2)
==1224==    by 0x4E2AB2B: rml::internal::BackRefMaster::findFreeBlock() (in /home/is_admin/tbb40_233oss/build/linux_intel64_gcc_cc4.1.2_libc2.5_kernel2.6.18_release/libtbbmalloc.so.2)
==1224==    by 0x4E2AE49: rml::internal::BackRefIdx::newBackRef(bool) (in /home/is_admin/tbb40_233oss/build/linux_intel64_gcc_cc4.1.2_libc2.5_kernel2.6.18_release/libtbbmalloc.so.2)
==1224==    by 0x4E26C49: rml::internal::MemoryPool::getEmptyBlock(unsigned long) (in /home/is_admin/tbb40_233oss/build/linux_intel64_gcc_cc4.1.2_libc2.5_kernel2.6.18_release/libtbbmalloc.so.2)
==1224==    by 0x4E27676: rml::internal::internalPoolMalloc(rml::MemoryPool*, unsigned long) (in /home/is_admin/tbb40_233oss/build/linux_intel64_gcc_cc4.1.2_libc2.5_kernel2.6.18_release/libtbbmalloc.so.2)
==1224==    by 0x4E27825: scalable_malloc (in /home/is_admin/tbb40_233oss/build/linux_intel64_gcc_cc4.1.2_libc2.5_kernel2.6.18_release/libtbbmalloc.so.2)
==1224==    by 0x4C21278: operator new(unsigned long) (in /home/is_admin/tbb40_233oss/build/linux_intel64_gcc_cc4.1.2_libc2.5_kernel2.6.18_release/libtbbmalloc_proxy.so.2)
==1224==    by 0x453A97: readKeysFromFile(char const*, int) (keypoint.cpp:329)
==1224==    by 0x45D929: KeypointDB::Add(char const*) (keypointdb.cpp:201)
==1224==    by 0x44A264: MRSystem::MRServer::AddFingerPrint(std::string) (mrserver.cpp:68)
==1224==    by 0x445D68: MRSystem::Slave::ConstructHashTable() (Slave.cpp:242)
==1224== 
==1224== LEAK SUMMARY:
==1224==    definitely lost: 0 bytes in 0 blocks
==1224==    indirectly lost: 0 bytes in 0 blocks
==1224==      possibly lost: 0 bytes in 0 blocks
==1224==    still reachable: 147,456 bytes in 9 blocks
==1224==         suppressed: 0 bytes in 0 blocks
==1224== 
==1224== For counts of detected and suppressed errors, rerun with: -v
==1224== Use --track-origins=yes to see where uninitialised values come from
==1224== ERROR SUMMARY: 3 errors from 1 contexts (suppressed: 4 from 4)
1个回答

3

在4.0版本之前,由tbbmalloc用于分配“小”(<8K)对象的内存块仅供请求它的线程重复使用。因此,在问题中描述的情况下,“一个线程将始终在其池中具有先前分配的内存块,并且当其他线程需要内存块时,可能会导致内存不足”的情况是可能发生的。

自2011年发布的v4.0以来,TBB内存分配器可以将所有内存块返回给操作系统。因此,上述问题已不再相关。如果仍在使用较旧版本的tbbmalloc并遇到上述问题,则解决方法是升级分配器。


网页内容由stack overflow 提供, 点击上面的
可以查看英文原文,
原文链接