在/proc/$pid/maps中，“deleted”是什么意思？

Question

我下载了libhugetlbfs.so并有一个简单的测试源代码：

int glbarr[1024*1024]={0} ;
int main()
{
    char * ptr ;
    ptr = (char *) malloc( 1024 * 1024 * 1 ) ;
    printf(" press any key to go on \n");
    getchar() ;
    for(int idx=0;idx<100;idx++){
        char strtmp[64] = {0} ;
        sprintf(strtmp,"%020d",idx) ;
        strcpy( ptr+1024*idx , strtmp ) ;
    } //for 
    for(int idx=0;idx<100;idx++){
        glbarr[idx] = idx ;
    }
    printf(" press any key to go on \n");
    getchar() ;
} // main

然后设置环境变量：

export LD_PRELOAD=libhugetlbfs.so
export HUGETLB_MORECORE=yes
export HUGETLB_DEBUG=1

最后，执行test_malloc.exe测试：

 INFO: Found pagesize 2048 kB
 INFO: Detected page sizes:
 INFO:    Size: 2048 kB (default)  Mount: /mnt/SharedMem_2M
 INFO: Parsed kernel version: [3] . [10] . [0] 
 INFO: Feature private_reservations is present in this kernel
 INFO: Feature noreserve_safe is present in this kernel
 INFO: Feature map_hugetlb is present in this kernel
 INFO: Kernel has MAP_PRIVATE reservations.  Disabling heap prefaulting.
 INFO: Kernel supports MAP_HUGETLB
 INFO: HUGETLB_SHARE=0, sharing disabled
 INFO: HUGETLB_NO_RESERVE=no, reservations enabled
 INFO: Segment 0 (phdr 2): 0x400000-0x400a04  (filesz=0xa04) (prot = 0x5)
 INFO: Segment 1 (phdr 3): 0x600de0-0xa01080  (filesz=0x274) (prot = 0x3)
 DEBUG: symbol to copy at 0x601060: stdin
 DEBUG: Total memsz = 0x400ca4, memsz of largest segment = 0x4002a0
 INFO: libhugetlbfs version: 2.20
 INFO: Mapped hugeseg at 0x2aaaaac00000. Copying 0xa04 bytes and 0 extra bytes from 0x400000...done
 INFO: Prepare succeeded
 INFO: Mapped hugeseg at 0x2aaaaac00000. Copying 0x274 bytes and 0x14 extra bytes from 0x600de0...done
 INFO: Prepare succeeded
 INFO: setup_morecore(): heapaddr = 0x1c00000
 INFO: hugetlbfs_morecore(2101248) = ...
 INFO: heapbase = 0x1c00000, heaptop = 0x1c00000, mapsize = 0, delta=2101248
 INFO: Attempting to map 4194304 bytes
 INFO: ... = 0x1c00000
 INFO: hugetlbfs_morecore(0) = ...
 INFO: heapbase = 0x1c00000, heaptop = 0x1e01000, mapsize = 400000, delta=-2093056
 INFO: ... = 0x1e01000

并且，/proc/test_malloc.exe的pid/maps：

00400000-00600000 r-xp 00000000 00:2b 6019488                            /mnt/SharedMem_2M/libhugetlbfs.tmp.uI55WD (deleted)
00600000-00c00000 rw-p 00000000 00:2b 6123885                            /mnt/SharedMem_2M/libhugetlbfs.tmp.VUALYM (deleted)
01c00000-02000000 rw-p 00000000 00:0d 6123886                            /anon_hugepage (deleted)

numastat -m 显示当前8M的大页是否真正被使用， 我困惑的是maps输出中的“deleted”是什么意思？无论是/mnt/SharedMem_2M还是/anon_hugepage。

编辑：

还有调试信息：

INFO: Found pagesize 2048 kB
INFO: Detected page sizes:
INFO:    Size: 2048 kB (default)  Mount: /mnt/SharedMem_2M
INFO: Parsed kernel version: [3] . [10] . [0] 
INFO: Feature private_reservations is present in this kernel
INFO: Feature noreserve_safe is present in this kernel
INFO: Feature map_hugetlb is present in this kernel
INFO: Kernel has MAP_PRIVATE reservations.  Disabling heap prefaulting.
INFO: Kernel supports MAP_HUGETLB
INFO: HUGETLB_SHARE=0, sharing disabled
INFO: HUGETLB_NO_RESERVE=no, reservations enabled
INFO: Segment 0 (phdr 3): 0x600de0-0xa01080  (filesz=0x274) (prot = 0x3)
DEBUG: symbol to copy at 0x601060: stdin
DEBUG: Total memsz = 0x4002a0, memsz of largest segment = 0x4002a0
INFO: libhugetlbfs version: 2.20
INFO: Mapped hugeseg at 0x2aaaaac00000. Copying 0x274 bytes and 0x14 extra bytes from 0x600de0...done
INFO: Prepare succeeded
INFO: setup_morecore(): heapaddr = 0x2200000
INFO: hugetlbfs_morecore(2101248) = ...
INFO: heapbase = 0x2200000, heaptop = 0x2200000, mapsize = 0, delta=2101248
INFO: Attempting to map 4194304 bytes
INFO: ... = 0x2200000
INFO: hugetlbfs_morecore(0) = ...
INFO: heapbase = 0x2200000, heaptop = 0x2401000, mapsize = 400000, delta=-2093056
INFO: ... = 0x2401000

并且地图：

00400000 default file=/home/marschen/test/posix-memalign/test_malloc.exe mapped=1 N0=1 kernelpagesize_kB=4
00600000 default file=/mnt/SharedMem_2M/libhugetlbfs.tmp.85Y41e\040(deleted) huge anon=1 dirty=1 N0=1 kernelpagesize_kB=2048
02200000 default file=/anon_hugepage\040(deleted) huge anon=1 dirty=1 N0=1 kernelpagesize_kB=2048

- barfatchen

---

“deleted” 表示该文件已被 mmap 并从文件系统中可见部分删除。 它仍然由文件系统处理（使用 hugepages 进行 mmap），但现在无法与另一个进程共享（文件无法打开）。 您有什么问题吗？ 您可以在没有 --text 选项的情况下发布相同的结果吗？在执行之前，libhugetlbfs 可能会将文本部分复制到某个新的大型映射中。 - osgx

@osgx，谢谢，我的问题是为什么在地图中删除了某些内容，这是否意味着我做错了什么？！ - barfatchen

不，"deleted"并没有任何错误，这只是libhugetlbfs内部工作的方式 - （在某些情况下）它会在特殊挂载的伪文件系统上创建文件，将该文件映射到内存中并删除它（从文件系统中删除文件名）。文件数据仍由hugetlbfs处理，而进程正在运行。你的grep hugetlbfs /proc/filesystems是什么？ - osgx

@osgx，谢谢，nodev hugetlbfs用于在/proc/filesystems中查找hugetlbfs。 - barfatchen

Answer 1

当libhugetlbfs使用hugetlb fs伪文件系统(grep hugetlbfs /proc/filesystems)获取由hugetlb页面支持的mmaps时，临时文件删除（unlinking）是正常的。

例如，libhugetlbfs/hugeutils.c中有一个hugetlbfs_unlinked_fd函数。 https://github.com/libhugetlbfs/libhugetlbfs/blob/e44180072b796c0e28e53c4d01ef6279caaa2a99/hugeutils.c#L1033

int hugetlbfs_unlinked_fd_for_size(long page_size)
{
    const char *path;
    char name[PATH_MAX+1];
    int fd;

    path = hugetlbfs_find_path_for_size(page_size);
    ..
    name[sizeof(name)-1] = '\0';

    strcpy(name, path);
    strncat(name, "/libhugetlbfs.tmp.XXXXXX", sizeof(name)-1);
    /* FIXME: deal with overflows */

    fd = mkstemp64(name);
    ....

    unlink(name);

    return fd;
}

mkstemp函数会随机生成临时文件名，同时创建并打开该文件。此后，该文件会被从文件系统中删除（文件名在目录中被标记为已删除，但inode和文件数据仍然存在，其他程序无法通过名称访问该文件）(man 2 unlink)。

在文件被删除前，可以使用已经打开的未命名fd进行hugetlb mmap操作以及存储数据，当这个fd被关闭时，文件数据才会被实际地由文件系统删除。

通常情况下，mktemp文件的早期删除被广泛应用：When a file created with mkstemp() is deleted?

libhugetlbfs项目的HOWTO中还可以列出一些有用的信息：https://github.com/libhugetlbfs/libhugetlbfs/blob/master/HOWTO

- osgx