空白格在fread中无法被识别为NA

27

我有一个大文件需要在R中导入。为此,我使用了fread。但是,fread将数值字段中的空格识别为NA,但未将字符和integer64字段中的空格识别为NA。

fread将字符字段中的空格识别为空单元格,将integer64字段中的空格识别为0。

当我使用read.table导入相同数据时,它将所有空格识别为NA。

请查看以下可重现示例:

library(data.table)
x1 <- c("","","")
x2 <- c("1006678566","","1011160152")
x3 <- c("","ac","")
x4 <- c("","2","3")
df <- cbind.data.frame(x1,x2,x3,x4)
write.csv(df,"tr.csv")

tr1 <- fread("tr.csv", header=T, fill = T,
             sep= ",", na.strings = c("",NA), data.table = F,
             stringsAsFactors = FALSE)

tr2 <- read.table("tr.csv", fill = TRUE, header=T, 
                  sep= ",", na.strings = c(""," ", NA), 
                  stringsAsFactors = FALSE)

通过fread导入

详细输出:

Input contains no \n. Taking this to be a filename to open
[01] Check arguments
  Using 4 threads (omp_get_max_threads()=4, nth=4)
  NAstrings = [<<>>, <<NA>>]
  None of the NAstrings look like numbers.
  show progress = 1
  0/1 column will be read as integer
[02] Opening the file
  Opening file tr.csv
  File opened, size = 409 bytes.
  Memory mapped ok
[03] Detect and skip BOM
[04] Arrange mmap to be \0 terminated
  \n has been found in the input and different lines can end with different line endings (e.g. mixed \n and \r\n in one file). This is common and ideal.
[05] Skipping initial rows if needed
  Positioned on line 1 starting: <<"","x1","x2","x3","x4","x5","x>>
[06] Detect separator, quoting rule, and ncolumns
  Using supplied sep ','
  sep=','  with 7 fields using quote rule 0
  Detected 7 columns on line 1. This line is either column names or first data row. Line starts as: <<"","x1","x2","x3","x4","x5","x>>
  Quote rule picked = 0
  fill=true and the most number of columns found is 7
[07] Detect column types, good nrow estimate and whether first row is column names
  'header' changed by user from 'auto' to true
  Number of sampling jump points = 1 because (407 bytes from row 1 to eof) / (2 * 407 jump0size) == 0
  Type codes (jump 000)    : 56A255A  Quote rule 0
  All rows were sampled since file is small so we know nrow=16 exactly
[08] Assign column names
[09] Apply user overrides on column types
  After 0 type and 0 drop user overrides : 56A255A
[10] Allocate memory for the datatable
  Allocating 7 column slots (7 - 0 dropped) with 16 rows
[11] Read the data
  jumps=[0..1), chunk_size=1048576, total_size=373
Read 16 rows x 7 columns from 409 bytes file in 00:00.042 wall clock time
[12] Finalizing the datatable
  Type counts:
         1 : bool8     '2'
         3 : int32     '5'
         1 : int64     '6'
         2 : string    'A'
=============================
   0.009s ( 22%) Memory map 0.000GB file
   0.029s ( 68%) sep=',' ncol=7 and header detection
   0.002s (  5%) Column type detection using 16 sample rows
   0.001s (  2%) Allocation of 16 rows x 7 cols (0.000GB) of which 16 (100%) rows used
   0.001s (  2%) Reading 1 chunks (0 swept) of 1.000MB (each chunk 16 rows) using 1 threads
   +    0.000s (  0%) Parse to row-major thread buffers (grown 0 times)
   +    0.000s (  0%) Transpose
   +    0.001s (  2%) Waiting
   0.000s (  0%) Rereading 0 columns due to out-of-sample type exceptions
   0.042s        Total

请帮我解决这个问题。

谢谢!


@SatZ,请查看带有屏幕截图的更新问题。x3中的值未被识别为NA。 - SJB
1
@docendodiscimus 是的,还有一些与此相关的未解决问题,例如 https://github.com/Rdatatable/data.table/issues/2857 和 https://github.com/Rdatatable/data.table/issues/2714。 - MichaelChirico
5
我也遇到了同样的问题。为了解决这个问题,我使用fread函数进行导入,因为它速度很快,然后用以下代码将空白字段替换为NA:lapply(tr1, function(x) {ifelse(x == "", NA, x)})。 - Dealec
1
很不幸,目前在fread中似乎还没有解决这个问题的方法。我能想到的最快的方式是使用sapply(fread("tr.csv", quote = ""), function(x) gsub(""", NA, x))一行完成。如果需要多于一行:dt <- fread("tr.csv"),dt[dt==""] <- NA - koolmees
显示剩余9条评论
3个回答

4

@SJB 在fread()中使用参数na.strings = c(NA_character_, ""),空格/单元格将被读取为NA。

不同数据类型有各种形式的NA。请参见help(NA): NA_character_ NA_real_ NA_integer_ 等等。


2

如果您想在读取文件后避免进行额外的操作,可以尝试在写入csv时使用

quote = FALSE

。这将防止在值周围使用引号" ",并且所有缺失的值现在应该被读取为NA。它应该看起来像这样 -

# also turned off row names to prevent an additional column when reading the file.
write.csv(df, "tr.csv", quote = FALSE, row.names = FALSE) 

输出 -

tr1 <- fread("tr.csv", header=T, fill = T,
             sep= ",", na.strings = c("",NA), data.table = F,
             stringsAsFactors = FALSE)
tr1

 x1         x2   x3 x4
1 NA 1006678566 <NA> NA
2 NA         NA   ac  2
3 NA 1011160152 <NA>  3

tr2 <- read.table("tr.csv", fill = TRUE, header=T, 
                  sep= ",", na.strings = c(""," ", NA), 
                  stringsAsFactors = FALSE)
tr2

  x1         x2   x3 x4
1 NA 1006678566 <NA> NA
2 NA         NA   ac  2
3 NA 1011160152 <NA>  3

那个完美地工作了,应该是正确的答案。另外一个问题是,使用fill=TRUE有什么负面影响(风险)吗? - GitHunter0
1
@hunter_git 并不完全正确。它会使用NA填充行中缺失的列(如果有的话),而不是引发“提前停止”警告。因此,除非您确定行的长度,否则应将fill保持为FALSE,以便在行长度存在任何差异时引发警告。 - Oshan

0

我发现的一件事是,当我们使用write.csv()时,数据保存的方式。

打开csv文件,在X4中删除空单元格并保存。如果现在导入它,NA将会在R中显示。

检查方法:

apply(tr1, 2, function(x) length(which(is.na(x))))

V1 x1 x2 x3 x4

0 3 1 2 1

如果有一个包含空白的csv文件并且我们使用

na.strings("", NA)

fread读取数据,那么字符数据类型中的空值也会显示为"NA"。


@ Deb:na.strings = c("", NA) 适用于 fread() 并且如果 colSums(is.na(DT)) 将 <NA> 计数为 NA,则是正确的解决方案。但是它没有这样做。 - Dragos Bandur

网页内容由stack overflow 提供, 点击上面的
可以查看英文原文,
原文链接