在ffdf对象中进行子集筛选（subset vs ffwhich）

Question

我正在对大型ffdf对象执行子集操作，发现当我使用subset.ff时会生成大量的NAs。我尝试了另一种方法，使用ffwhich，结果速度更快且不会生成NAs。以下是我的测试：

library(ffbase)
# deals is the ffdf I would like to subset
unique(deals$COMMODITY)
ff (open) integer length=7 (7) levels: CASH CO2 COAL ELEC GAS GCERT OIL
  [1]   [2]   [3]   [4]   [5]   [6]   [7] 
CASH  CO2   COAL  ELEC  GAS   GCERT OIL   

# Using subset.ff
started.at=proc.time()
deals0 <- subset.ff(deals,deals$COMMODITY %in% c("CASH","COAL","CO2","ELEC","GCERT"))
cat("Finished in",timetaken(started.at),"\n")
Finished in 12.640sec
# NAs are generated
unique(deals0$COMMODITY)
ff (open) integer length=8 (8) levels: CASH CO2 COAL ELEC GAS GCERT OIL <NA>
  [1]   [2]   [3]   [4]   [5]   [6]   [7]   [8] 
CASH  CO2   COAL  ELEC  GAS   GCERT OIL   NA    

# Subset using ffwhich
started.at=proc.time()
idx <- ffwhich(deals,COMMODITY %in% c("CASH","COAL","CO2","ELEC","GCERT"))
deals1 <- deals[idx,]
cat("Finished in",timetaken(started.at),"\n")
Finished in 3.130sec
# No NAs are generated
unique(deals1$COMMODITY)
ff (open) integer length=7 (7) levels: CASH CO2 COAL ELEC GAS GCERT OIL
  [1]   [2]   [3]   [4]   [5]   [6]   [7] 
CASH  CO2   COAL  ELEC  GAS   GCERT OIL   

有什么想法为什么会发生这种情况？

- ddg

Answer 1

subset.ff可能使用了 [ 和你的条件，但没有包含 !is.na(.) 子句。对于 [ 的默认行为是返回既为 TRUE 又为 NA 的向量。regular subset 函数添加了一个 !is.na(.) 子句，但也许 ffbase 的作者还没有完全做到这一点。

- IRTFM

---

正确！因此，在这种情况下，首选选项是使用ffwhich。 - user1600826

该问题目前已在CRAN上的ffbase 0.6.2版本中得到解决。请参阅新闻文件：http://cran.r-project.org/web/packages/ffbase/NEWS - user1600826