我正在使用R语言,尝试在按年份和城市结构的数据集中通过人名进行匹配。由于存在一些拼写错误,因此无法进行精确匹配,所以我尝试使用agrep()函数进行模糊匹配。
数据集的一个样例如下:
df <- data.frame(matrix( c("1200013","1200013","1200013","1200013","1200013","1200013","1200013","1200013", "1996","1996","1996","1996","2000","2000","2004","2004","AGUSTINHO FORTUNATO FILHO","ANTONIO PEREIRA NETO","FERNANDO JOSE DA COSTA","PAULO CEZAR FERREIRA DE ARAUJO","PAULO CESAR FERREIRA DE ARAUJO","SEBASTIAO BOCALOM RODRIGUES","JOAO DE ALMEIDA","PAULO CESAR FERREIRA DE ARAUJO"), ncol=3,dimnames=list(seq(1:8),c("citycode","year","candidate")) ))
简洁版:
citycode year candidate
1 1200013 1996 AGUSTINHO FORTUNATO FILHO
2 1200013 1996 ANTONIO PEREIRA NETO
3 1200013 1996 FERNANDO JOSE DA COSTA
4 1200013 1996 PAULO CEZAR FERREIRA DE ARAUJO
5 1200013 2000 PAULO CESAR FERREIRA DE ARAUJO
6 1200013 2000 SEBASTIAO BOCALOM RODRIGUES
7 1200013 2004 JOAO DE ALMEIDA
8 1200013 2004 PAULO CESAR FERREIRA DE ARAUJO
我希望能够单独在每个城市中检查是否有多年出现的候选人。例如,在示例中,“PAULO CEZAR FERREIRA DE ARAUJO”和“PAULO CESAR FERREIRA DE ARAUJO”(拼写错误)都出现了两次。整个数据集中的每个候选人应分配唯一的数字候选人ID。数据集相当大(5500个城市,约100K条目),因此需要一些高效的编码。请问如何实现这一点?
编辑:这是我尝试过的方法(在评论的帮助下),但执行速度非常慢(低效)。请问有什么改进建议吗?
f <- function(x) {matches <- lapply(levels(x), agrep, x=levels(x),fixed=TRUE, value=FALSE)
levels(x) <- levels(x)[unlist(lapply(matches, function(x) x[1]))]
x
}
temp <- tapply(df$candidate, df$citycode, f, simplify=TRUE)
df$candidatenew <- unlist(temp)
df$spellerror <- ifelse(as.character(df$candidate)==as.character(df$candidatenew), 0, 1)
编辑 2:现在运行速度良好。问题是每一步都与许多因素进行比较(感谢Blue Magister指出这一点)。将比较仅限于一个组(即一个城市)中的候选人可以在5秒内对80,000行运行命令 - 这是我可以接受的速度。
df$candidate <- as.character(df$candidate)
f <- function(x) {x <- as.factor(x)
matches <- lapply(levels(x), agrep, x=levels(x),fixed=TRUE, value=FALSE)
levels(x) <- levels(x)[unlist(lapply(matches, function(x) x[1]))]
as.character(x)
}
temp <- tapply(df$candidate, df$citycode, f, simplify=TRUE)
df$candidatenew <- unlist(temp)
df$spellerror <- ifelse(as.character(df$candidate)==as.character(df$candidatenew), 0, 1)
agrep
匹配还是如何高效地进行匹配? - Ari B. Friedman