我对pandas/r还比较新手,不太确定如何将这些数据读入pandas或r进行分析。
目前,我想使用readr的read_chunkwise
或pandas的chunksize
,但这可能不是我需要的。这个问题是否可以使用for循环或使用purr来迭代所有元素来轻松解决?
数据:
wine/name: 1981 Château de Beaucastel Châteauneuf-du-Pape
wine/wineId: 18856
wine/variant: Red Rhone Blend
wine/year: 1981
review/points: 96
review/time: 1160179200
review/userId: 1
review/userName: Eric
review/text: Olive, horse sweat, dirty saddle, and smoke. This actually got quite a bit more spicy and expressive with significant aeration. This was a little dry on the palate first but filled out considerably in time, lovely, loaded with tapenade, leather, dry and powerful, very black olive, meaty. This improved considerably the longer it was open. A terrific bottle of 1981, 96+ and improving. This may well be my favorite vintage of Beau except for perhaps the 1990.
wine/name: 1995 Château Pichon-Longueville Baron
wine/wineId: 3495 wine/variant: Red Bordeaux Blend
wine/year: 1995
review/points: 93
review/time: 1063929600
review/userId: 1
review/userName: Eric
review/text: A remarkably floral nose with violet and chambord. On the palate this is super sweet and pure with a long, somewhat searing finish. My notes are very terse, but this was a lovely wine.
目前,这是我的一个函数,但我遇到了一个错误:
>
convertchunkfile <- function(df){ for(i in 1:length(df)){
> #While the length of any line is not 0, process it with the following loop
> while(nchar(df[[i]]) != 0){
> case_when(
>
> #When data at x index == wine/name, then extract the data after that clause
> #Wine Name parsing
> cleandf$WineName[[i]] <- df[i] == str_sub(df[1],0, 10) ~ str_trim(substr(df[1], 11, nchar(df[1]))),
> #Wine ID parsing
> cleandf$WineID[[i]] <- df[i] == str_sub(df[2],0,11) ~ str_trim(substr(df[2], 13, nchar(df[1])))
> #same format for other attributes
> )
> }
> }
> }
Error in cleandf$BeerName[[i]] <- df[i] == str_sub(df[1], 0, 10) ~ str_trim(substr(df[1], :
more elements supplied than there are to replace
编辑:
在解决了一些问题之后,我认为这可能是最好的解决方案,参考了@hereismyname的解决方案:
#Use Bash's iconv to force convert the file in OS X
iconv -c -t UTF-8 cellartracker-clean.txt > cellartracker-iconv.txt
#Check number of lines within the file
wc -l cellartracker-iconv.txt
20259950 cellartracker-iconv.txt
#Verify new encoding of the file
file -I cellartracker-clean.txt
ReadEmAndWeep <- function(file, chunk_size) {
f <- function(chunk, pos) {
data_frame(text = chunk) %>%
filter(text != "") %>%
separate(text, c("var", "value"), ":", extra = "merge") %>%
mutate(
chunk_id = rep(1:(nrow(.) / 9), each = 9),
value = trimws(value)
) %>%
spread(var, value)
}
read_lines_chunked(file, DataFrameCallback$new(f), chunk_size = chunk_size)
}
#Final Function call to read in the file
dataframe <- ReadEmAndWeep(file, chunk_size = 100000)
wine/year: 1981 review/points: 96
)。这是正确的吗? - A5C1D2H2I1M1N2O1R2T1