我希望能够从目标文件夹中读取多个CSV文件(列数不同),将它们合并成一个Python Pandas DataFrame以便高效地搜索和提取数据。
示例文件:
示例文件:
Events
1,0.32,0.20,0.67
2,0.94,0.19,0.14,0.21,0.94
3,0.32,0.20,0.64,0.32
4,0.87,0.13,0.61,0.54,0.25,0.43
5,0.62,0.21,0.77,0.44,0.16
以下是我目前的进展:
# get a list of all csv files in target directory
my_dir = "C:\\Data\\"
filelist = []
os.chdir( my_dir )
for files in glob.glob( "*.csv" ) :
filelist.append(files)
# read each csv file into single dataframe and add a filename reference column
# (i.e. file1, file2, file 3) for each file read
df = pd.DataFrame()
columns = range(1,100)
for c, f in enumerate(filelist) :
key = "file%i" % c
frame = pd.read_csv( (my_dir + f), skiprows = 1, index_col=0, names=columns )
frame['key'] = key
df = df.append(frame,ignore_index=True)
索引不正常工作。
基本上,下面的脚本完全符合我的要求(经过尝试和测试),但需要循环遍历10个或更多的CSV文件:
df1 = pd.DataFrame()
df2 = pd.DataFrame()
columns = range(1,100)
df1 = pd.read_csv("C:\\Data\\Currambene_001y09h00m_events.csv",
skiprows = 1, index_col=0, names=columns)
df2 = pd.read_csv("C:\\Data\\Currambene_001y12h00m_events.csv",
skiprows = 1, index_col=0, names=columns)
keys = [('file1'), ('file2')]
df = pd.concat([df1, df2], keys=keys, names=['fileno'])
我找到了很多相关的链接,但是我仍然不能让它正常运行:
pandas.concat
可以让你使用一个包含任意长度的 DataFrame 列表。将第一个参数设置为一个包含所有文件的单个列表,就不必再循环脚本了。 - dmvianna