我有一个Pandas数据框，df：
c1 c2
0 10 100
1 11 110
2 12 120
for row in df.rows:
print(row['c1'], row['c2'])
我找到了一个类似的问题，建议使用以下任一方法：
对于日期和行数据，使用df.T.iteritems()进行循环：
使用df.iterrows()进行循环：
但我不理解 row 对象是什么以及如何使用它。
对于查看和修改值，我会使用iterrows()。在for循环中并使用元组拆包（参见示例：i，row），我仅使用row来查看该值，并在需要修改值时使用i与loc方法。如先前的回答所述，在迭代过程中不应修改正在迭代的内容。
for i, row in df.iterrows():
df_column_A = df.loc[i, 'A']
if df_column_A == 'Old_Value':
df_column_A = 'New_value'
在循环中，row 是行的副本而不是视图。因此，不应编写像 row['A'] = 'New_Value' 这样的代码，它不会修改 DataFrame。但是，您可以使用 i 和 loc 并指定 DataFrame 来完成操作。
apply函数。def print_row(row):
print row['c1'], row['c2']
df.apply(lambda row: print_row(row), axis=1)
有很多方法可以遍历Pandas数据帧中的行。一个非常简单和直观的方法是:
df = pd.DataFrame({'A':[1, 2, 3], 'B':[4, 5, 6], 'C':[7, 8, 9]})
print(df)
for i in range(df.shape[0]):
# For printing the second column
print(df.iloc[i, 1])
# For printing more than one columns
print(df.iloc[i, [0, 2]])
列表推导和向量化（可能还包括布尔索引）是你所需要的全部。
使用列表推导（好）和向量化（最佳）。我认为纯向量化总是可能的，但在复杂计算中可能需要额外的工作。在这个答案中搜索"布尔索引"、"布尔数组"和"布尔掩码"（这三者是同一回事），以查看一些更复杂的情况，纯向量化可以在其中使用。
8_pure_vectorization__with_df.loc[]_boolean_array_indexing_for_if_statment_corner_case6_vectorization__with_apply_for_if_statement_corner_case7_vectorization__with_list_comprehension_for_if_statment_corner_case11_list_comprehension_w_zip_and_direct_variable_assignment_calculated_in_place10_list_comprehension_w_zip_and_direct_variable_assignment_passed_to_func12_list_comprehension_w_zip_and_row_tuple_passed_to_func5_itertuples_in_for_loop13_list_comprehension_w__to_numpy__and_direct_variable_assignment_passed_to_func9_apply_function_with_lambda1_raw_for_loop_using_regular_df_indexing2_raw_for_loop_using_df.loc[]_indexing4_iterrows_in_for_loop3_raw_for_loop_using_df.iloc[]_indexing假设我们有以下的Pandas DataFrame。它有200万行和4列（A，B，C和D），每列的值都是从-1000到1000的随机值：
df =
A B C D
0 -365 842 284 -942
1 532 416 -102 888
2 397 321 -296 -616
3 -215 879 557 895
4 857 701 -157 480
... ... ... ... ...
1999995 -101 -233 -377 -939
1999996 -989 380 917 145
1999997 -879 333 -372 -970
1999998 738 982 -743 312
1999999 -306 -103 459 745
import numpy as np
import pandas as pd
# Create an array (numpy list of lists) of fake data
MIN_VAL = -1000
MAX_VAL = 1000
# NUM_ROWS = 10_000_000
NUM_ROWS = 2_000_000 # default for final tests
# NUM_ROWS = 1_000_000
# NUM_ROWS = 100_000
# NUM_ROWS = 10_000 # default for rapid development & initial tests
NUM_COLS = 4
data = np.random.randint(MIN_VAL, MAX_VAL, size=(NUM_ROWS, NUM_COLS))
# Now convert it to a Pandas DataFrame with columns named "A", "B", "C", and "D"
df_original = pd.DataFrame(data, columns=["A", "B", "C", "D"])
print(f"df = \n{df_original}")
我想要展示这些技术在非平凡的函数或方程上是可行的，所以我故意设计了这个需要计算的方程，它需要：
if语句我们将为每一行计算的方程如下。我随意编写了它，但我认为它足够复杂，你可以在我所做的基础上扩展，以在Pandas中执行任何你想要的方程，并实现完全向量化：
# Calculate and return a new value, `val`, by performing the following equation:
val = (
2 * A_i_minus_2
+ 3 * A_i_minus_1
+ 4 * A
+ 5 * A_i_plus_1
# Python ternary operator; don't forget parentheses around the entire
# ternary expression!
+ ((6 * B) if B > 0 else (60 * B))
+ 7 * C
- 8 * D
)
# Calculate and return a new value, `val`, by performing the following equation:
if B > 0:
B_new = 6 * B
else:
B_new = 60 * B
val = (
2 * A_i_minus_2
+ 3 * A_i_minus_1
+ 4 * A
+ 5 * A_i_plus_1
+ B_new
+ 7 * C
- 8 * D
)
def calculate_val(
A_i_minus_2,
A_i_minus_1,
A,
A_i_plus_1,
B,
C,
D):
val = (
2 * A_i_minus_2
+ 3 * A_i_minus_1
+ 4 * A
+ 5 * A_i_plus_1
# Python ternary operator; don't forget parentheses around the
# entire ternary expression!
+ ((6 * B) if B > 0 else (60 * B))
+ 7 * C
- 8 * D
)
return val
完整的代码可以在我的python/pandas_dataframe_iteration_vs_vectorization_vs_list_comprehension_speed_tests.py文件中下载和运行，该文件位于我的eRCaGuy_hello_world存储库中。
1_raw_for_loop_using_regular_df_indexing
val = [np.NAN]*len(df)
for i in range(len(df)):
if i < 2 or i > len(df)-2:
continue
val[i] = calculate_val(
df["A"][i-2],
df["A"][i-1],
df["A"][i],
df["A"][i+1],
df["B"][i],
df["C"][i],
df["D"][i],
)
df["val"] = val # put this column back into the dataframe
2_raw_for_loop_using_df.loc[]_indexing
val = [np.NAN]*len(df)
for i in range(len(df)):
if i < 2 or i > len(df)-2:
continue
val[i] = calculate_val(
df.loc[i-2, "A"],
df.loc[i-1, "A"],
df.loc[i, "A"],
df.loc[i+1, "A"],
df.loc[i, "B"],
df.loc[i, "C"],
df.loc[i, "D"],
)
df["val"] = val # put this column back into the dataframe
3_raw_for_loop_using_df.iloc[]_indexing
# column indices
i_A = 0
i_B = 1
i_C = 2
i_D = 3
val = [np.NAN]*len(df)
for i in range(len(df)):
if i < 2 or i > len(df)-2:
continue
val[i] = calculate_val(
df.iloc[i-2, i_A],
df.iloc[i-1, i_A],
df.iloc[i, i_A],
df.iloc[i+1, i_A],
df.iloc[i, i_B],
df.iloc[i, i_C],
df.iloc[i, i_D],
)
df["val"] = val # put this column back into the dataframe
4_iterrows_in_for_loop
val = [np.NAN]*len(df)
for index, row in df.iterrows():
if index < 2 or index > len(df)-2:
continue
val[index] = calculate_val(
df["A"][index-2],
df["A"][index-1],
row["A"],
df["A"][index+1],
row["B"],
row["C"],
row["D"],
)
df["val"] = val # put this column back into the dataframe
df_original["A_i_minus_2"] = df_original["A"].shift(2) # val at index i-2
df_original["A_i_minus_1"] = df_original["A"].shift(1) # val at index i-1
df_original["A_i_plus_1"] = df_original["A"].shift(-1) # val at index i+1
# Note: to ensure that no partial calculations are ever done with rows which
# have NaN values due to the shifting, we can either drop such rows with
# `.dropna()`, or set all values in these rows to NaN. I'll choose the latter
# so that the stats that will be generated with the techniques below will end
# up matching the stats which were produced by the prior techniques above. ie:
# the number of rows will be identical to before.
#
# df_original = df_original.dropna()
df_original.iloc[:2, :] = np.NAN # slicing operators: first two rows,
# all columns
df_original.iloc[-1:, :] = np.NAN # slicing operators: last row, all columns
5_itertuples_in_for_loop
val = [np.NAN]*len(df)
for row in df.itertuples():
val[row.Index] = calculate_val(
row.A_i_minus_2,
row.A_i_minus_1,
row.A,
row.A_i_plus_1,
row.B,
row.C,
row.D,
)
df["val"] = val # put this column back into the dataframe
6_vectorization__with_apply_for_if_statement_corner_case
def calculate_new_column_b_value(b_value):
# Python ternary operator
b_value_new = (6 * b_value) if b_value > 0 else (60 * b_value)
return b_value_new
# In this particular example, since we have an embedded `if-else` statement
# for the `B` column, pure vectorization is less intuitive. So, first we'll
# calculate a new `B` column using
# **`apply()`**, then we'll use vectorization for the rest.
df["B_new"] = df["B"].apply(calculate_new_column_b_value)
# OR (same thing, but with a lambda function instead)
# df["B_new"] = df["B"].apply(lambda x: (6 * x) if x > 0 else (60 * x))
# Now we can use vectorization for the rest. "Vectorization" in this case
# means to simply use the column series variables in equations directly,
# without manually iterating over them. Pandas DataFrames will handle the
# underlying iteration automatically for you. You just focus on the math.
df["val"] = (
2 * df["A_i_minus_2"]
+ 3 * df["A_i_minus_1"]
+ 4 * df["A"]
+ 5 * df["A_i_plus_1"]
+ df["B_new"]
+ 7 * df["C"]
- 8 * df["D"]
)
7_vectorization__with_list_comprehension_for_if_statment_corner_case
# In this particular example, since we have an embedded `if-else` statement
# for the `B` column, pure vectorization is less intuitive. So, first we'll
# calculate a new `B` column using **list comprehension**, then we'll use
# vectorization for the rest.
df["B_new"] = [
calculate_new_column_b_value(b_value) for b_value in df["B"]
]
# Now we can use vectorization for the rest. "Vectorization" in this case
# means to simply use the column series variables in equations directly,
# without manually iterating over them. Pandas DataFrames will handle the
# underlying iteration automatically for you. You just focus on the math.
df["val"] = (
2 * df["A_i_minus_2"]
+ 3 * df["A_i_minus_1"]
+ 4 * df["A"]
+ 5 * df["A_i_plus_1"]
+ df["B_new"]
+ 7 * df["C"]
- 8 * df["D"]
)
8_pure_vectorization__with_df.loc[]_boolean_array_indexing_for_if_statment_corner_case
This uses boolean indexing, AKA: a boolean mask, to accomplish the equivalent of the if statement in the equation. In this way, pure vectorization can be used for the entire equation, thereby maximizing performance and speed.
# If statement to evaluate:
#
# if B > 0:
# B_new = 6 * B
# else:
# B_new = 60 * B
#
# In this particular example, since we have an embedded `if-else` statement
# for the `B` column, we can use some boolean array indexing through
# `df.loc[]` for some pure vectorization magic.
#
# Explanation:
#
# Long:
#
# The format is: `df.loc[rows, columns]`, except in this case, the rows are
# specified by a "boolean array" (AKA: a boolean expression, list of
# booleans, or "boolean mask"), specifying all rows where `B` is > 0. Then,
# only in that `B` column for those rows, set the value accordingly. After
# we do this for where `B` is > 0, we do the same thing for where `B`
# is <= 0, except with the other equation.
#
# Short:
#
# For all rows where the boolean expression applies, set the column value
# accordingly.
#
# GitHub CoPilot first showed me this `.loc[]` technique.
# See also the official documentation:
# https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.loc.html
#
# ===========================
# 1st: handle the > 0 case
# ===========================
df["B_new"] = df.loc[df["B"] > 0, "B"] * 6
#
# ===========================
# 2nd: handle the <= 0 case, merging the results into the
# previously-created "B_new" column
# ===========================
# - NB: this does NOT work; it overwrites and replaces the whole "B_new"
# column instead:
#
# df["B_new"] = df.loc[df["B"] <= 0, "B"] * 60
#
# This works:
df.loc[df["B"] <= 0, "B_new"] = df.loc[df["B"] <= 0, "B"] * 60
# Now use normal vectorization for the rest.
df["val"] = (
2 * df["A_i_minus_2"]
+ 3 * df["A_i_minus_1"]
+ 4 * df["A"]
+ 5 * df["A_i_plus_1"]
+ df["B_new"]
+ 7 * df["C"]
- 8 * df["D"]
)
9_apply_function_with_lambda
df["val"] = df.apply(
lambda row: calculate_val(
row["A_i_minus_2"],
row["A_i_minus_1"],
row["A"],
row["A_i_plus_1"],
row["B"],
row["C"],
row["D"]
),
axis='columns' # same as `axis=1`: "apply function to each row",
# rather than to each column
)
10_list_comprehension_w_zip_and_direct_variable_assignment_passed_to_func
df["val"] = [
# Note: you *could* do the calculations directly here instead of using a
# function call, so long as you don't have indented code blocks such as
# sub-routines or multi-line if statements.
#
# I'm using a function call.
calculate_val(
A_i_minus_2,
A_i_minus_1,
A,
A_i_plus_1,
B,
C,
D
) for A_i_minus_2, A_i_minus_1, A, A_i_plus_1, B, C, D
in zip(
df["A_i_minus_2"],
df["A_i_minus_1"],
df["A"],
df["A_i_plus_1"],
df["B"],
df["C"],
df["D"]
)
]
11_list_comprehension_w_zip_and_direct_variable_assignment_calculated_in_place
df["val"] = [
2 * A_i_minus_2
+ 3 * A_i_minus_1
+ 4 * A
+ 5 * A_i_plus_1
# Python ternary operator; don't forget parentheses around the entire
# ternary expression!
+ ((6 * B) if B > 0 else (60 * B))
+ 7 * C
- 8 * D
for A_i_minus_2, A_i_minus_1, A, A_i_plus_1, B, C, D
in zip(
df["A_i_minus_2"],
df["A_i_minus_1"],
df["A"],
df["A_i_plus_1"],
df["B"],
df["C"],
df["D"]
)
]
12_list_comprehension_w_zip_and_row_tuple_passed_to_func
df["val"] = [
calculate_val(
row[0],
row[1],
row[2],
row[3],
row[4],
row[5],
row[6],
) for row
in zip(
df["A_i_minus_2"],
df["A_i_minus_1"],
df["A"],
df["A_i_plus_1"],
df["B"],
df["C"],
df["D"]
)
]
13_list_comprehension_w__to_numpy__and_direct_variable_assignment_passed_to_func
df["val"] = [
# Note: you *could* do the calculations directly here instead of using a
# function call, so long as you don't have indented code blocks such as
# sub-routines or multi-line if statements.
#
# I'm using a function call.
calculate_val(
A_i_minus_2,
A_i_minus_1,
A,
A_i_plus_1,
B,
C,
D
) for A_i_minus_2, A_i_minus_1, A, A_i_plus_1, B, C, D
# Note: this `[[...]]` double-bracket indexing is used to select a
# subset of columns from the dataframe. The inner `[]` brackets
# create a list from the column names within them, and the outer
# `[]` brackets accept this list to index into the dataframe and
# select just this list of columns, in that order.
# - See the official documentation on it here:
# https://pandas.pydata.org/docs/user_guide/indexing.html#basics
# - Search for the phrase "You can pass a list of columns to [] to
# select columns in that order."
# - I learned this from this comment here:
# https://dev59.com/h2Qn5IYBdhLWcg3w5qlg#TYmZZowBVUcc3sd71Izc
# - One of the **list comprehension** examples in this answer here
# uses `.to_numpy()` like this:
# https://dev59.com/h2Qn5IYBdhLWcg3w5qlg#55557758
in df[[
"A_i_minus_2",
"A_i_minus_1",
"A",
"A_i_plus_1",
"B",
"C",
"D"
]].to_numpy() # NB: `.values` works here too, but is deprecated. See:
# https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.values.html
]
if i < 2 or i > len(df)-2:
continue
...所以我用这些修改创建了这个文件：pandas_dataframe_iteration_vs_vectorization_vs_list_comprehension_speed_tests_mod.py。在文件中搜索“MOD:”以找到4种新的修改技术。
只有轻微的改进。现在这17种技术的结果如下，其中4种新技术的名称开头附近有单词_MOD_，紧跟在它们的编号之后。这次是超过500k行，而不是2M：
.iterrtuples()使用.iterrtuples()时实际上有更多细微之处。要深入了解其中一些细节，请阅读Romain Capron的这个回答。这是我根据他的结果制作的柱状图：
python/pandas_plot_bar_chart_better_GREAT_AUTOLABEL_DATA.py文件中，该文件存放在我的eRCaGuy_hello_world存储库中。subset = row['c1'][0:5]
all = row['c1'][:]
您可能还希望将其转换为数组。这些索引/选择已经应该像NumPy数组一样运行，但我遇到了问题并需要进行转换。
np.asarray(all)
imgs[:] = cv2.resize(imgs[:], (224,224) ) # Resize every image in an hdf5 file
df.index迭代并通过at[]访问一种非常易读的方法是通过索引进行迭代（如@Grag2015所建议）。但是，不要像那里使用链式索引，而是使用at以提高效率：
for ind in df.index:
print(df.at[ind, 'col A'])
相较于 for i in range(len(df)) 的方法，这种方法的优点在于即使索引不是 RangeIndex 也能正常工作。请参考以下示例：
df = pd.DataFrame({'col A': list('ABCDE'), 'col B': range(5)}, index=list('abcde'))
for ind in df.index:
print(df.at[ind, 'col A'], df.at[ind, 'col B']) # <---- OK
df.at[ind, 'col C'] = df.at[ind, 'col B'] * 2 # <---- can assign values
for ind in range(len(df)):
print(df.at[ind, 'col A'], df.at[ind, 'col B']) # <---- KeyError
如果需要行的整数位置（例如获取上一行的值），请使用enumerate()进行包装：
for i, ind in enumerate(df.index):
prev_row_ind = df.index[i-1] if i > 0 else df.index[i]
df.at[ind, 'col C'] = df.at[prev_row_ind, 'col B'] * 2
get_loc 和 itertuples()尽管它比 iterrows() 快得多，但 itertuples() 的一个主要缺点是，如果列标签中包含空格（例如 'col C' 变成了 _1 等），它会混淆列标签，这使得在迭代中访问值变得困难。
您可以使用 df.columns.get_loc() 来获取列标签的整数位置，并将其用于索引 namedtuples。请注意，每个 namedtuple 的第一个元素是索引标签，因此为了正确地通过整数位置访问列，您需要将从 get_loc 返回的任何内容加上 1 或在开头对元组进行解包。
df = pd.DataFrame({'col A': list('ABCDE'), 'col B': range(5)}, index=list('abcde'))
for row in df.itertuples(name=None):
pos = df.columns.get_loc('col B') + 1 # <---- add 1 here
print(row[pos])
for ind, *row in df.itertuples(name=None):
# ^^^^^^^^^ <---- unpacked here
pos = df.columns.get_loc('col B') # <---- already unpacked
df.at[ind, 'col C'] = row[pos] * 2
print(row[pos])
dict_items另一种循环遍历数据框的方法是将其转换为orient='index'中的字典，然后迭代dict_items或dict_values。
df = pd.DataFrame({'col A': list('ABCDE'), 'col B': range(5)})
for row in df.to_dict('index').values():
# ^^^^^^^^^ <--- iterate over dict_values
print(row['col A'], row['col B'])
for index, row in df.to_dict('index').items():
# ^^^^^^^^ <--- iterate over dict_items
df.at[index, 'col A'] = row['col A'] + str(row['col B'])
这个方法不像iterrows那样混淆数据类型，也不像itertuples那样混淆列标签，并且对于列数是不可知的（如果有很多列，zip(df['col A'], df['col B'], ...)会很快变得笨重）。
最后，正如@cs95所提到的，尽可能避免循环。特别是如果你的数据是数字，如果你深入挖掘一下库，就会有一个针对你任务的优化方法。
话虽如此，有些情况下迭代比向量化操作更有效率。一个常见的这种任务是将pandas dataframe转储为嵌套的json。至少在pandas 1.5.3中，使用itertuples()循环比任何涉及groupby.apply方法的向量化操作要快得多。
df.iterrows() 返回一个元组(a, b)，其中 a 是 index，b 是 row。
import pandas as pd
a = [1, 2, 3, 4]
b = [5, 6, 7, 8]
mjr = pd.DataFrame({'a':a, 'b':b})
size = mjr.shape
for i in range(size[0]):
for j in range(size[1]):
print(mjr.iloc[i, j])
age和gender。for index, row in users_df.iterrows():
user_id = row['user_id']
# Trigger expensive network request once for each row
response_dict = backend_api.get(f'/api/user-data/{user_id}')
# Extend dataframe with multiple data from response
users_df.at[index, 'age'] = response_dict.get('age')
users_df.at[index, 'gender'] = response_dict.get('gender')
pandas也是读取csv文件的首选。使用API来操作数据更加易于编程。 - F.S.