熊猫数据框架连接与附加

我有一个4个熊猫数据框的列表，包含一天的蜱数据，我想合并到一个单一的数据框。我无法理解我的时间戳上的联系行为。详情如下:

data


[<class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 35228 entries, 2013-03-28 00:00:07.089000+02:00 to 2013-03-28 18:59:20.357000+02:00
Data columns:
Price       4040  non-null values
Volume      4040  non-null values
BidQty      35228  non-null values
BidPrice    35228  non-null values
AskPrice    35228  non-null values
AskQty      35228  non-null values
dtypes: float64(6),
<class 'pandas.core.frame.DataFrame'>


DatetimeIndex: 33088 entries, 2013-04-01 00:03:17.047000+02:00 to 2013-04-01 18:59:58.175000+02:00
Data columns:
Price       3969  non-null values
Volume      3969  non-null values
BidQty      33088  non-null values
BidPrice    33088  non-null values
AskPrice    33088  non-null values
AskQty      33088  non-null values
dtypes: float64(6),
<class 'pandas.core.frame.DataFrame'>


DatetimeIndex: 50740 entries, 2013-04-02 00:03:27.470000+02:00 to 2013-04-02 18:59:58.172000+02:00
Data columns:
Price       7326  non-null values
Volume      7326  non-null values
BidQty      50740  non-null values
BidPrice    50740  non-null values
AskPrice    50740  non-null values
AskQty      50740  non-null values
dtypes: float64(6),
<class 'pandas.core.frame.DataFrame'>


DatetimeIndex: 60799 entries, 2013-04-03 00:03:06.994000+02:00 to 2013-04-03 18:59:58.180000+02:00
Data columns:
Price       8258  non-null values
Volume      8258  non-null values
BidQty      60799  non-null values
BidPrice    60799  non-null values
AskPrice    60799  non-null values
AskQty      60799  non-null values
dtypes: float64(6)]

使用 append我得到:

pd.DataFrame().append(data)


<class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 179855 entries, 2013-03-28 00:00:07.089000+02:00 to 2013-04-03 18:59:58.180000+02:00
Data columns:
AskPrice    179855  non-null values
AskQty      179855  non-null values
BidPrice    179855  non-null values
BidQty      179855  non-null values
Price       23593  non-null values
Volume      23593  non-null values
dtypes: float64(6)

使用 concat我得到:

pd.concat(data)


<class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 179855 entries, 2013-03-27 22:00:07.089000+02:00 to 2013-04-03 16:59:58.180000+02:00
Data columns:
Price       23593  non-null values
Volume      23593  non-null values
BidQty      179855  non-null values
BidPrice    179855  non-null values
AskPrice    179855  non-null values
AskQty      179855  non-null values
dtypes: float64(6)

注意使用 concat时索引的变化。为什么会发生这种情况，我应该如何使用 concat来重现使用 append获得的结果？(因为 concat看起来快得多; 每个循环24.6 ms 比每个循环3.02 s)

201410 次浏览
小开
最佳答案

So what are you doing is with append and concat is almost equivalent. The difference is the empty DataFrame. For some reason this causes a big slowdown, not sure exactly why, will have to look at some point. Below is a recreation of basically what you did.

I almost always use concat (though in this case they are equivalent, except for the empty frame); if you don't use the empty frame they will be the same speed.

In [17]: df1 = pd.DataFrame(dict(A = range(10000)),index=pd.date_range('20130101',periods=10000,freq='s'))


In [18]: df1
Out[18]:
<class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 10000 entries, 2013-01-01 00:00:00 to 2013-01-01 02:46:39
Freq: S
Data columns (total 1 columns):
A    10000  non-null values
dtypes: int64(1)


In [19]: df4 = pd.DataFrame()


The concat


In [20]: %timeit pd.concat([df1,df2,df3])
1000 loops, best of 3: 270 us per loop


This is equavalent of your append


In [21]: %timeit pd.concat([df4,df1,df2,df3])
10 loops, best of


3: 56.8 ms per loop
小开

tl;dr Always use concat since

  1. there is no significant difference between concat and append (see benchmark below) and
  2. append is deprecated anyway.

I cannot reproduce your results: I have implemented a tiny benchmark (please find the code on Gist) to evaluate the pandas' concat and append. I updated the code snippet and the results after the comment by ssk08 - thanks a lot!

The benchmark ran on a Mac OS X 10.13 system with Python 3.6.2 and pandas 0.20.3.

+--------+---------------------------------+---------------------------------+
|        | ignore_index=False              | ignore_index=True               |
+--------+---------------------------------+---------------------------------+
| size   | append | concat | append/concat | append | concat | append/concat |
+--------+--------+--------+---------------+--------+--------+---------------+
| small  | 0.4635 | 0.4891 | 94.77 %       | 0.4056 | 0.3314 | 122.39 %      |
+--------+--------+--------+---------------+--------+--------+---------------+
| medium | 0.5532 | 0.6617 | 83.60 %       | 0.3605 | 0.3521 | 102.37 %      |
+--------+--------+--------+---------------+--------+--------+---------------+
| large  | 0.9558 | 0.9442 | 101.22 %      | 0.6670 | 0.6749 | 98.84 %       |
+--------+--------+--------+---------------+--------+--------+---------------+

Using ignore_index=False append is slightly faster, with ignore_index=True concat is slightly faster.

小开

Pandas concat vs append vs join vs merge

  • Concat gives the flexibility to join based on the axis( all rows or all columns)

  • Append is the specific case(axis=0, join='outer') of concat (being deprecated use concat)

  • Join is based on the indexes (set by set_index) on how variable =['left','right','inner','couter']

  • Merge is based on any particular column each of the two dataframes, this columns are variables on like 'left_on', 'right_on', 'on'

小开

One more thing you have to keep in mind that the APPEND() method in Pandas doesn't modify the original object. Instead it creates a new one with combined data. Because of involving creation and data buffer, its performance is not well. You'd better use CONCAT() function when doing multi-APPEND operations.


提交答案