您需要模拟Ajax请求来获取表格数据:
import requests
from time import time
optionstable = "http://www.barchart.com/options/optdailyvol?type=stocks"
params = {"type": "stocks",
"dir": "desc",
"_": str(time()),
"f": "base_symbol,type,strike,expiration_date,bid,ask,last,volume,open_interest,volatility,timestamp",
"sEcho": "1",
"iDisplayStart": "0",
"iDisplayLength": "100",
"iSortCol_0": "7",
"sSortDir_0": "desc",
"iSortingCols": "1",
"bSortable_0": "true",
"bSortable_1": "true",
"bSortable_2": "true",
"bSortable_3": "true",
"bSortable_4": "true",
"bSortable_5": "true",
"bSortable_6": "true",
"bSortable_7": "true",
"bSortable_82": "true",
"bSortable_9": "true",
"bSortable_10": "true",
"sortby": "Volume"}
然后执行一个带参数的get请求:
js = requests.get("http://www.barchart.com/option-center/getData.php", params=params).json()
这给您带来:
{u'aaData': [[u'<a href="/quotes/BAC">BAC</a>', u'Call', u'16.00', `u'12/16/16', u'0.89', u'0.90', u'0.91', u'52,482', u'146,378', u'0.26', u'01:43'], [u'<a href="/quotes/ETE">ETE</a>', u'Call', u'20.00', u'01/20/17', u'0.38', u'0.41', u'0.40', u'40,785', u'72,011', u'0.42', u'01:27'], [u'<a href="/quotes/BAC">BAC</a>', u'Call', u'15.00', u'10/21/16', u'1.34', u'1.36', u'1.33', u'35,663', u'90,342', u'0.35', u'01:44'], [u'<a href="/quotes/COTY">COTY</a>', u'Put', u'38.00', u'10/21/16', u'15.00', u'15.30', u'15.10', u'32,321', u'242,382', u'1.24', u'01:44'], [u'<a href="/quotes/COTY">COTY</a>', u'Call', u'38.00', u'10/21/16', u'0.00', u'0.05', u'0.01', u'32,320', u'256,589', u'1.34', u'01:44'], [u'<a href="/quotes/WFC">WFC</a>', u'Put', u'40.00', u'10/21/16', u'0.01', u'0.03', u'0.02', u'32,121', u'37,758', u'0.39', u'01:43'], [u'<a href="/quotes/WFC">WFC</a>', u'Put', u'40.00', u'11/18/16', u'0.16', u'0.17', u'0.16', u'32,023', u'8,789', u'0.30', u'01:44']..................
如果您在Chrome工具中查看XHR选项卡下的请求,您可以看到所有params。上述参数是获取结果所需的最低限度。还有很多其他参数,由于数量较多,我不会在此列出,留给您自己思考如何影响结果。
如果您遍历js[u'aaData']
,您可以看到每个子列表,其中每个条目对应于列,例如:
#base_symbol,type,strike,expiration_date,bid,ask,last,volume,open_interest,volatility,timestamp
[u'<a href="/quotes/AAPL">AAPL</a>', u'Call', u'116.00', u'10/14/16', u'1.36', u'1.38', u'1.37', u'21,812', u'7,258', u'0.23', u'10/10/16']
所以,如果你想根据某些条件过滤行,例如打击率>15:
for d in filter(lambda row: float(row[2]) > 15, js[u'aaData']):
print(d)
你可能也会发现pandas很有用,只需稍微整理一下就可以创建一个漂亮的数据框:
for v in js[u'aaData']:
v[0] = BeautifulSoup(v[0]).a.text
import pandas as pd
cols = "base_symbol,type,strike,expiration_date,bid,ask,last,volume,open_interest,volatility,timestamp"
df = pd.DataFrame(js[u'aaData'],
columns=cols.split(","))
print(df.head(5))
这将为您提供一个不错的数据框以供使用:
base_symbol type strike expiration_date bid ask last volume \
0 BAC Call 16.00 12/16/16 0.89 0.90 0.91 52,482
1 ETE Call 20.00 01/20/17 0.38 0.41 0.40 40,785
2 BAC Call 15.00 10/21/16 1.34 1.36 1.33 35,663
3 COTY Put 38.00 10/21/16 15.00 15.30 15.10 32,321
4 COTY Call 38.00 10/21/16 0.00 0.05 0.01 32,320
open_interest volatility timestamp
0 146,378 0.26 10/10/16
1 72,011 0.42 10/10/16
2 90,342 0.35 10/10/16
3 242,382 1.24 10/10/16
4 256,589 1.34 10/10/16
您可能只想更改数据类型,例如 df["strike"] = df["strike"].astype(float)
。
find(id="tableContainer")
,则会发现它为空。 - Padraic Cunningham