如何在Python中从XML/SOAP中提取数据

Question

英国国家天然气系统发布了大量数据，可以通过SOAP服务器访问，下面是返回的LNG数据示例。我已经编写代码生成请求并处理响应，但在如何提取返回的信息方面遇到了问题。目标是将数据上传到后端数据库或Pandas DataFrame中。之前，我只是使用XPATH遍历XML，然后迭代标记并提取子数据，因此，我希望提取以下内容：

GetPublicationDataWMResult, ApplicableAt, ApplicableFor, Value, ...
LNG Stock Level,2016-03-13T15:00:07Z, 2016-03-12T00:00:00Z, 7050.42286, ...
LNG Capacity,2016-03-13T15:00:07Z, 2016-03-12T00:00:00Z, 6515042480, ...

尝试使用XPATH遍历子元素(/Envelope/Body/GetPublicationDataWMResponse/GetPublicationDataWMResult/)，但失败了。

如果我通过添加一系列字符串删除来清理代码，则逻辑可行，但这是次优的，并且将来可能会出现问题。

示例代码：

import requests
from lxml import objectify

def getXML():

    toDate = "2016-03-12"
    fromDate = "2016-03-12"
    dateType = "gasday"

    url="http://marketinformation.natgrid.co.uk/MIPIws-public/public/publicwebservice.asmx"
    headers = {'content-type': 'application/soap+xml; charset=utf-8'}

    body ="""<soap12:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap12="http://www.w3.org/2003/05/soap-envelope">
        <soap12:Body>
            <GetPublicationDataWM xmlns="http://www.NationalGrid.com/MIPI/">
                <reqObject>
                    <LatestFlag>Y</LatestFlag>
                    <ApplicableForFlag>Y</ApplicableForFlag>
                    <ToDate>%s</ToDate>
                    <FromDate>%s</FromDate>
                    <DateType>%s</DateType>
                    <PublicationObjectNameList>
                        <string>LNG Stock Level</string>
                        <string>LNG, Daily Aggregated Available Capacity, D+1</string>
                    </PublicationObjectNameList>
                </reqObject>
            </GetPublicationDataWM>
        </soap12:Body>
    </soap12:Envelope>""" % (toDate, fromDate,dateType)


    response = requests.post(url,data=body,headers=headers)

    return response.content

root = objectify.fromstring(getXML())

返回的XML：

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope
    xmlns:soap="http://www.w3.org/2003/05/soap-envelope"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:xsd="http://www.w3.org/2001/XMLSchema">
    <soap:Body>
        <GetPublicationDataWMResponse
            xmlns="http://www.NationalGrid.com/MIPI/">
            <GetPublicationDataWMResult>
                <CLSMIPIPublicationObjectBE>
                    <PublicationObjectName>LNG Stock Level</PublicationObjectName>
                    <PublicationObjectData>
                        <CLSPublicationObjectDataBE>
                            <ApplicableAt>2016-03-13T15:00:07Z</ApplicableAt>
                            <ApplicableFor>2016-03-12T00:00:00Z</ApplicableFor>
                            <Value>7050.42286</Value>
                            <GeneratedTimeStamp>2016-03-13T15:56:00Z</GeneratedTimeStamp>
                            <QualityIndicator></QualityIndicator>
                            <Substituted>N</Substituted>
                            <CreatedDate>2016-03-13T15:56:28Z</CreatedDate>
                        </CLSPublicationObjectDataBE>
                    </PublicationObjectData>
                </CLSMIPIPublicationObjectBE>
                <CLSMIPIPublicationObjectBE>
                    <PublicationObjectName>LNG Capacity</PublicationObjectName>
                    <PublicationObjectData>
                        <CLSPublicationObjectDataBE>
                            <ApplicableAt>2016-03-12T15:30:00Z</ApplicableAt>
                            <ApplicableFor>2016-03-12T00:00:00Z</ApplicableFor>
                            <Value>6515042480</Value>
                            <GeneratedTimeStamp>2016-03-12T16:00:00Z</GeneratedTimeStamp>
                            <QualityIndicator></QualityIndicator>
                            <Substituted>N</Substituted>
                            <CreatedDate>2016-03-12T16:00:20Z</CreatedDate>
                        </CLSPublicationObjectDataBE>
                    </PublicationObjectData>
                </CLSMIPIPublicationObjectBE>
            </GetPublicationDataWMResult>
        </GetPublicationDataWMResponse>
    </soap:Body>
</soap:Envelope>

- Patrick A

Answer 1

使用您现有的代码，我只是添加了这个：

。

res= getXML()

from bs4 import BeautifulSoup
soup = BeautifulSoup(res, 'html.parser')

searchTerms= ['PublicationObjectName','ApplicableAt','ApplicableFor','Value']
# LNG Stock Level,2016-03-13T15:00:07Z, 2016-03-12T00:00:00Z, 7050.42286, ...

for st in searchTerms:
    print st+'\t',
    print soup.find(st.lower()).contents[0]

输出：

PublicationObjectName   LNG Stock Level
ApplicableAt    2016-03-13T15:00:07Z
ApplicableFor   2016-03-12T00:00:00Z
Value   7050.42286

- roadrunner66

Answer 2

这是一个涉及XML和XPath主题的常见问题解答，其中涉及到具有默认命名空间的XML。

在声明默认命名空间的XML元素中，其没有前缀的后代元素会隐式继承相同的默认命名空间。而在XPath表达式中，要引用命名空间中的元素，您需要使用已映射到相应命名空间URI的前缀。使用“lxml”，代码将如下所示：

root = etree.fromstring(getXML())

# map prefix 'd' to the default namespace URI
ns = { 'd': 'http://www.NationalGrid.com/MIPI/'}

publication_objects = root.xpath('//d:CLSMIPIPublicationObjectBE', namespaces=ns)
for obj in publication_objects:
    name = obj.find('d:PublicationObjectName', ns).text

    data = obj.find('d:PublicationObjectData/d:CLSPublicationObjectDataBE', ns)
    applicable_at = data.find('d:ApplicableAt', ns).text
    applicable_for = data.find('d:ApplicableFor', ns).text
    # todo: extract other relevant data and process as needed

- har07