使用Beautifulsoup抓取href链接

Question

使用Beautifulsoup抓取href链接

3

我有这样的一段代码

onexurl = "https://1xbet.com/en/live/Football/"
reply = requests.get(onexurl)
soup = BeautifulSoup(reply.content, "html.parser")
links = soup.find_all("a", {"class": "c-events__name"})
print(links)
urls = []
for matchlink in links:
    urls.append("https://1xbet.com/en/"+(matchlink.get("href")))
print(urls)

从页面获取链接。

其中一个结果如下： https://1xbet.com/en/live/Football/24581-AFC-Champions-League/207140194--/ 但原始源代码为：

<a href="live/Football/24581-AFC-Champions-League/207140194-Kashima-Antlers-Guangzhou-Evergrande/" class="c-events__name"><span title="Kashima Antlers — Guangzhou Evergrande  " class="c-events__teams"><div class="c-events-scoreboard__team-wrap"><div class="c-events__team">Kashima Antlers</div> <!----> <!----></div> <div class="c-events-scoreboard__team-wrap"><div class="c-events__team"> Guangzhou Evergrande</div> <!----> <!----></div> <!----> <!----> <!----></span></a>

为什么(matchlink.get("href")不会获取链接的全部文本？

- lture

简短回答，.get("href") 确实可以获取链接的文本。真正的问题是为什么你的BeautifulSoup过滤操作返回的值不符合你的期望。 - Nathan

2个回答

0

这里还有其他事情发生。让我们检查解析器的行为：

html_doc = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>

<p class="story">Once upon a time there were three little sisters; and their names were
<a href="live/Football/24581-AFC-Champions-League/207140194-Kashima-Antlers-Guangzhou-Evergrande/" class="c-events__name"><span title="Kashima Antlers — Guangzhou Evergrande  " class="c-events__teams"><div class="c-events-scoreboard__team-wrap"><div class="c-events__team">Kashima Antlers</div> <!----> <!----></div> <div class="c-events-scoreboard__team-wrap"><div class="c-events__team"> Guangzhou Evergrande</div> <!----> <!----></div> <!----> <!----> <!----></span></a>,
<a href="live/Football/24581-AFC-Champions-League/207140194-Kashima-Antlers-Guangzhou-Evergrande/" class="c-events__name"><span title="Kashima Antlers — Guangzhou Evergrande  " class="c-events__teams"><div class="c-events-scoreboard__team-wrap"><div class="c-events__team">Kashima Antlers</div> <!----> <!----></div> <div class="c-events-scoreboard__team-wrap"><div class="c-events__team"> Guangzhou Evergrande</div> <!----> <!----></div> <!----> <!----> <!----></span></a>
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>"""


soup = BeautifulSoup(html_doc, 'html.parser')
for link in soup.find_all('a', {"class": "c-events__name"}):
    print(link.get('href'))

返回您提供的href标签的完整值。

live/Football/24581-AFC-Champions-League/207140194-Kashima-Antlers-Guangzhou-Evergrande/
live/Football/24581-AFC-Champions-League/207140194-Kashima-Antlers-Guangzhou-Evergrande/

下一步是检查链接如何附加到您创建的列表中以保存它们。我们可以简化这个表达式：

my_list = []
for ml in links:
    my_list.append("http://url.com/" + ml.get("href"))

转换为列表推导式：

my_list = ["http://url.com/" + ml.get("href") for ml in links]

href值应该存储在一个好的列表中。如果出现问题，请确保您的BeautifulSoup 过滤器返回您认为的内容。

- Nathan

感谢@Nathan的建议。我已经查看了BeautifulSoup的过滤替代方案，但没有找到合适的。但我会再次查看，应该会有的。 - lture

网页内容由stack overflow 提供, 点击上面的

可以查看英文原文，
原文链接

- Mrugesh Kadia · Accepted Answer

import requests
from bs4 import BeautifulSoup

onexurl = "https://1xbet.com/en/live/Football/"
reply = requests.get(onexurl)
soup = BeautifulSoup(reply.content, "html.parser")
links = soup.find_all("a", {"class": "c-events__name"})

urls = []
for matchlink in links:
    url = "https://1xbet.com/en/"+(matchlink["href"]).replace('--/', '')
    teams = matchlink.text
    remaining_url = ( teams.strip().replace('\n', '-').replace('(', '-').replace(')', '-').replace(' ', '-').replace('--', '-'))
    final_url = url + '-' + remaining_url
    urls.append(final_url.lower())
print(urls)

这将给您一个URL列表：

['https://1xbet.com/en/live/football/1999982-5h5-dragon-league-league-b/207278079-manchester-city-team-manchester-united-team', 'https://1xbet.com/en/live/football/1471313-indonesia-liga-1/207271440-badak-lampung-kalteng-putra', 'https://1xbet.com/en/live/football/1471313-indonesia-liga-1/207271451-psm-makassar-ps-tira', ]