使用网络爬虫爬取多个相似页面

Question

我刚学习Python网络爬虫，尝试获取加拿大不同Winmar位置的地址，并将结果放入CSV文件中。到目前为止，我发现唯一区分不同位置网站的方法是地址末尾的代码（数字）。问题是，当程序运行时，结果不会改变，而是在打印和写入CSV文件时产生第一个位置（305）的结果。感谢您的时间和考虑！

以下是我的代码：

import csv
import requests
from bs4 import BeautifulSoup

x = 0
numbers = ['305', '405', '306', '307', '308', '309', '4273']

f = csv.writer(open('Winmar_locations.csv', 'w'))
f.writerow(['City:', 'Address:'])

for links in numbers:

    for x in range(0, 6):
        url = 'https://www.winmar.ca/find-a-location/' + str(numbers[x])
        r = requests.get(url)
        soup = BeautifulSoup(r.content, "html.parser")

    location_name = soup.find("div", attrs={"class": "title_block"})
    location_name_items = location_name.find_all('h2')

    location_list = soup.find(class_='quick_info')
    location_list_items = location_list.find_all('p')

    for name in location_name_items:
        names = name.text
        names = names.replace('Location | ', '')

    for location in location_list_items:
        locations = location.text.strip()
        locations = locations.replace('24 Hour Emergency | [phone]','')

    print(names, locations)
    x = x+1

    f.writerow([names, locations])

- Yolomashi

Answer 1

你的代码和你所爬取的网站有一些问题。

• 首先，以这种方式访问网址https://www.winmar.ca/find-a-location/308将不能正确更改位置，需要像这样https://www.winmar.ca/find-a-location/#308在数字前加入hashbang。

• 该网站有重复的html和相同的类，这意味着你几乎已经加载了所有位置并且它们只是从他们的js代码中选择要显示哪个 -这是一个不好的做法-，这会使你的匹配器始终获取相同的位置，这就解释了为什么你一直重复相同的位置。

• 最后，你有很多不必要的循环，你只需要循环数字数组就可以了。

以下是你的修改后的代码：

import csv
import requests
from bs4 import BeautifulSoup

x = 0
numbers = ['305', '405', '306', '307', '308', '309', '4273']


names = []
locations = []
for x in range(0, 6):
    url = 'https://www.winmar.ca/find-a-location/#' + str(numbers[x])
    print(f"pinging url {url}")

    r = requests.get(url)
    soup = BeautifulSoup(r.content, "html.parser")
    scope = soup.find(attrs={"data-id": str(numbers[x])})

    location_name = scope.find("div", attrs={"class": "title_block"})
    location_name_items = location_name.find_all('h2')


    location_list = scope.find(class_='quick_info')
    location_list_items = location_list.find_all('p')

    name = location_name.find_all("h2")[0].text
    print(name)

    names.append(name)

    for location in location_list_items:
        loc = location.text.strip()
        if '24 Hour Emergency' in loc: 
            continue
        print(loc)
        locations.append(loc)

    x = x+1

注意我所做的作用域

    scope = soup.find(attrs={"data-id": str(numbers[x])})

这使得您的代码不受html中加载了多少位置的影响，您只需要针对您想要的位置定位它的范围。

这会导致：

pinging url https://www.winmar.ca/find-a-location/#305
Location | Annapolis
70 Donald E Hiltz Connector Road
Kentville, NS
B4N 3V7
pinging url https://www.winmar.ca/find-a-location/#405
Location | Bridgewater
15585 Highway # 3
Hebbville, NS
B4V 6X7
pinging url https://www.winmar.ca/find-a-location/#306
Location | Halifax
9 Isnor Dr
Dartmouth, NS
B3B 1M1
pinging url https://www.winmar.ca/find-a-location/#307
Location | New Glasgow
5074 Hwy. #4, RR #1
Westville, NS
B0K 2A0
pinging url https://www.winmar.ca/find-a-location/#308
Location | Port Hawkesbury
8 Industrial Park Rd
Lennox Passage, NS
B0E 1V0
pinging url https://www.winmar.ca/find-a-location/#309
Location | Sydney
358 Keltic Drive
Sydney River, NS
B1R 1V7

- kareem_emad

---

非常感谢！我不知道网站上的这些内容。非常感谢您深入的解释，我不知道该如何感谢您！祝您有美好的一天。 - Yolomashi

Answer 2

尽管您已经得到了一个合格的答案，但我想提出我的看法。我试图让脚本简洁明了，摆脱冗长。请确保您的bs4版本是4.7.0或更高版本，以便支持伪选择器，我在脚本中定义了它来定位地址。

import csv
import requests
from bs4 import BeautifulSoup

base = 'https://www.winmar.ca/find-a-location/#{}'

numbers = ['305', '405', '306', '307', '308', '309', '4273']

with open("Winmar_locations.csv","w",newline="") as f:
    writer = csv.writer(f)
    writer.writerow(['City','Address'])

    while numbers:
        num = numbers.pop(0)
        r = requests.get(base.format(num))
        soup = BeautifulSoup(r.content,"html.parser")

        location_name = soup.select_one(f"[data-id='{num}'] .title_block > h2.title").contents[-1]
        location_address = soup.select_one(f"[data-id='{num}'] .heading:contains('Address') + p").get_text(strip=True)
        writer.writerow([location_name,location_address])
        print(location_name,location_address)

- asmitu

---

非常感谢您的解决方案，我会在有机会时尝试两种方法（我现在正在使用手机）。谢谢你，祝你有愉快的一天！ - Yolomashi