使用Kimurai宝石进行网络爬虫

Question

使用Kimurai宝石进行网络爬虫

3

我正在使用Kimurai Ruby gem进行网页抓取。我有一个很好的脚本：

require 'kimurai'

class SimpleSpider < Kimurai::Base
  @name = "simple_spider"
  @engine = :selenium_chrome
  @start_urls = ["https://apply.workable.com/taxjar/"]

  def parse(response, url:, data: {})
    # Update response to current response after interaction with a browser
    count = 0
    # browser.click_button "Show more"
    doc = browser.current_response
    returned_jobs = doc.css('.careers-jobs-list-styles__jobsList--3_v12')
    returned_jobs.css('li').each do |char_element|
        # puts char_element
        title = char_element.css('a')[0]['aria-label']
        link = "https://apply.workable.com" + char_element.css('a')[0]['href']

        #click on job link and get description
        browser.visit(link)
        job_page = browser.current_response
        description = job_page.xpath('/html/body/div[1]/div/div[1]/div[2]/div[2]/div[2]').text

        puts '*******'
        puts title
      puts link
        puts description
        puts count += 1
    end
    puts "There are #{count} jobs total"
  end
end

SimpleSpider.crawl!

然而，我希望所有这些都返回一个对象数组...或者在这种情况下是工作。我想在解析方法中创建一个jobs数组，并在returned_jobs循环内执行类似于jobs << [title, link, description, company]的操作，当我调用SimpleSpider.crawl!时返回它，但那样做不起作用。

任何帮助都会受到赞赏。

- Mark Locklear

2个回答

2

原来有一个解析方法，可以返回一个值。以下是一个可用的示例：

解析方法：

require 'open-uri'
require 'nokogiri'
require 'kimurai'

class TaxJar < Kimurai::Base
  @name = "tax_jar"
  @engine = :selenium_chrome
  @start_urls = ["https://apply.workable.com/taxjar/"]

  def parse(response, url:, data: {})
  jobs = Array.new

    doc = browser.current_response
    returned_jobs = doc.css('.careers-jobs-list-styles__jobsList--3_v12')
    returned_jobs.css('li').each do |char_element|
      title = char_element.css('a')[0]['aria-label']
      link = "https://apply.workable.com" + char_element.css('a')[0]['href']

      #click on job link and get description
      browser.visit(link)
      job_page = browser.current_response
      description = job_page.xpath('/html/body/div[1]/div/div[1]/div[2]/div[2]/div[2]').text
      company = 'TaxJar'
      puts "title is: #{title}, link is: #{link}, \n description is: #{description}"
      jobs << [title, link, description, company]
    end
    return jobs
  end
end

jobs = TaxJar.parse!(:parse, url: "https://apply.workable.com/taxjar/")
puts jobs.inspect

如果您正在爬取JavaScript网站，与我试过的其他工具（waitr/selenium）相比，这个宝石看起来非常强大。

- Mark Locklear

网页内容由stack overflow 提供, 点击上面的

可以查看英文原文，
原文链接

- Lalu · Accepted Answer

您可以稍微修改您的代码，像这样：

class SimpleSpider < Kimurai::Base
  @name = "simple_spider"
  @engine = :selenium_chrome
  @start_urls = ["https://apply.workable.com/taxjar/"]

  def parse(response, url:, data: {})
    # Update response to current response after interaction with a browser
    count = 0
    # browser.click_button "Show more"
    doc = browser.current_response
    returned_jobs = doc.css('.careers-jobs-list-styles__jobsList--3_v12')

    jobs = []
    returned_jobs.css('li').each do |char_element|
        # puts char_element
        title = char_element.css('a')[0]['aria-label']
        link = "https://apply.workable.com" + char_element.css('a')[0]['href']

        #click on job link and get description
        browser.visit(link)
        job_page = browser.current_response
        description = job_page.xpath('/html/body/div[1]/div/div[1]/div[2]/div[2]/div[2]').text

        jobs << [title, link, description]
    end

    puts "There are #{jobs.count} jobs total"
    puts jobs
  end
end

我不确定公司是否在你的代码中有定义该变量。然而，你可以查看上面调用数组并对其进行操作的想法。

以下是在终端运行时的部分输出：

我还有一篇关于如何从Ruby on Rails应用程序中使用Kimurai框架的博客文章在这里。