在特定类之后获取href标签中的文本

Question

我正在尝试爬取一个网页

library(RCurl)
webpage <- getURL("https://somewebpage.com")

webpage

<div class='CredibilityFacts'><span id='qZyoLu'><a class='answer_permalink'
action_mousedown='AnswerPermalinkClickthrough' href='/someurl/answer/my_id' 
id ='__w2_yeSWotR_link'>
<a class='another_class' action_mousedown='AnswerPermalinkClickthrough' 
href='/ignore_url/answer/some_id' id='__w2_ksTVShJ_link'>
<a class='answer_permalink' action_mousedown='AnswerPermalinkClickthrough' 
href='/another_url/answer/new_id' id='__w2_ksTVShJ_link'>

class(webpage)
[1] "character"

我要提取所有 href 值，但仅在其前面有 answer_permalink 类时才提取。

输出结果应该是：

[1] "/someurl/answer/my_id"  "/another_url/answer/new_id"

/ignore_url/answer/some_id应该被忽略，因为它前面有another_class而不是answer_permalink类。

现在，我考虑使用正则表达式的方法。我认为可以在stri_extract_all中使用类似这样的正则表达式。

class='answer_permalink'.*href='

但这不是我想要的。

我应该怎么做才能实现我的需求？此外，除了使用正则表达式，是否在 R 中有类似于 JavaScript 中按 class 提取元素的函数？

- Ronak Shah

---

你应该可以使用 rvest 包来完成这个任务，类似于 read_html(webpage) %>% html_nodes("answer_permalink") %>% html_attr("href")。 - Andrew Gustar

@AndrewGustar 返回我 character(0)。 - Ronak Shah

Answer 1

利用 dplyr 和 rvest ，我们可以实现以下操作：

library(rvest)
library(dplyr)

"https://www.quora.com/profile/Ronak-Shah-96" %>% 
  read_html() %>% 
  html_nodes("[class='answer_permalink']") %>% 
  html_attr("href")

[1] "/How-can-we-adjust-in-engineering-if-we-are-not-in-IITs-or-NITs-How-can-we-enjoy-engineering-if-we-are-pursuing-it-from-a-local-private-college/answer/Ronak-Shah-96"                                                                        
[2] "/Do-you-think-it-is-worth-it-to-change-my-career-path-For-the-past-2-years-I-was-pursuing-a-career-in-tax-advisory-in-a-BIG4-company-I-just-got-a-job-offer-that-will-allow-me-to-learn-coding-It-is-not-that-well-paid/answer/Ronak-Shah-96"
[3] "/Why-cant-India-opt-for-40-hours-work-a-week-for-all-professions-when-it-is-proved-and-working-well-in-terms-of-efficiency/answer/Ronak-Shah-96"

[4] "/Why-am-I-still-confused-and-thinking-about-my-career-after-working-more-than-one-year-in-software-engineering/answer/Ronak-Shah-96"

[5] "/Would-you-rather-be-a-jack-of-all-trades-or-the-master-of-one-trade/answer/Ronak-Shah-96"

- tyluRp

Answer 2

可以使用像rvest或者xml2这样的包，而不是字符串解析:

library(xml2)
xml <- read_html(webpage)
l <- as_list(xml)[[1]][[1]][[1]][[1]]  #not sure why you need to go this deep.

l2 <- l[sapply(l, attr, ".class") == "answer_permalink"]
sapply(l2, attr, "href")

                       a                            a 
 "/someurl/answer/my_id" "/another_url/answer/new_id"

- Axeman

---

sapply(l, attr, ".class") 给我输出为 [[1]] NULL。我做错了什么吗？而且 length(l) 是 1。 - Ronak Shah

我不知道，我只是将你的“网页”作为字符串读取，并运行上面的确切代码。 - Axeman

不得不少深入一层。l <- as_list(xml)[[1]][[1]]。不确定发生了什么变化。 - Ronak Shah

是的，我不确定这个可靠性如何，一般来说安德鲁的评论应该是正确的，但我也无法让它工作。 - Axeman

Answer 3

require(XML)
require(RCurl)

doc <- getURL("https://www.quora.com/profile/Ronak-Shah-96" )
html <- htmlTreeParse(doc, useInternalNodes = TRUE)
nodes <- getNodeSet(html, "//a[@class='answer_permalink']")
sapply(nodes, function(x) x <- xmlAttrs(x)[["href"]])

[1] "/Do-you-think-it-is-worth-it-to-change-my-career-path-For-the-past-2-years-I-was-pursuing-a-career-in-tax-advisory-in-a-BIG4-company-I-just-got-a-job-offer-that-will-allow-me-to-learn-coding-It-is-not-that-well-paid/answer/Ronak-Shah-96"
[2] "/Why-cant-India-opt-for-40-hours-work-a-week-for-all-professions-when-it-is-proved-and-working-well-in-terms-of-efficiency/answer/Ronak-Shah-96"                                                                                             
[3] "/Why-am-I-still-confused-and-thinking-about-my-career-after-working-more-than-one-year-in-software-engineering/answer/Ronak-Shah-96"                                                                                                         
[4] "/Would-you-rather-be-a-jack-of-all-trades-or-the-master-of-one-trade/answer/Ronak-Shah-96"                                                                                                                                                   
[5] "/Is-software-engineering-a-good-career-choice-I-know-it-pays-well-initially-but-if-you-look-at-the-managing-directors-of-most-companies-they-are-people-with-MBAs/answer/Ronak-Shah-96"                                                      

- germcd