如何在Python中使用selenium webdriver和browsermob代理来捕获网络流量？

Question

我希望您能够用Python的Selenium Webdriver来捕获网络流量。因此，我必须使用代理（如BrowserMobProxy）。

当我使用webdriver.Chrome时：

from browsermobproxy import Server

server = Server("~/browsermob-proxy")

server.start()
proxy = server.create_proxy()

from selenium import webdriver
co = webdriver.ChromeOptions()
co.add_argument('--proxy-server={host}:{port}'.format(host='localhost', port=proxy.port))

driver = webdriver.Chrome(executable_path = "~/chromedriver", chrome_options=co)

proxy.new_har
driver.get(url)
proxy.har # returns a HAR 

for ent in proxy.har['log']['entries']:
    print ent['request']['url']

网页已经正确加载，所有请求在HAR文件中都可用和可访问。 但当我使用webdriver.Firefox时：

# The same as above
# ...
from selenium import webdriver
profile  = webdriver.FirefoxProfile()
driver = webdriver.Firefox(firefox_profile=profile, proxy = proxy.selenium_proxy())

proxy.new_har
driver.get(url)
proxy.har # returns a HAR

for ent in proxy.har['log']['entries']:
    print ent['request']['url']

网页无法正常加载，HAR文件中的请求数量小于应有的请求数量。

你对第二段代码中的代理设置问题有什么想法吗？我该如何修复它以便正确地使用webdriver.Firefox达到我的目的？

- Jose

Answer 1

我刚刚偶然发现了这个项目https://github.com/derekargueta/selenium-profiler。它可以输出URL的所有网络数据。很容易就能够将其集成到你正在运行的任何测试中。原始来源：https://www.openhub.net/p/selenium-profiler。

- Derek Argueta

Answer 2

对我来说，以下代码组件运行良好。

profile = webdriver.FirefoxProfile()
profile.set_proxy(proxy.selenium_proxy())
driver = webdriver.Firefox(firefox_profile=profile)

- proprius

---

当使用https时，这对您有效吗？对我来说不起作用（即使设置了profile.accept_untrusted_certs = True）。 - user3468054

Answer 3

I am sharing my solution, this would not write any logs to any file
but you can collect all sort of messages such as Errors, 
Warnings, Logs, Info, Debug , CSS, XHR as well as Requests(traffic)

1. We are going to create Firefox profile so that we can enable option of 
"Persist Logs" on Firefox (you can try it to enable on your default browser and see 
if it launches with "Persist Logs" without creating firefox profile )

2. we need to modify the Firefox initialize code 
where this line will do magic : options.AddArgument("--jsconsole");

so complete Selenium Firefox code would be,  this will open Browser Console
everytime you execute your automation :

 else if (browser.Equals(Constant.Firefox))
            {

                var profileManager = new FirefoxProfileManager();
                FirefoxProfile profile = profileManager.GetProfile("ConsoleLogs");
                FirefoxDriverService service = FirefoxDriverService.CreateDefaultService(DrivePath);
                service.FirefoxBinaryPath = DrivePath;                
                profile.SetPreference("security.sandbox.content.level", 5);
                profile.SetPreference("dom.webnotifications.enabled", false);
                profile.AcceptUntrustedCertificates = true;
                FirefoxOptions options = new FirefoxOptions();
                options.AddArgument("--jsconsole");
                options.AcceptInsecureCertificates = true;
                options.Profile = profile;
                options.SetPreference("browser.popups.showPopupBlocker", false);
                driver = new FirefoxDriver(service.FirefoxBinaryPath, options, TimeSpan.FromSeconds(100));
                driver.Manage().Timeouts().ImplicitWait = TimeSpan.FromSeconds(10);               

            }

3. Now you can write your logic since you have traffic/ logging window open so don't 
go to next execution if test fails. That way Browser Console will keep your errors 
messages and help you to troubleshoot further 

Browser : Firefox  v 61 

How can you launch Browser Console for firefox:
1. open firefox (and give any URL )
2. Press Ctrl+Shift+J (or Cmd+Shift+J on a Mac) 

Link : https://developer.mozilla.org/en-US/docs/Tools/Browser_Console  

- Mike ASP