我正在尝试编写一个快速的HTML爬虫，目前我只关注如何最大化吞吐量而不进行解析。我已经缓存了URL的IP地址：
我的下一步是通过从互联网获取100个URL，读取前64KB并移动到下一个URL来测试速度。我创建了一个
public class Data {
private static final ArrayList<String> sites = new ArrayList<String>();
public static final ArrayList<URL> URL_LIST = new ArrayList<URL>();
public static final ArrayList<InetAddress> ADDRESSES = new ArrayList<InetAddress>();
static{
/*
add all the URLs to the sites array list
*/
// Resolve the DNS prior to testing the throughput
for(int i = 0; i < sites.size(); i++){
try {
URL tmp = new URL(sites.get(i));
InetAddress address = InetAddress.getByName(tmp.getHost());
ADDRESSES.add(address);
URL_LIST.add(new URL("http", address.getHostAddress(), tmp.getPort(), tmp.getFile()));
System.out.println(tmp.getHost() + ": " + address.getHostAddress());
} catch (MalformedURLException e) {
} catch (UnknownHostException e) {
}
}
}
}
我的下一步是通过从互联网获取100个URL，读取前64KB并移动到下一个URL来测试速度。我创建了一个
FetchTaskConsumer的线程池，并尝试运行多个线程（在i7四核机器上运行了16到64个线程），每个消费者的外观如下：public class FetchTaskConsumer implements Runnable{
private final CountDownLatch latch;
private final int[] urlIndexes;
public FetchTaskConsumer (int[] urlIndexes, CountDownLatch latch){
this.urlIndexes = urlIndexes;
this.latch = latch;
}
@Override
public void run() {
URLConnection resource;
InputStream is = null;
for(int i = 0; i < urlIndexes.length; i++)
{
int numBytes = 0;
try {
resource = Data.URL_LIST.get(urlIndexes[i]).openConnection();
resource.setRequestProperty("User-Agent", "Mozilla/5.0");
is = resource.getInputStream();
while(is.read()!=-1 && numBytes < 65536 )
{
numBytes++;
}
} catch (IOException e) {
System.out.println("Fetch Exception: " + e.getMessage());
} finally {
System.out.println(numBytes + " bytes for url index " + urlIndexes[i] + "; remaining: " + remaining.decrementAndGet());
if(is!=null){
try {
is.close();
} catch (IOException e1) {/*eat it*/}
}
}
}
latch.countDown();
}
}
我最多可以在30秒内浏览100个URL，但文献表明我应该能够每秒处理150个URL，需要注意的是，我可以访问Gigabit以太网，尽管我目前在家中使用20 Mbit连接运行测试...无论哪种情况，连接都没有真正被充分利用。
我已经尝试直接使用Socket连接，但速度更慢了！您有什么建议可以提高吞吐量吗？
P.S.
我有一个大约包含100万个流行URL的列表，因此如果100个URL不足以进行基准测试，我可以添加更多URL。
更新：
我所指的文献是与Najork Web爬虫相关的论文，Najork表示：
更新:
好吧，最终结果出来了！事实证明，100个URL对于基准测试来说有点低。我将其增加到1024个URL、64个线程，并设置每次获取的超时时间为2秒，我能够达到每秒21页（实际上我的连接速度约为10.5 Mbps，因此每秒21页*每页64KB大约为10.5 Mbps）。以下是抓取程序的外观：
public class FetchTask implements Runnable{
private final int timeoutMS = 2000;
private final CountDownLatch latch;
private final int[] urlIndexes;
public FetchTask(int[] urlIndexes, CountDownLatch latch){
this.urlIndexes = urlIndexes;
this.latch = latch;
}
@Override
public void run() {
URLConnection resource;
InputStream is = null;
for(int i = 0; i < urlIndexes.length; i++)
{
int numBytes = 0;
try {
resource = Data.URL_LIST.get(urlIndexes[i]).openConnection();
resource.setConnectTimeout(timeoutMS);
resource.setRequestProperty("User-Agent", "Mozilla/5.0");
is = resource.getInputStream();
while(is.read()!=-1 && numBytes < 65536 )
{
numBytes++;
}
} catch (IOException e) {
System.out.println("Fetch Exception: " + e.getMessage());
} finally {
System.out.println(numBytes + "," + urlIndexes[i] + "," + remaining.decrementAndGet());
if(is!=null){
try {
is.close();
} catch (IOException e1) {/*eat it*/}
}
}
}
latch.countDown();
}
}