public class Data {
private static final ArrayList<String> sites = new ArrayList<String>();
public static final ArrayList<URL> URL_LIST = new ArrayList<URL>();
public static final ArrayList<InetAddress> ADDRESSES = new ArrayList<InetAddress>();
static{
/*
add all the URLs to the sites array list
*/
// Resolve the DNS prior to testing the throughput
for(int i = 0; i < sites.size(); i++){
try {
URL tmp = new URL(sites.get(i));
InetAddress address = InetAddress.getByName(tmp.getHost());
ADDRESSES.add(address);
URL_LIST.add(new URL("http", address.getHostAddress(), tmp.getPort(), tmp.getFile()));
System.out.println(tmp.getHost() + ": " + address.getHostAddress());
} catch (MalformedURLException e) {
} catch (UnknownHostException e) {
}
}
}
}
我的下一步是通过从互联网获取100个URL,读取前64KB并移动到下一个URL来测试速度。我创建了一个
FetchTaskConsumer
的线程池,并尝试运行多个线程(在i7四核机器上运行了16到64个线程),每个消费者的外观如下:public class FetchTaskConsumer implements Runnable{
private final CountDownLatch latch;
private final int[] urlIndexes;
public FetchTaskConsumer (int[] urlIndexes, CountDownLatch latch){
this.urlIndexes = urlIndexes;
this.latch = latch;
}
@Override
public void run() {
URLConnection resource;
InputStream is = null;
for(int i = 0; i < urlIndexes.length; i++)
{
int numBytes = 0;
try {
resource = Data.URL_LIST.get(urlIndexes[i]).openConnection();
resource.setRequestProperty("User-Agent", "Mozilla/5.0");
is = resource.getInputStream();
while(is.read()!=-1 && numBytes < 65536 )
{
numBytes++;
}
} catch (IOException e) {
System.out.println("Fetch Exception: " + e.getMessage());
} finally {
System.out.println(numBytes + " bytes for url index " + urlIndexes[i] + "; remaining: " + remaining.decrementAndGet());
if(is!=null){
try {
is.close();
} catch (IOException e1) {/*eat it*/}
}
}
}
latch.countDown();
}
}
我最多可以在30秒内浏览100个URL,但文献表明我应该能够每秒处理150个URL,需要注意的是,我可以访问Gigabit以太网,尽管我目前在家中使用20 Mbit连接运行测试...无论哪种情况,连接都没有真正被充分利用。
我已经尝试直接使用Socket
连接,但速度更慢了!您有什么建议可以提高吞吐量吗?
P.S.
我有一个大约包含100万个流行URL的列表,因此如果100个URL不足以进行基准测试,我可以添加更多URL。
更新:
我所指的文献是与Najork Web爬虫相关的论文,Najork表示:
更新:
好吧,最终结果出来了!事实证明,100个URL对于基准测试来说有点低。我将其增加到1024个URL、64个线程,并设置每次获取的超时时间为2秒,我能够达到每秒21页(实际上我的连接速度约为10.5 Mbps,因此每秒21页*每页64KB大约为10.5 Mbps)。以下是抓取程序的外观:
public class FetchTask implements Runnable{
private final int timeoutMS = 2000;
private final CountDownLatch latch;
private final int[] urlIndexes;
public FetchTask(int[] urlIndexes, CountDownLatch latch){
this.urlIndexes = urlIndexes;
this.latch = latch;
}
@Override
public void run() {
URLConnection resource;
InputStream is = null;
for(int i = 0; i < urlIndexes.length; i++)
{
int numBytes = 0;
try {
resource = Data.URL_LIST.get(urlIndexes[i]).openConnection();
resource.setConnectTimeout(timeoutMS);
resource.setRequestProperty("User-Agent", "Mozilla/5.0");
is = resource.getInputStream();
while(is.read()!=-1 && numBytes < 65536 )
{
numBytes++;
}
} catch (IOException e) {
System.out.println("Fetch Exception: " + e.getMessage());
} finally {
System.out.println(numBytes + "," + urlIndexes[i] + "," + remaining.decrementAndGet());
if(is!=null){
try {
is.close();
} catch (IOException e1) {/*eat it*/}
}
}
}
latch.countDown();
}
}