Package it.unimi.di.law.bubing.frontier

Class FetchingThread

java.lang.Object

java.lang.Thread

it.unimi.di.law.bubing.frontier.FetchingThread

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable, java.lang.Runnable

public final class FetchingThread
extends java.lang.Thread
implements java.io.Closeable

A thread fetching pages that will be then analyzed by a ParsingThread.

Instances of this class iteratively extract from Frontier.todo (using polling and exponential backoff) a ready VisitState that has been previously enqueued by the TodoThread and use their embedded FetchData to fetch the first URL (or possibly the first few URLs, depending on RuntimeConfiguration.keepAliveTime) from the VisitState queue. Once the fetch is over, the embedded FetchData is enqueued in Frontier.results and the thread waits for a signal on the FetchData. The ParsingThread that retrieves the FetchData must signal back that the FetchData instances can be reused when it has finished to use its contents.

The design of the interaction between instances of this class, the TodoThread and instances of ParsingThread minimizes contention by sandwiching all FetchingThread instances between two wait-free queues (signalling back that a FetchData can be reused causes of course no contention). Exponential backoff should happen rarely in a full-speed crawl, as the todo queue is almost always nonempty.

Instances of this class do not access any shared data structure, except for logging. It is expected that large instances of BUbiNG use thousands of fetching threads to download from a large number of sites in parallel.

This class implements Closeable: the close() methods simply closes the underlying FetchData instance.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
protected static class	FetchingThread.BasicHttpClientConnectionManagerWithAlternateDNS	A support class that makes it possible to plug in a custom DNS resolver.

Nested classes/interfaces inherited from class java.lang.Thread

java.lang.Thread.State, java.lang.Thread.UncaughtExceptionHandler

Field Summary

Fields

Modifier and Type	Field	Description
boolean	stop	Whether we should stop (used also to reduce the number of threads).

Fields inherited from class java.lang.Thread

MAX_PRIORITY, MIN_PRIORITY, NORM_PRIORITY

Constructor Summary

Constructors

Constructor	Description
FetchingThread(Frontier frontier, int index)	Creates a new fetching thread.

Method Summary

Modifier and Type	Method	Description
void	abort()	Causes the FetchData used by this thread to be FetchData.abort() (whence, the corresponding connection to be closed).
void	close()
static org.apache.http.cookie.Cookie[]	getCookies(java.net.URI url, org.apache.http.client.CookieStore cookieStore, int cookieMaxByteSize)	Returns the list of cookies in a given store in the form of an array, limiting their overall size (only the maximal prefix of cookies satisfying the size limit is returned).
void	run()

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from class java.lang.Thread

activeCount, checkAccess, clone, countStackFrames, currentThread, destroy, dumpStack, enumerate, getAllStackTraces, getContextClassLoader, getDefaultUncaughtExceptionHandler, getId, getName, getPriority, getStackTrace, getState, getThreadGroup, getUncaughtExceptionHandler, holdsLock, interrupt, interrupted, isAlive, isDaemon, isInterrupted, join, join, join, onSpinWait, resume, setContextClassLoader, setDaemon, setDefaultUncaughtExceptionHandler, setName, setPriority, setUncaughtExceptionHandler, sleep, sleep, start, stop, stop, suspend, toString, yield

Field Detail

stop

public volatile boolean stop

Whether we should stop (used also to reduce the number of threads).

Constructor Detail

FetchingThread

public FetchingThread(Frontier frontier,
                      int index)
               throws java.security.NoSuchAlgorithmException,
                      java.lang.IllegalArgumentException,
                      java.io.IOException

Creates a new fetching thread.

Parameters:

frontier - a reference to the Frontier.

index - the index of this thread (only for logging purposes).

Throws:

java.security.NoSuchAlgorithmException

java.lang.IllegalArgumentException

java.io.IOException

Method Detail

getCookies

public static org.apache.http.cookie.Cookie[] getCookies(java.net.URI url,
                                                         org.apache.http.client.CookieStore cookieStore,
                                                         int cookieMaxByteSize)

Returns the list of cookies in a given store in the form of an array, limiting their overall size (only the maximal prefix of cookies satisfying the size limit is returned). Note that the size limit is expressed in bytes, but the actual memory footprint is larger because of object overheads and because strings are made of 16-bits characters. Moreover, cookie attributes are not measured when evaluating size.

Parameters:

url - the URL which generated the cookies (for logging purposes).

cookieStore - a cookie store, usually generated by a response.

cookieMaxByteSize - the maximum overall size of cookies in bytes.

Returns:

the list of cookies in a given store in the form of an array, with limited overall size.

abort

public void abort()

Causes the FetchData used by this thread to be FetchData.abort() (whence, the corresponding connection to be closed).

run

public void run()

Specified by:

run in interface java.lang.Runnable

Overrides:

run in class java.lang.Thread

close

public void close()
           throws java.io.IOException

Specified by:

close in interface java.lang.AutoCloseable

Specified by:

close in interface java.io.Closeable

Throws:

java.io.IOException