Package it.unimi.di.law.bubing.frontier

Class Distributor

java.lang.Object

java.lang.Thread

it.unimi.di.law.bubing.frontier.Distributor

All Implemented Interfaces:

java.lang.Runnable

public final class Distributor
extends java.lang.Thread

A thread that distributes ready URLs (coming out of the sieve) into the Workbench queues with the help of a WorkbenchVirtualizer. We invite the reader to consult the documentation of WorkbenchVirtualizer first.

The behaviour of this distributor is controlled by two conditions:

1. Whether the workbench is full.
2. Whether the front is large enough, that is, whether the number of visit states in the todo list (which are actually thought as being representative of their own IP address) plus Frontier.workbench.size() is below the current required size (i.e., the front is counted in IP addresses).

Then, this distributor iteratively:

• Checks that the workbench is not full and that the front is not large enough. If either condition fails, there is no point in doing I/O.
• If the conditions are fulfilled, the distributor checks the Frontier.refill queue to see whether there are visit states requiring a refill from the WorkbenchVirtualizer, in which case it performs a refill.
• Otherwise, if there are no ready URLs and it is too early to force a flush of the sieve, this thread is put to sleep with an exponential backoff.
• Otherwise, (possibly after a flush) a ready URL is loaded from Frontier.readyURLs and either deleted (if we already have too many URLs for its scheme+authority), or enqueued to the workbench (if its visit state has no virtualized URLs and has not reached VisitState.pathQueryLimit()), or otherwise enqueued to the WorkbenchVirtualizer.

Nested Class Summary

Nested classes/interfaces inherited from class java.lang.Thread

java.lang.Thread.State, java.lang.Thread.UncaughtExceptionHandler

Field Summary

Fields

Modifier and Type	Field	Description
protected long	lastHighCostStat	The last time we produced a high-cost statistics.
protected long	lastPurgeCheck	The last time we checked for visit states to be purged.
protected VisitStateSet	schemeAuthority2VisitState	An unsynchronized map from scheme+authorities to the corresponding VisitState.
protected StatsThread	statsThread	The thread printing statistics.

Fields inherited from class java.lang.Thread

MAX_PRIORITY, MIN_PRIORITY, NORM_PRIORITY

Constructor Summary

Constructors

Constructor	Description
Distributor(Frontier frontier)	Creates a distributor for the given frontier.

Method Summary

Modifier and Type	Method	Description
void	run()

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from class java.lang.Thread

activeCount, checkAccess, clone, countStackFrames, currentThread, destroy, dumpStack, enumerate, getAllStackTraces, getContextClassLoader, getDefaultUncaughtExceptionHandler, getId, getName, getPriority, getStackTrace, getState, getThreadGroup, getUncaughtExceptionHandler, holdsLock, interrupt, interrupted, isAlive, isDaemon, isInterrupted, join, join, join, onSpinWait, resume, setContextClassLoader, setDaemon, setDefaultUncaughtExceptionHandler, setName, setPriority, setUncaughtExceptionHandler, sleep, sleep, start, stop, stop, suspend, toString, yield

Field Detail

schemeAuthority2VisitState

protected final VisitStateSet schemeAuthority2VisitState

An unsynchronized map from scheme+authorities to the corresponding VisitState.

statsThread

protected final StatsThread statsThread

The thread printing statistics.

lastHighCostStat

protected volatile long lastHighCostStat

The last time we produced a high-cost statistics.

lastPurgeCheck

protected volatile long lastPurgeCheck

The last time we checked for visit states to be purged.

Constructor Detail

Distributor

public Distributor(Frontier frontier)

Creates a distributor for the given frontier.

Parameters:

frontier - the frontier instantiating this distribution.

Method Detail

run

public void run()

Specified by:

run in interface java.lang.Runnable

Overrides:

run in class java.lang.Thread