Frontier (BUbiNG 0.9.15)

java.lang.Object
- it.unimi.di.law.bubing.frontier.Frontier

All Implemented Interfaces:

AbstractSieve.NewFlowReceiver<ByteArrayList>, it.unimi.dsi.jai4j.JobListener<BubingJob>
```
public class Frontier
extends java.lang.Object
implements it.unimi.dsi.jai4j.JobListener<BubingJob>, AbstractSieve.NewFlowReceiver<ByteArrayList>
```
The BUbiNG frontier: a class structure that encompasses most of the logic behind the way BUbiNG fetches URLs. To understand how this class works, we invite the reader to consult the documentation of VisitState first.
Every BUbiNG agent contains an instance of this class, which is responsible for starting and orchestrating the mutual interaction of many different elements:
- the workbench (described below), that contains the VisitStates to be visited next;
- the Distributor, that fills the workbench, possibly virtualizing part of it through a WorkbenchVirtualizer;
- the TodoThread and the DoneThread, that move around VisitStates during the crawl;
- a number of worker threads doing useful work, like DNSThread, FetchingThread and ParsingThread.
How the frontier handles URLs

URLs in BUbiNG belong to one of four states:
- unknown: we have never seen the URL as yet;
- waiting (to be filtered): we have seen the URL and we have enqueued it in the sieve, but the URL is waiting to be flushed (i.e., to be poured from the sieve to the readyURLs queue, which happens only the first time for every given URL);
- ready: the URL has come out of the sieve, so it will be visited at some point in the future unless some limiting parameters (e.g., RuntimeConfiguration.maxUrlsPerSchemeAuthority) forbid it;
- visited: the URL became ready and was somehow handled (e.g., fetched and possibly stored).
Note that waiting URLs that happen to be ready or visited will never come out of the sieve: it is the logic of the sieve itself that inhibits the same object to be emitted twice.
All ready URLs are initially stored in a ByteArrayDiskQueue called readyURLs, from which they are moved to the FIFO queue of their VisitState by the Distributor. Inside a VisitState, we only store a byte-array represention of the path+query of ready URLs. Some of them may be stored outside of the visit state, through virtualization. Keeping path+queries in byte-array form in all components reduces enormously object creation, and provides a simple form of compression by prefix omission.
The lifecycle of a URL

URLs are enqueued to the frontier either because they are part of the visit seed, or because a ParsingThread has found them, or because they have been submitted using JAI4J (this includes both manual submission and URLs sent by other agents). The method enqueue(ByteArrayList) will first check whether we have stored already too many URLs for the URL scheme+authority, in which case the URL is discarded; then, it will check whether the URL already appears in the URL cache (also in this case, the URL is discarded); finally it will check whether there is another agent responsible for the URL, and in this case the URL will be sent to the appropriate agent using the JAI4J infrastructure. If all checks have passed, the URL will be put in the sieve.
The URL might never be dequeued from the sieve (because we have seen it already), but if it is, it will become ready by going through the append(long, ByteArrayList) method, which will again check that we do not have too many URLs for the URL authority, and then will enqueue the URL to the disk queue of ready URLs.
Note that we expect that the vast majority of URLs will be of the first kind. This is very important, as there is much less contention on visit state locks than on the frontier lock.

Nested Class Summary

Nested Classes
Modifier and Type Class Description

static class Frontier.PropertyKeys
Names of the scalar fields saved by snap().

Nested Classes
Modifier and Type	Class	Description
`static class`	`Frontier.PropertyKeys`	Names of the scalar fields saved by `snap()`.

Field Summary

Fields
Modifier and Type	Field	Description
`protected Agent`	`agent`	The agent that created this frontier.
`java.util.concurrent.atomic.AtomicLong[]`	`archetypesStatus`	In position `i`, with 0 < `i` <6, the number of pages stored (does not include duplicates) having status `i`xx.
`protected double`	`averageSpeed`	The average speeds of all visit states.
`java.util.concurrent.atomic.AtomicLong`	`brokenVisitStates`	The number of broken visit states.
`static it.unimi.dsi.sux4j.mph.AbstractHashFunction<byte[]>`	`BYTE_ARRAY_HASHING_STRATEGY`	A hash function using `MurmurHash3`.
`static it.unimi.dsi.sux4j.mph.AbstractHashFunction<ByteArrayList>`	`BYTE_ARRAY_LIST_HASHING_STRATEGY`	A hash function using `MurmurHash3`.
`SummaryStats`	`contentLength`	Statistic about the content length of each archetype
`java.util.concurrent.atomic.AtomicLong`	`contentTypeApplication`	Number of archetypes whose indicated content type starts with application (case insensitive)
`java.util.concurrent.atomic.AtomicLong`	`contentTypeImage`	Number of archetypes whose indicated content type starts with image (case insensitive)
`java.util.concurrent.atomic.AtomicLong`	`contentTypeOthers`	Number of archetypes whose indicated content type does not start with text, image, or application (case insensitive)
`java.util.concurrent.atomic.AtomicLong`	`contentTypeText`	Number of archetypes whose indicated content type starts with text (case insensitive)
`org.apache.http.client.config.RequestConfig`	`defaultRequestConfig`	The default configuration for a non-`robots.txt` request.
`BloomFilter<java.lang.Void>`	`digests`	A Bloom filter storing page digests for duplicate detection.
`protected Distributor`	`distributor`	The thread constantly moving ready URLs into the workbench.
`protected ObjectArrayList<DNSThread>`	`dnsThreads`	The threads resolving DNS for new visit states.
`LockFreeQueue<VisitState>`	`done`	A lock-free list of visit states ready to be released; it is filled by fetching threads and emptied by the `DoneThread`.
`java.util.concurrent.atomic.AtomicLong`	`duplicates`	The number of duplicate pages.
`SummaryStats`	`externalOutdegree`	Statistics about the number of out-links of each archetype, without considering the links to the same corresponding host
`java.util.concurrent.atomic.AtomicLong`	`fetchedResources`	The number of fetched resources (updated by `ParsingThread` instances).
`java.util.concurrent.atomic.AtomicLong`	`fetchedRobots`	The number of fetched `robots.txt` files (updated by `ParsingThread` instances).
`java.util.concurrent.atomic.AtomicLong`	`fetchingThreadWaitingTimeSum`	The sum of the waiting time of the waiting fetching threads: every time a fetching thread waits this sum is updated; every time the statistics are printed this value is reset.
`java.util.concurrent.atomic.AtomicLong`	`fetchingThreadWaits`	The number of waits performed by fetching threads; every time the statistics are printed this value is reset.
`static long`	`FRONT_INCREASE`	The increase of the front size used by `updateRequestedFrontSize()`.
`static java.net.InetAddress[]`	`LOOPBACK`	The loopback address, cached.
`static long`	`MIN_FLUSH_INTERVAL`	The minimum number of milliseconds between two flushes.
`java.util.concurrent.LinkedBlockingQueue<VisitState>`	`newVisitStates`	The queue of new visit states; filled by the distributor and emptied by the DNS threads.
`protected long`	`nextFlush`	The time at which the next flush can happen.
`java.util.concurrent.atomic.AtomicLong`	`numberOfReceivedURLs`	The overall number of URLs sent by other agents.
`SummaryStats`	`outdegree`	Statistics about the number of out-links of each archetype
`ObjectArrayList<ParsingThread>`	`parsingThreads`	The parsing threads.
`java.util.concurrent.atomic.AtomicLong`	`pathQueriesInQueues`	The overall number of path+queries stored in `VisitState` queues.
`java.util.concurrent.ArrayBlockingQueue<ByteArrayList>`	`quickReceivedURLs`	A queue to quickly buffer URLs communicated by `receive(BubingJob)`.
`RuntimeConfiguration`	`rc`	The runtime configuration.
`static int`	`READY_URLS_BUFFER_SIZE`	The size of the buffer used for `readyURLs`.
`ByteArrayDiskQueue`	`readyURLs`	A queue to store URLs coming out of the `sieve`.
`ByteArrayDiskQueue`	`receivedURLs`	A queue to buffer in the long run URLs communicated by `receive(BubingJob)`.
`protected LockFreeQueue<VisitState>`	`refill`	A queue of visit states ready to be reilled; it is filled by fetching threads and emptied by the `Distributor`.
`java.util.concurrent.atomic.AtomicLong`	`requiredFrontSize`	The current estimation for the size of the front in IP addresses.
`LockFreeQueue<FetchData>`	`results`	A lock-free list of `FetchData` to be parsed; it is filled by fetching threads and emptied by the parsing threads.
`org.apache.http.client.config.RequestConfig`	`robotsRequestConfig`	The default configuration for a `robots.txt` request.
`ParallelBufferedWarcWriter`	`robotsWarcParallelOutputStream`	The Warc file where to write (if so requested) the downloaded `robots.txt` files.
`ConcurrentCountingMap`	`schemeAuthority2Count`	A synchronized, highly concurrent map from scheme+authorities to number of stored URLs.
`AbstractSieve<ByteArrayList,java.lang.Void>`	`sieve`	An instance of a `MercatorSieve`.
`java.util.concurrent.atomic.AtomicLongArray`	`speedDist`	The logarithmically binned statistics of download speed in bits/s.
`protected Store`	`store`	The store.
`LockFreeQueue<VisitState>`	`todo`	A lock-free list of visit states ready to be visited; it is filled by the `TodoThread` and emptied by the fetching threads.
`java.util.concurrent.atomic.AtomicLong`	`transferredBytes`	The overall number of transferred bytes.
`java.util.concurrent.DelayQueue<VisitState>`	`unknownHosts`	The queue of unknown hosts.
`FastApproximateByteArrayCache`	`urlCache`	The URL cache.
`protected WorkbenchVirtualizer`	`virtualizer`	The workbench virtualizer used by this frontier.
`java.util.concurrent.atomic.AtomicLong`	`weightOfpathQueriesInQueues`	The overall memory (in bytes) used by path+queries stored in `VisitState` queues.
`Workbench`	`workbench`	The workbench.
`long`	`workbenchSizeInPathQueries`	An estimation of the number of path+query objects that the workbench can store.

Constructor Summary

Constructors
Constructor Description

Frontier(RuntimeConfiguration rc, Store store, Agent agent)
Creates the frontier.

Constructors
Constructor	Description
`Frontier(RuntimeConfiguration rc, Store store, Agent agent)`	Creates the frontier.

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method	Description
`void`	`append(long hash, ByteArrayList list)`	A new key is appended.
`long`	`archetypes()`	The number of pages stored (does not include duplicates).
`void`	`close()`	Closes the frontier: threads are stopped (if necessary, aborted), sieve and store and robots stream are closed.
`void`	`dnsThreads(int newDnsThreads)`	Changes the number of DNS threads.
`void`	`enqueue(ByteArrayList url)`	Enqueues a URL to the BUbiNG crawl.
`void`	`enqueueLocal(ByteArrayList url)`	Enqueues a local URL represented by a byte array to the crawl of this agent.
`void`	`fetchingThreads(int numFetchingThreads)`	Changes the number of fetching threads.
`void`	`finishedAppending()`	The new flow of keys is over.
`StatsThread`	`getStatsThread()`	Returns the `StatsThread`.
`void`	`noMoreAppend()`	There will be no more new flows (because the sieve that is calling this method was closed).
`void`	`parsingThreads(int newParsingThreads)`	Changes the number of parsing threads.
`void`	`prepareToAppend()`	A new flow of keys is ready and will start being appended.
`void`	`receive(BubingJob job)`
`void`	`resetFetchingThreadsWaitingStats()`	Resets the statistics relative to the wait time of `FetchingThread`s.
`void`	`restore()`	Restores data from the given directory.
`void`	`snap()`	Snaps fields to files in the given directory.
`void`	`updateFetchingThreadsWaitingStats(long waitTime)`	Updates the statistics relative to the wait time of `FetchingThread`s.
`void`	`updateRequestedFrontSize()`	Update, if necessary, the `requiredFrontSize`.
`boolean`	`workbenchIsFull()`	Returns whether the workbench is full.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - READY_URLS_BUFFER_SIZE
```
public static final int READY_URLS_BUFFER_SIZE
```
    The size of the buffer used for readyURLs.
    
    See Also:
    
    Constant Field Values
  - LOOPBACK
```
public static final java.net.InetAddress[] LOOPBACK
```
    The loopback address, cached.
  - MIN_FLUSH_INTERVAL
```
public static final long MIN_FLUSH_INTERVAL
```
    The minimum number of milliseconds between two flushes.
    
    See Also:
    
    Constant Field Values
  - FRONT_INCREASE
```
public static final long FRONT_INCREASE
```
    The increase of the front size used by updateRequestedFrontSize().
    
    See Also:
    
    Constant Field Values
  - BYTE_ARRAY_HASHING_STRATEGY
```
public static final it.unimi.dsi.sux4j.mph.AbstractHashFunction<byte[]> BYTE_ARRAY_HASHING_STRATEGY
```
    A hash function using MurmurHash3.
  - BYTE_ARRAY_LIST_HASHING_STRATEGY
```
public static final it.unimi.dsi.sux4j.mph.AbstractHashFunction<ByteArrayList> BYTE_ARRAY_LIST_HASHING_STRATEGY
```
    A hash function using MurmurHash3.
  - store
```
protected final Store store
```
    The store.
  - agent
```
protected final Agent agent
```
    The agent that created this frontier.
  - rc
```
public final RuntimeConfiguration rc
```
    The runtime configuration.
  - sieve
```
public AbstractSieve<ByteArrayList,java.lang.Void> sieve
```
    An instance of a MercatorSieve.
  - readyURLs
```
public ByteArrayDiskQueue readyURLs
```
    A queue to store URLs coming out of the sieve.
  - quickReceivedURLs
```
public java.util.concurrent.ArrayBlockingQueue<ByteArrayList> quickReceivedURLs
```
    A queue to quickly buffer URLs communicated by receive(BubingJob).
  - receivedURLs
```
public ByteArrayDiskQueue receivedURLs
```
    A queue to buffer in the long run URLs communicated by receive(BubingJob).
  - parsingThreads
```
public final ObjectArrayList<ParsingThread> parsingThreads
```
    The parsing threads.
  - robotsWarcParallelOutputStream
```
public final ParallelBufferedWarcWriter robotsWarcParallelOutputStream
```
    The Warc file where to write (if so requested) the downloaded robots.txt files.
  - pathQueriesInQueues
```
public final java.util.concurrent.atomic.AtomicLong pathQueriesInQueues
```
    The overall number of path+queries stored in VisitState queues.
  - weightOfpathQueriesInQueues
```
public final java.util.concurrent.atomic.AtomicLong weightOfpathQueriesInQueues
```
    The overall memory (in bytes) used by path+queries stored in VisitState queues.
  - brokenVisitStates
```
public final java.util.concurrent.atomic.AtomicLong brokenVisitStates
```
    The number of broken visit states.
  - numberOfReceivedURLs
```
public final java.util.concurrent.atomic.AtomicLong numberOfReceivedURLs
```
    The overall number of URLs sent by other agents.
  - nextFlush
```
protected volatile long nextFlush
```
    The time at which the next flush can happen.
  - workbench
```
public final Workbench workbench
```
    The workbench.
  - unknownHosts
```
public final java.util.concurrent.DelayQueue<VisitState> unknownHosts
```
    The queue of unknown hosts.
  - newVisitStates
```
public final java.util.concurrent.LinkedBlockingQueue<VisitState> newVisitStates
```
    The queue of new visit states; filled by the distributor and emptied by the DNS threads.
  - digests
```
public BloomFilter<java.lang.Void> digests
```
    A Bloom filter storing page digests for duplicate detection.
  - dnsThreads
```
protected final ObjectArrayList<DNSThread> dnsThreads
```
    The threads resolving DNS for new visit states.
  - distributor
```
protected final Distributor distributor
```
    The thread constantly moving ready URLs into the workbench.
  - urlCache
```
public final FastApproximateByteArrayCache urlCache
```
    The URL cache. This cache stores the most recent URLs that have been enqueued.
  - virtualizer
```
protected final WorkbenchVirtualizer virtualizer
```
    The workbench virtualizer used by this frontier.
  - todo
```
public final LockFreeQueue<VisitState> todo
```
    A lock-free list of visit states ready to be visited; it is filled by the TodoThread and emptied by the fetching threads.
  - done
```
public final LockFreeQueue<VisitState> done
```
    A lock-free list of visit states ready to be released; it is filled by fetching threads and emptied by the DoneThread.
  - refill
```
protected final LockFreeQueue<VisitState> refill
```
    A queue of visit states ready to be reilled; it is filled by fetching threads and emptied by the Distributor.
  - requiredFrontSize
```
public final java.util.concurrent.atomic.AtomicLong requiredFrontSize
```
    The current estimation for the size of the front in IP addresses. It is adaptively increased when a FetchingThread has to wait to retrieve a VisitState from the todo queue. It is never more than half the number of path+queries that the workbench can hold.
  - fetchingThreadWaits
```
public final java.util.concurrent.atomic.AtomicLong fetchingThreadWaits
```
    The number of waits performed by fetching threads; every time the statistics are printed this value is reset.
  - fetchingThreadWaitingTimeSum
```
public final java.util.concurrent.atomic.AtomicLong fetchingThreadWaitingTimeSum
```
    The sum of the waiting time of the waiting fetching threads: every time a fetching thread waits this sum is updated; every time the statistics are printed this value is reset.
  - results
```
public final LockFreeQueue<FetchData> results
```
    A lock-free list of FetchData to be parsed; it is filled by fetching threads and emptied by the parsing threads.
  - archetypesStatus
```
public final java.util.concurrent.atomic.AtomicLong[] archetypesStatus
```
    In position i, with 0 < i <6, the number of pages stored (does not include duplicates) having status ixx. In position 0, the number of pages with other status.
  - outdegree
```
public final SummaryStats outdegree
```
    Statistics about the number of out-links of each archetype
  - externalOutdegree
```
public final SummaryStats externalOutdegree
```
    Statistics about the number of out-links of each archetype, without considering the links to the same corresponding host
  - contentLength
```
public final SummaryStats contentLength
```
    Statistic about the content length of each archetype
  - contentTypeText
```
public final java.util.concurrent.atomic.AtomicLong contentTypeText
```
    Number of archetypes whose indicated content type starts with text (case insensitive)
  - contentTypeImage
```
public final java.util.concurrent.atomic.AtomicLong contentTypeImage
```
    Number of archetypes whose indicated content type starts with image (case insensitive)
  - contentTypeApplication
```
public final java.util.concurrent.atomic.AtomicLong contentTypeApplication
```
    Number of archetypes whose indicated content type starts with application (case insensitive)
  - contentTypeOthers
```
public final java.util.concurrent.atomic.AtomicLong contentTypeOthers
```
    Number of archetypes whose indicated content type does not start with text, image, or application (case insensitive)
  - duplicates
```
public final java.util.concurrent.atomic.AtomicLong duplicates
```
    The number of duplicate pages.
  - fetchedResources
```
public final java.util.concurrent.atomic.AtomicLong fetchedResources
```
    The number of fetched resources (updated by ParsingThread instances).
  - fetchedRobots
```
public final java.util.concurrent.atomic.AtomicLong fetchedRobots
```
    The number of fetched robots.txt files (updated by ParsingThread instances).
  - transferredBytes
```
public final java.util.concurrent.atomic.AtomicLong transferredBytes
```
    The overall number of transferred bytes.
  - schemeAuthority2Count
```
public ConcurrentCountingMap schemeAuthority2Count
```
    A synchronized, highly concurrent map from scheme+authorities to number of stored URLs.
  - speedDist
```
public final java.util.concurrent.atomic.AtomicLongArray speedDist
```
    The logarithmically binned statistics of download speed in bits/s.
  - averageSpeed
```
protected double averageSpeed
```
    The average speeds of all visit states.
  - workbenchSizeInPathQueries
```
public volatile long workbenchSizeInPathQueries
```
    An estimation of the number of path+query objects that the workbench can store.
  - defaultRequestConfig
```
public final org.apache.http.client.config.RequestConfig defaultRequestConfig
```
    The default configuration for a non-robots.txt request.
  - robotsRequestConfig
```
public final org.apache.http.client.config.RequestConfig robotsRequestConfig
```
    The default configuration for a robots.txt request.
- Constructor Detail
  - Frontier
```
public Frontier(RuntimeConfiguration rc,
                Store store,
                Agent agent)
         throws java.io.IOException,
                java.lang.IllegalArgumentException,
                ConfigurationException,
                java.lang.ClassNotFoundException,
                java.lang.InterruptedException
```
    Creates the frontier.
    
    Parameters:
    
    rc - the configuration to be used to set all parameters.
    
    store - the place where fetched pages will be stored.
    
    agent - the BUbiNG agent possessing this frontier.
    
    Throws:
    
    java.io.IOException
    
    java.lang.IllegalArgumentException
    
    ConfigurationException
    
    java.lang.ClassNotFoundException
    
    java.lang.InterruptedException
- Method Detail
  - dnsThreads
```
public void dnsThreads(int newDnsThreads)
                throws java.lang.IllegalArgumentException
```
    Changes the number of DNS threads.
    Note that when the number of thread is reduced, the stopped thread will actually terminate their execution as soon as they check the DNSThread.stop field.
    
    Parameters:
    
    newDnsThreads - the new number of threads.
    
    Throws:
    
    java.lang.IllegalArgumentException
  - fetchingThreads
```
public void fetchingThreads(int numFetchingThreads)
                     throws java.lang.IllegalArgumentException,
                            java.security.NoSuchAlgorithmException,
                            java.io.IOException
```
    Changes the number of fetching threads.
    Note that when the number of thread is reduced, the stopped thread will actually terminate their execution as soon as they check the ParsingThread.stop field.
    
    Parameters:
    
    numFetchingThreads - the new number of threads.
    
    Throws:
    
    java.lang.IllegalArgumentException
    
    java.security.NoSuchAlgorithmException
    
    java.io.IOException
  - parsingThreads
```
public void parsingThreads(int newParsingThreads)
                    throws java.lang.IllegalArgumentException
```
    Changes the number of parsing threads.
    Note that when the number of thread is reduced, the stopped thread will actually terminate their execution as soon as they check the ParsingThread.stop field.
    
    Parameters:
    
    newParsingThreads - the new number of threads.
    
    Throws:
    
    java.lang.IllegalArgumentException
  - enqueue
```
public void enqueue(ByteArrayList url)
             throws java.io.IOException,
                    java.lang.InterruptedException
```
    Enqueues a URL to the BUbiNG crawl.
    Before enqueueing the URL to the sieve we perform a number of checks:
    - if there are too many URLs for the URL scheme+authority, we discard the URL;
    - if the URL appears in the URL cache, we discard it; otherwise, we add it to the cache;
    - if another agent is responsible for the URL, we submit it to the agent.
    The difference between this method and enqueueLocal(ByteArrayList) is that the latter does not check whether the argument is local.
    Parameters:
    
    url - a BUbiNG URL to be enqueued to the BUbiNG crawl.
    
    Throws:
    
    java.lang.InterruptedException - from AbstractSieve.enqueue(Object, Object).
    
    java.io.IOException
  - workbenchIsFull
```
public boolean workbenchIsFull()
```
    Returns whether the workbench is full.
    
    Returns:
    
    whether the workbench is full.
  - enqueueLocal
```
public void enqueueLocal(ByteArrayList url)
                  throws java.io.IOException,
                         java.lang.InterruptedException
```
    Enqueues a local URL represented by a byte array to the crawl of this agent.
    Before enqueueing the URL to the sieve we perform a number of checks:
    - if there are too many URLs for the URL scheme+authority, we discard the URL;
    - if the URL appears in the URL cache, we discard it; otherwise, we add it to the cache.
    The difference between this method and enqueue(ByteArrayList) is that the latter checks whether the argument is local.
    Parameters:
    
    url - a BUbiNG URL to be enqueued to the BUbiNG crawl, in byte-array representation.
    
    Throws:
    
    java.lang.InterruptedException - from AbstractSieve.enqueue(Object, Object).
    
    java.io.IOException
  - receive
```
public void receive(BubingJob job)
```
    Specified by:
    
    receive in interface it.unimi.dsi.jai4j.JobListener<BubingJob>
  - close
```
public void close()
           throws java.io.IOException,
                  java.lang.InterruptedException
```
    Closes the frontier: threads are stopped (if necessary, aborted), sieve and store and robots stream are closed.
    
    Throws:
    
    java.io.IOException
    
    java.lang.InterruptedException
  - prepareToAppend
```
public void prepareToAppend()
                     throws java.io.IOException
```
    Description copied from interface: AbstractSieve.NewFlowReceiver
    
    A new flow of keys is ready and will start being appended.
    
    Specified by:
    
    prepareToAppend in interface AbstractSieve.NewFlowReceiver<ByteArrayList>
    
    Throws:
    
    java.io.IOException
  - append
```
public void append(long hash,
                   ByteArrayList list)
            throws java.io.IOException
```
    Description copied from interface: AbstractSieve.NewFlowReceiver
    
    A new key is appended.
    
    Specified by:
    
    append in interface AbstractSieve.NewFlowReceiver<ByteArrayList>
    
    Parameters:
    
    hash - the key hash.
    
    list - the key itself.
    
    Throws:
    
    java.io.IOException
  - finishedAppending
```
public void finishedAppending()
                       throws java.io.IOException
```
    Description copied from interface: AbstractSieve.NewFlowReceiver
    
    The new flow of keys is over.
    
    Specified by:
    
    finishedAppending in interface AbstractSieve.NewFlowReceiver<ByteArrayList>
    
    Throws:
    
    java.io.IOException
  - noMoreAppend
```
public void noMoreAppend()
                  throws java.io.IOException
```
    Description copied from interface: AbstractSieve.NewFlowReceiver
    
    There will be no more new flows (because the sieve that is calling this method was closed).
    
    Specified by:
    
    noMoreAppend in interface AbstractSieve.NewFlowReceiver<ByteArrayList>
    
    Throws:
    
    java.io.IOException
  - updateRequestedFrontSize
```
public void updateRequestedFrontSize()
```
    Update, if necessary, the requiredFrontSize. The current front size is the number of visit states present in the workbench and in the todo queue. If this quantity is larged than the currently-required front size, the latter is increase by FRONT_INCREASE, although it will never be set to a value larger than half of the workbench (two queries per visit state).
  - updateFetchingThreadsWaitingStats
```
public void updateFetchingThreadsWaitingStats(long waitTime)
```
    Updates the statistics relative to the wait time of FetchingThreads.
    
    Parameters:
    
    waitTime - the length of a new pause by one of the fetching threads.
  - resetFetchingThreadsWaitingStats
```
public void resetFetchingThreadsWaitingStats()
```
    Resets the statistics relative to the wait time of FetchingThreads.
  - snap
```
public void snap()
          throws ConfigurationException,
                 java.lang.IllegalArgumentException,
                 java.io.IOException
```
    Snaps fields to files in the given directory. Fields that are of scalar are written into a single file named frontier.data. Other fields are written each in a file of its own, named with the name of the field.
    
    Throws:
    
    ConfigurationException
    
    java.lang.IllegalArgumentException
    
    java.io.IOException
  - restore
```
public void restore()
             throws ConfigurationException,
                    java.lang.IllegalArgumentException,
                    java.io.IOException,
                    java.lang.ClassNotFoundException,
                    java.lang.InterruptedException
```
    Restores data from the given directory.
    
    Throws:
    
    java.lang.InterruptedException
    
    ConfigurationException
    
    java.lang.IllegalArgumentException
    
    java.io.IOException
    
    java.lang.ClassNotFoundException
    
    See Also:
    
    snap()
  - getStatsThread
```
public StatsThread getStatsThread()
```
    Returns the StatsThread.
    
    Returns:
    
    the stats thread.
  - archetypes
```
public long archetypes()
```
    The number of pages stored (does not include duplicates).
    
    Returns:
    
    the number of pages stored.

Class Frontier

How the frontier handles URLs

The lifecycle of a URL

Nested Class Summary

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

READY_URLS_BUFFER_SIZE

LOOPBACK

MIN_FLUSH_INTERVAL

FRONT_INCREASE

BYTE_ARRAY_HASHING_STRATEGY

BYTE_ARRAY_LIST_HASHING_STRATEGY

store

agent

rc

sieve

readyURLs

quickReceivedURLs

receivedURLs

parsingThreads

robotsWarcParallelOutputStream

pathQueriesInQueues

weightOfpathQueriesInQueues

brokenVisitStates

numberOfReceivedURLs

nextFlush

workbench

unknownHosts

newVisitStates

digests

dnsThreads

distributor

urlCache

virtualizer

todo

done

refill

requiredFrontSize

fetchingThreadWaits

fetchingThreadWaitingTimeSum

results

archetypesStatus

outdegree

externalOutdegree

contentLength

contentTypeText

contentTypeImage

contentTypeApplication

contentTypeOthers

duplicates

fetchedResources

fetchedRobots

transferredBytes

schemeAuthority2Count

speedDist

averageSpeed

workbenchSizeInPathQueries

defaultRequestConfig

robotsRequestConfig

Constructor Detail

Frontier

Method Detail

dnsThreads

fetchingThreads

parsingThreads

enqueue

workbenchIsFull

enqueueLocal

receive

close

prepareToAppend

append

finishedAppending

noMoreAppend

updateRequestedFrontSize

updateFetchingThreadsWaitingStats

resetFetchingThreadsWaitingStats