RuntimeConfiguration (BUbiNG 0.9.15)

java.lang.Object
- it.unimi.di.law.bubing.RuntimeConfiguration

```
public class RuntimeConfiguration
extends java.lang.Object
```
Global data shared by all threads.
All BUbiNG components must share a certain number of global variables, such as filters and pool of objects. A single instance of this class is created at agent construction time: it is used to pass around a single reference to global data.
All fields in this class are either final or volatile, depending on whether they can be modified at runtime (usually by means of JMX methods in Agent).

Field Summary

Fields
Modifier and Type	Field	Description
`boolean`	`acceptAllCertificates`
`IntOpenHashSet`	`blackListedHostHashes`	The set of hashes of hosts that should be blacklisted.
`java.util.concurrent.locks.ReadWriteLock`	`blackListedHostHashesLock`	A lock used to access `blackListedHostHashes`.
`IntOpenHashSet`	`blackListedIPv4Addresses`
`java.util.concurrent.locks.ReadWriteLock`	`blackListedIPv4Lock`	A lock used to access `blackListedIPv4Addresses`.
`double`	`bloomFilterPrecision`
`int`	`connectionTimeout`
`int`	`cookieMaxByteSize`
`java.lang.String`	`cookiePolicy`
`boolean`	`crawlIsNew`
`java.lang.String`	`digestAlgorithm`
`int`	`dnsCacheMaxSize`
`long`	`dnsNegativeTtl`
`long`	`dnsPositiveTtl`
`org.apache.http.conn.DnsResolver`	`dnsResolver`	The DNS resolver used throughout the crawler.
`int`	`dnsThreads`
`static java.util.regex.Pattern`	`DOTTED_ADDRESS`	A pattern used to identify hosts specified directed via their address in dotted notation.
`static boolean`	`FETCH_ROBOTS`	Whether to fetch and use `robots.txt`.
`int`	`fetchDataBufferByteSize`
`Filter<java.net.URI>`	`fetchFilter`
`int`	`fetchingThreads`
`Filter<URIResponse>`	`followFilter`
`java.io.File`	`frontierDir`
`java.lang.String`	`group`
`long`	`ipDelay`
`double`	`ipDelayFactor`
`long`	`keepAliveTime`
`long`	`maxUrls`
`int`	`maxUrlsPerSchemeAuthority`
`java.lang.String`	`name`
`Filter<URIResponse>`	`parseFilter`
`java.util.ArrayList<Parser<?>>`	`parsers`	The parser, instantiated.
`int`	`parsingThreads`
`boolean`	`paused`	Whether the crawler is currently paused.
`java.lang.String`	`proxyHost`
`int`	`proxyPort`
`int`	`responseBodyMaxByteSize`
`java.io.File`	`responseCacheDir`
`long`	`robotsExpiration`
`java.io.File`	`rootDir`
`Filter<Link>`	`scheduleFilter`
`long`	`schemeAuthorityDelay`
`java.util.Iterator<java.net.URI>`	`seed`	An iterator returning URIs that are then used as a seed; this iterator may return `null` (when invalid or relative URLs are specified).
`int`	`sieveAuxFileIOBufferByteSize`
`java.io.File`	`sieveDir`
`int`	`sieveSize`
`int`	`sieveStoreIOBufferByteSize`
`int`	`socketTimeout`
`int`	`spamDetectionPeriodicity`
`int`	`spamDetectionThreshold`
`SpamDetector<?>`	`spamDetector`
`boolean`	`startPaused`
`boolean`	`stopping`	Whether the crawler is currently being stopping.
`java.lang.Class<? extends Store>`	`storeClass`
`java.io.File`	`storeDir`
`Filter<URIResponse>`	`storeFilter`
`long`	`urlCacheMaxByteSize`
`java.lang.String`	`userAgent`
`java.lang.String`	`userAgentFrom`
`long`	`virtualizerMaxByteSize`
`int`	`weight`
`long`	`workbenchMaxByteSize`

Constructor Summary

Constructors
Constructor Description

RuntimeConfiguration(StartupConfiguration startupConfiguration)

Constructors
Constructor	Description
`RuntimeConfiguration(StartupConfiguration startupConfiguration)`

Method Summary

All Methods Static Methods Instance Methods Concrete Methods
Modifier and Type	Method	Description
`void`	`addBlackListedHost(java.lang.String spec)`	Adds a (or a set of) new host to the black list; the host can be specified directly or it can be a file (prefixed by `file:`).
`void`	`addBlackListedIPv4(java.lang.String spec)`	Adds a (or a set of) new IPv4 to the black list; the IPv4 can be specified directly or it can be a file (prefixed by `file:`).
`void`	`ensureNotPaused()`
`static java.util.ArrayList<Parser<?>>`	`parsersFromSpecs(java.lang.String[] specs)`	Given an array of parser specifications, it returns the corresponding list of parsers (only the correct specifications are put in the list.
`java.lang.String`	`toString()`

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

- Field Detail
  - FETCH_ROBOTS
```
public static final boolean FETCH_ROBOTS
```
    Whether to fetch and use robots.txt. This value cannot be configured and it requires recompilation from the sources.
    You should be better knowing what you are doing if you change this to false.
    
    See Also:
    
    Constant Field Values
  - name
```
public final java.lang.String name
```
    See Also:
    
    StartupConfiguration.name
  - group
```
public final java.lang.String group
```
    See Also:
    
    StartupConfiguration.group
  - weight
```
public final int weight
```
    See Also:
    
    StartupConfiguration.weight
  - maxUrlsPerSchemeAuthority
```
public final int maxUrlsPerSchemeAuthority
```
    See Also:
    
    StartupConfiguration.maxUrlsPerSchemeAuthority
  - fetchingThreads
```
public volatile int fetchingThreads
```
    See Also:
    
    StartupConfiguration.fetchingThreads
  - parsingThreads
```
public volatile int parsingThreads
```
    See Also:
    
    StartupConfiguration.parsingThreads
  - dnsThreads
```
public volatile int dnsThreads
```
    See Also:
    
    StartupConfiguration.dnsThreads
  - fetchFilter
```
public volatile Filter<java.net.URI> fetchFilter
```
    See Also:
    
    StartupConfiguration.fetchFilter
  - scheduleFilter
```
public volatile Filter<Link> scheduleFilter
```
    See Also:
    
    StartupConfiguration.scheduleFilter
  - parseFilter
```
public volatile Filter<URIResponse> parseFilter
```
    See Also:
    
    StartupConfiguration.parseFilter
  - followFilter
```
public volatile Filter<URIResponse> followFilter
```
    See Also:
    
    StartupConfiguration.followFilter
  - storeFilter
```
public volatile Filter<URIResponse> storeFilter
```
    See Also:
    
    StartupConfiguration.storeFilter
  - keepAliveTime
```
public volatile long keepAliveTime
```
    See Also:
    
    StartupConfiguration.keepAliveTime
  - schemeAuthorityDelay
```
public volatile long schemeAuthorityDelay
```
    See Also:
    
    StartupConfiguration.schemeAuthorityDelay
  - ipDelay
```
public volatile long ipDelay
```
    See Also:
    
    StartupConfiguration.ipDelay
  - ipDelayFactor
```
public volatile double ipDelayFactor
```
    See Also:
    
    StartupConfiguration.ipDelayFactor
  - maxUrls
```
public volatile long maxUrls
```
    See Also:
    
    StartupConfiguration.maxUrls
  - bloomFilterPrecision
```
public final double bloomFilterPrecision
```
    See Also:
    
    StartupConfiguration.bloomFilterPrecision
  - seed
```
public final java.util.Iterator<java.net.URI> seed
```
    An iterator returning URIs that are then used as a seed; this iterator may return null (when invalid or relative URLs are specified).
    
    See Also:
    
    StartupConfiguration.seed
  - blackListedIPv4Addresses
```
public final IntOpenHashSet blackListedIPv4Addresses
```
    See Also:
    
    StartupConfiguration.seed
  - blackListedIPv4Lock
```
public final java.util.concurrent.locks.ReadWriteLock blackListedIPv4Lock
```
    A lock used to access blackListedIPv4Addresses.
  - blackListedHostHashes
```
public final IntOpenHashSet blackListedHostHashes
```
    The set of hashes of hosts that should be blacklisted.
    
    See Also:
    
    StartupConfiguration.blackListedHosts
  - blackListedHostHashesLock
```
public final java.util.concurrent.locks.ReadWriteLock blackListedHostHashesLock
```
    A lock used to access blackListedHostHashes.
  - socketTimeout
```
public volatile int socketTimeout
```
    See Also:
    
    StartupConfiguration.socketTimeout
  - connectionTimeout
```
public volatile int connectionTimeout
```
    See Also:
    
    StartupConfiguration.connectionTimeout
  - fetchDataBufferByteSize
```
public final int fetchDataBufferByteSize
```
    See Also:
    
    StartupConfiguration.fetchDataBufferByteSize
  - proxyHost
```
public final java.lang.String proxyHost
```
    See Also:
    
    StartupConfiguration.proxyHost
  - proxyPort
```
public final int proxyPort
```
    See Also:
    
    StartupConfiguration.proxyPort
  - cookiePolicy
```
public final java.lang.String cookiePolicy
```
    See Also:
    
    StartupConfiguration.cookiePolicy
  - cookieMaxByteSize
```
public final int cookieMaxByteSize
```
    See Also:
    
    StartupConfiguration.cookieMaxByteSize
  - userAgent
```
public final java.lang.String userAgent
```
    See Also:
    
    StartupConfiguration.userAgent
  - userAgentFrom
```
public final java.lang.String userAgentFrom
```
    See Also:
    
    StartupConfiguration.userAgentFrom
  - robotsExpiration
```
public volatile long robotsExpiration
```
    See Also:
    
    StartupConfiguration.robotsExpiration
  - acceptAllCertificates
```
public volatile boolean acceptAllCertificates
```
    See Also:
    
    StartupConfiguration.acceptAllCertificates
  - rootDir
```
public final java.io.File rootDir
```
    See Also:
    
    StartupConfiguration.rootDir
  - storeDir
```
public final java.io.File storeDir
```
    See Also:
    
    StartupConfiguration.storeDir
  - responseCacheDir
```
public final java.io.File responseCacheDir
```
    See Also:
    
    StartupConfiguration.responseCacheDir
  - sieveDir
```
public final java.io.File sieveDir
```
    See Also:
    
    StartupConfiguration.sieveDir
  - frontierDir
```
public final java.io.File frontierDir
```
    See Also:
    
    StartupConfiguration.frontierDir
  - responseBodyMaxByteSize
```
public volatile int responseBodyMaxByteSize
```
    See Also:
    
    StartupConfiguration.responseBodyMaxByteSize
  - digestAlgorithm
```
public final java.lang.String digestAlgorithm
```
    See Also:
    
    StartupConfiguration.digestAlgorithm
  - startPaused
```
public final boolean startPaused
```
    See Also:
    
    StartupConfiguration.startPaused
  - storeClass
```
public final java.lang.Class<? extends Store> storeClass
```
    See Also:
    
    StartupConfiguration.storeClass
  - workbenchMaxByteSize
```
public volatile long workbenchMaxByteSize
```
    See Also:
    
    StartupConfiguration.workbenchMaxByteSize
  - virtualizerMaxByteSize
```
public final long virtualizerMaxByteSize
```
    See Also:
    
    StartupConfiguration.virtualizerMaxByteSize
  - urlCacheMaxByteSize
```
public volatile long urlCacheMaxByteSize
```
    See Also:
    
    StartupConfiguration.urlCacheMaxByteSize
  - sieveSize
```
public final int sieveSize
```
    See Also:
    
    StartupConfiguration.sieveSize
  - sieveStoreIOBufferByteSize
```
public final int sieveStoreIOBufferByteSize
```
    See Also:
    
    StartupConfiguration.sieveStoreIOBufferByteSize
  - sieveAuxFileIOBufferByteSize
```
public final int sieveAuxFileIOBufferByteSize
```
    See Also:
    
    StartupConfiguration.sieveAuxFileIOBufferByteSize
  - dnsCacheMaxSize
```
public final int dnsCacheMaxSize
```
    See Also:
    
    StartupConfiguration.dnsCacheMaxSize
  - dnsPositiveTtl
```
public final long dnsPositiveTtl
```
    See Also:
    
    StartupConfiguration.dnsPositiveTtl
  - dnsNegativeTtl
```
public final long dnsNegativeTtl
```
    See Also:
    
    StartupConfiguration.dnsNegativeTtl
  - crawlIsNew
```
public final boolean crawlIsNew
```
    See Also:
    
    StartupConfiguration.crawlIsNew
  - spamDetector
```
public final SpamDetector<?> spamDetector
```
    See Also:
    
    StartupConfiguration.spamDetectorUri
  - spamDetectionThreshold
```
public final int spamDetectionThreshold
```
    See Also:
    
    StartupConfiguration.spamDetectionThreshold
  - spamDetectionPeriodicity
```
public final int spamDetectionPeriodicity
```
    See Also:
    
    StartupConfiguration.spamDetectionPeriodicity
  - parsers
```
public final java.util.ArrayList<Parser<?>> parsers
```
    The parser, instantiated. Parsers used by ParsingThread instances are obtained by copying this parsers.
  - paused
```
public volatile boolean paused
```
    Whether the crawler is currently paused. When this variable changes to false, a notifyAll() is issued on this runtime configuration.
  - stopping
```
public volatile boolean stopping
```
    Whether the crawler is currently being stopping. The change of state to true is stable—this variable will never become false again.
  - dnsResolver
```
public final org.apache.http.conn.DnsResolver dnsResolver
```
    The DNS resolver used throughout the crawler.
    
    See Also:
    
    StartupConfiguration.dnsResolverClass
  - DOTTED_ADDRESS
```
public static final java.util.regex.Pattern DOTTED_ADDRESS
```
    A pattern used to identify hosts specified directed via their address in dotted notation. Note the dot at the end. It covers both IPv6 addresses (where hexadecimal notation is accepted by default) and IPv4 addresses (where hexadecimal notation requires the 0x prefix on every single piece of the address).
- Constructor Detail
  - RuntimeConfiguration
```
public RuntimeConfiguration(StartupConfiguration startupConfiguration)
                     throws ConfigurationException,
                            java.io.IOException
```
    Throws:
    
    ConfigurationException
    
    java.io.IOException
- Method Detail
  - addBlackListedIPv4
```
public void addBlackListedIPv4(java.lang.String spec)
                        throws ConfigurationException,
                               java.io.FileNotFoundException
```
    Adds a (or a set of) new IPv4 to the black list; the IPv4 can be specified directly or it can be a file (prefixed by file:).
    
    Parameters:
    
    spec - the specification (an IP address, or a file prefixed by file).
    
    Throws:
    
    ConfigurationException
    
    java.io.FileNotFoundException
  - addBlackListedHost
```
public void addBlackListedHost(java.lang.String spec)
                        throws ConfigurationException,
                               java.io.FileNotFoundException
```
    Adds a (or a set of) new host to the black list; the host can be specified directly or it can be a file (prefixed by file:).
    
    Parameters:
    
    spec - the specification (a host, or a file prefixed by file).
    
    Throws:
    
    ConfigurationException
    
    java.io.FileNotFoundException
  - ensureNotPaused
```
public void ensureNotPaused()
                     throws java.lang.InterruptedException
```
    Throws:
    
    java.lang.InterruptedException
  - toString
```
public java.lang.String toString()
```
    Overrides:
    
    toString in class java.lang.Object
  - parsersFromSpecs
```
public static java.util.ArrayList<Parser<?>> parsersFromSpecs(java.lang.String[] specs)
                                                       throws java.lang.IllegalArgumentException,
                                                              java.lang.ClassNotFoundException,
                                                              java.lang.IllegalAccessException,
                                                              java.lang.reflect.InvocationTargetException,
                                                              java.lang.InstantiationException,
                                                              java.lang.NoSuchMethodException,
                                                              java.io.IOException
```
    Given an array of parser specifications, it returns the corresponding list of parsers (only the correct specifications are put in the list.
    
    Parameters:
    
    specs - the parser specifications (they will be parsed using ObjectParser.
    
    Returns:
    
    a list of parsers built according to the specifications (only the parseable items are put in the list).
    
    Throws:
    
    java.lang.IllegalArgumentException
    
    java.lang.ClassNotFoundException
    
    java.lang.IllegalAccessException
    
    java.lang.reflect.InvocationTargetException
    
    java.lang.InstantiationException
    
    java.lang.NoSuchMethodException
    
    java.io.IOException

Class RuntimeConfiguration

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

FETCH_ROBOTS

name

group

weight

maxUrlsPerSchemeAuthority

fetchingThreads

parsingThreads

dnsThreads

fetchFilter

scheduleFilter

parseFilter

followFilter

storeFilter

keepAliveTime

schemeAuthorityDelay

ipDelay

ipDelayFactor

maxUrls

bloomFilterPrecision

seed

blackListedIPv4Addresses

blackListedIPv4Lock

blackListedHostHashes

blackListedHostHashesLock

socketTimeout

connectionTimeout

fetchDataBufferByteSize

proxyHost

proxyPort

cookiePolicy

cookieMaxByteSize

userAgent

userAgentFrom

robotsExpiration

acceptAllCertificates

rootDir

storeDir

responseCacheDir

sieveDir

frontierDir

responseBodyMaxByteSize

digestAlgorithm

startPaused

storeClass

workbenchMaxByteSize

virtualizerMaxByteSize

urlCacheMaxByteSize

sieveSize

sieveStoreIOBufferByteSize

sieveAuxFileIOBufferByteSize

dnsCacheMaxSize

dnsPositiveTtl

dnsNegativeTtl

crawlIsNew

spamDetector

spamDetectionThreshold

spamDetectionPeriodicity

parsers

paused

stopping

dnsResolver

DOTTED_ADDRESS

Constructor Detail

RuntimeConfiguration

Method Detail

addBlackListedIPv4

addBlackListedHost

ensureNotPaused

toString

parsersFromSpecs