Agent (BUbiNG 0.9.15)

java.lang.Object
- it.unimi.dsi.jai4j.jgroups.JGroupsJobManager<BubingJob>
- - it.unimi.di.law.bubing.Agent

All Implemented Interfaces:

it.unimi.dsi.jai4j.JobManager<BubingJob>, org.jgroups.MembershipListener, org.jgroups.MessageListener, org.jgroups.Receiver
```
public class Agent
extends it.unimi.dsi.jai4j.jgroups.JGroupsJobManager<BubingJob>
```
A BUbiNG agent. This class contains the main method used to start a BUbiNG agent, and exposes on JMX a number of methods that expose internal settings. In many cases settings can be changed while BUbiNG is running.

Nested Class Summary
- Nested classes/interfaces inherited from class it.unimi.dsi.jai4j.jgroups.JGroupsJobManager
  it.unimi.dsi.jai4j.jgroups.JGroupsJobManager.JAI4JAddress

Field Summary

Fields
Modifier and Type	Field	Description
`static java.lang.String`	`JGROUPS_CONFIGURATION_PROPERTY_NAME`	The name of the system property that, if set, makes it possible to choose a JGroups configuration file.
`static java.lang.String`	`JMX_REMOTE_PORT_SYSTEM_PROPERTY`	The name of the standard Java system property that sets the JMX service port (it must be set for the agent to start).
`protected MessageThread`	`messageThread`
`protected QuickMessageThread`	`quickMessageThread`

Fields inherited from class it.unimi.dsi.jai4j.jgroups.JGroupsJobManager
assignmentStrategy, droppingThreadFactory, identifier2RemoteJobManager, jGroupsChannel, jGroupsGroupName, jGroupsView, jmxServiceURL, jobListener, jobQueue, name, pendingMessagesStrategy, remoteSelf, selfObjectName, weight

Constructor Summary

Constructors
Constructor Description

Agent(java.lang.String hostname, int jmxPort, RuntimeConfiguration rc)

Constructors
Constructor	Description
`Agent(java.lang.String hostname, int jmxPort, RuntimeConfiguration rc)`

Method Summary

All Methods Static Methods Instance Methods Concrete Methods
Modifier and Type	Method	Description
`void`	`addBlackListedHost(java.lang.String host)`
`void`	`addBlackListedIPv4(java.lang.String address)`
`void`	`flush()`
`BubingJob`	`fromByteArray(byte[] array, int offset)`
`BubingJob`	`fromString(java.lang.String s)`
`int`	`getActiveFecthingThreads()`
`java.lang.String`	`getArchetypeContentLength()`
`long`	`getArchetypeContentTypeApplication()`
`long`	`getArchetypeContentTypeImage()`
`long`	`getArchetypeContentTypeOthers()`
`long`	`getArchetypeContentTypeText()`
`java.lang.String`	`getArchetypeExternalOutdegree()`
`java.lang.String`	`getArchetypeOutdegree()`
`long`	`getArchetypes()`
`long`	`getArchetypes1xx()`
`long`	`getArchetypes2xx()`
`long`	`getArchetypes3xx()`
`long`	`getArchetypes4xx()`
`long`	`getArchetypes5xx()`
`long`	`getArchetypesOther()`
`long`	`getBroken()`
`long`	`getBrokenVisitStates()`
`long`	`getBrokenVisitStatesOnWorkbench()`
`long`	`getBytes()`
`int`	`getConnectionTimeout()`
`int`	`getDnsThreads()`
`double`	`getDuplicatePercentage()`
`long`	`getDuplicates()`
`double`	`getEntryAverage()`
`double`	`getEntryMax()`
`double`	`getEntryMin()`
`double`	`getEntryVariance()`
`java.lang.String`	`getFetchFilter()`
`int`	`getFetchingThreads()`
`long`	`getFetchingThreadTotalWaitTime()`
`long`	`getFetchingThreadWaits()`
`java.lang.String`	`getFollowFilter()`
`long`	`getIpDelay()`
`long`	`getIPOnWorkbench()`
`long`	`getKeepAliveTime()`
`int`	`getKnownCount()`	Returns the number of agents currently known to the JAI4J `RemoteJobManager`.
`java.lang.String`	`getManager(java.lang.String url)`
`long`	`getMaxUrls()`
`java.lang.String`	`getParseFilter()`
`int`	`getParsingThreads()`
`int[]`	`getQueueDistribution()`
`int`	`getReadyToParse()`
`long`	`getReadyURLs()`
`long`	`getReceivedURLs()`
`long`	`getRequests()`
`long`	`getRequiredFrontSize()`
`long`	`getResolvedVisitStates()`
`long`	`getResources()`
`long`	`getResponseBodyMaxByteSize()`
`long`	`getRobotsExpiration()`
`java.lang.String`	`getScheduleFilter()`
`int`	`getSocketTimeout()`
`java.lang.String`	`getStoreFilter()`
`long`	`getStoreSize()`
`long`	`getToDoSize()`
`int`	`getUnknownHosts()`
`long`	`getUnresolved()`
`long`	`getUrlCacheMaxByteSize()`
`long`	`getUrlDelay()`
`long`	`getURLsInQueues()`
`double`	`getURLsInQueuesPercentage()`
`int`	`getVisitStates()`
`long`	`getVisitStatesOnDisk()`
`long`	`getVisitStatesOnWorkbench()`
`int`	`getWaitingVisitStates()`
`long`	`getWorkbenchByteSize()`
`long`	`getWorkbenchMaxByteSize()`
`static void`	`main(java.lang.String[] arg)`
`void`	`pause()`
`void`	`resume()`
`void`	`setConnectionTimeout(int connectionTimeout)`
`void`	`setDnsThreads(int dnsThreads)`
`void`	`setFetchFilter(java.lang.String spec)`
`void`	`setFetchingThreads(int fetchingThreads)`
`void`	`setFollowFilter(java.lang.String spec)`
`void`	`setIpDelay(long ipDelay)`
`void`	`setKeepAliveTime(int keepAliveTime)`
`void`	`setMaxUrls(long maxUrls)`
`void`	`setParseFilter(java.lang.String spec)`
`void`	`setParsingThreads(int parsingThreads)`
`void`	`setResponseBodyMaxByteSize(int responseBodyMaxByteSize)`
`void`	`setRobotsExpiration(long robotsExpiration)`
`void`	`setScheduleFilter(java.lang.String spec)`
`void`	`setSchemeAuthorityDelay(long schemeAuthorityDelay)`
`void`	`setSocketTimeout(int socketTimeout)`
`void`	`setStoreFilter(java.lang.String spec)`
`void`	`setUrlCacheMaxByteSize(long urlCacheSize)`
`void`	`setWorkbenchMaxByteSize(long workbenchSize)`
`void`	`stop()`
`byte[]`	`toByteArray(BubingJob job)`

Methods inherited from class it.unimi.dsi.jai4j.jgroups.JGroupsJobManager
block, close, connect, createJobMessage, createJobMessageFromByteArray, drop, fromByteArray, fromInputStream, getAliveCount, getRemoteJobManagers, getState, getSuspectedCount, isConnected, local, poll, receive, receive, receive, receive, refresh, register, remoteJobManagers, setListener, setState, submit, submit, suspect, toOutputStream, toString, unblock, viewAccepted

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail
- JMX_REMOTE_PORT_SYSTEM_PROPERTY
```
public static final java.lang.String JMX_REMOTE_PORT_SYSTEM_PROPERTY
```
  The name of the standard Java system property that sets the JMX service port (it must be set for the agent to start).
  
  See Also:
  
  Constant Field Values
- JGROUPS_CONFIGURATION_PROPERTY_NAME
```
public static final java.lang.String JGROUPS_CONFIGURATION_PROPERTY_NAME
```
  The name of the system property that, if set, makes it possible to choose a JGroups configuration file.
  
  See Also:
  
  Constant Field Values
- messageThread
```
protected final MessageThread messageThread
```
  See Also:
  
  MessageThread
- quickMessageThread
```
protected final QuickMessageThread quickMessageThread
```
  See Also:
  
  QuickMessageThread

Constructor Detail

Agent

public Agent(java.lang.String hostname,
             int jmxPort,
             RuntimeConfiguration rc)
      throws java.lang.Exception

Throws:: java.lang.Exception

Method Detail

fromString
```
public BubingJob fromString(java.lang.String s)
```
Specified by:

fromString in interface it.unimi.dsi.jai4j.JobManager<BubingJob>

Overrides:

fromString in class it.unimi.dsi.jai4j.jgroups.JGroupsJobManager<BubingJob>

toByteArray

public byte[] toByteArray(BubingJob job)
                   throws java.lang.IllegalArgumentException

Throws:: java.lang.IllegalArgumentException

fromByteArray

public BubingJob fromByteArray(byte[] array,
                               int offset)
                        throws java.lang.IllegalArgumentException

Throws:: java.lang.IllegalArgumentException

getKnownCount
```
public int getKnownCount()
```
Returns the number of agents currently known to the JAI4J RemoteJobManager.
Note that this number will be larger than that returned by JGroupsJobManager.getAliveCount() if there are suspected agents.

Returns:

the number of agents currently known to the JAI4J RemoteJobManager.

stop
```
public void stop()
```

pause
```
public void pause()
```

resume
```
public void resume()
```

flush

public void flush()
           throws java.io.IOException,
                  java.lang.InterruptedException

Throws:: java.io.IOException; java.lang.InterruptedException

addBlackListedIPv4

public void addBlackListedIPv4(java.lang.String address)
                        throws ConfigurationException,
                               java.io.FileNotFoundException

Throws:: ConfigurationException; java.io.FileNotFoundException

addBlackListedHost

public void addBlackListedHost(java.lang.String host)
                        throws ConfigurationException,
                               java.io.FileNotFoundException

Throws:: ConfigurationException; java.io.FileNotFoundException

getManager

public java.lang.String getManager(java.lang.String url)
                            throws it.unimi.dsi.jai4j.NoSuchJobManagerException

Throws:: it.unimi.dsi.jai4j.NoSuchJobManagerException

setDnsThreads

public void setDnsThreads(int dnsThreads)
                   throws java.lang.IllegalArgumentException

Throws:: java.lang.IllegalArgumentException

getDnsThreads
```
public int getDnsThreads()
```

setFetchingThreads

public void setFetchingThreads(int fetchingThreads)
                        throws java.lang.IllegalArgumentException,
                               java.security.NoSuchAlgorithmException,
                               java.io.IOException

Throws:: java.lang.IllegalArgumentException; java.security.NoSuchAlgorithmException; java.io.IOException

getFetchingThreads
```
public int getFetchingThreads()
```

setParsingThreads

public void setParsingThreads(int parsingThreads)
                       throws java.lang.IllegalArgumentException

Throws:: java.lang.IllegalArgumentException

getParsingThreads
```
public int getParsingThreads()
```

setFetchFilter

public void setFetchFilter(java.lang.String spec)
                    throws ParseException

Throws:: ParseException

getFetchFilter

public java.lang.String getFetchFilter()

setScheduleFilter

public void setScheduleFilter(java.lang.String spec)
                       throws ParseException

Throws:: ParseException

getScheduleFilter

public java.lang.String getScheduleFilter()

setParseFilter

public void setParseFilter(java.lang.String spec)
                    throws ParseException

Throws:: ParseException

getParseFilter

public java.lang.String getParseFilter()

setFollowFilter

public void setFollowFilter(java.lang.String spec)
                     throws ParseException

Throws:: ParseException

getFollowFilter

public java.lang.String getFollowFilter()

setStoreFilter

public void setStoreFilter(java.lang.String spec)
                    throws ParseException

Throws:: ParseException

getStoreFilter

public java.lang.String getStoreFilter()

setResponseBodyMaxByteSize

public void setResponseBodyMaxByteSize(int responseBodyMaxByteSize)

getResponseBodyMaxByteSize

public long getResponseBodyMaxByteSize()

setKeepAliveTime

public void setKeepAliveTime(int keepAliveTime)

getKeepAliveTime
```
public long getKeepAliveTime()
```

setSchemeAuthorityDelay

public void setSchemeAuthorityDelay(long schemeAuthorityDelay)

getUrlDelay
```
public long getUrlDelay()
```

setIpDelay

public void setIpDelay(long ipDelay)

getIpDelay
```
public long getIpDelay()
```

setMaxUrls

public void setMaxUrls(long maxUrls)

getMaxUrls
```
public long getMaxUrls()
```

setSocketTimeout

public void setSocketTimeout(int socketTimeout)

getSocketTimeout
```
public int getSocketTimeout()
```

setConnectionTimeout

public void setConnectionTimeout(int connectionTimeout)

getConnectionTimeout
```
public int getConnectionTimeout()
```

setRobotsExpiration

public void setRobotsExpiration(long robotsExpiration)

getRobotsExpiration
```
public long getRobotsExpiration()
```

setWorkbenchMaxByteSize

public void setWorkbenchMaxByteSize(long workbenchSize)

getWorkbenchMaxByteSize
```
public long getWorkbenchMaxByteSize()
```

setUrlCacheMaxByteSize

public void setUrlCacheMaxByteSize(long urlCacheSize)

getUrlCacheMaxByteSize
```
public long getUrlCacheMaxByteSize()
```

getWorkbenchByteSize
```
public long getWorkbenchByteSize()
```

getStoreSize
```
public long getStoreSize()
```

getArchetypes
```
public long getArchetypes()
```

getArchetypesOther
```
public long getArchetypesOther()
```

getArchetypes1xx
```
public long getArchetypes1xx()
```

getArchetypes2xx
```
public long getArchetypes2xx()
```

getArchetypes3xx
```
public long getArchetypes3xx()
```

getArchetypes4xx
```
public long getArchetypes4xx()
```

getArchetypes5xx
```
public long getArchetypes5xx()
```

getArchetypeOutdegree

public java.lang.String getArchetypeOutdegree()

getArchetypeExternalOutdegree

public java.lang.String getArchetypeExternalOutdegree()

getArchetypeContentLength

public java.lang.String getArchetypeContentLength()

getArchetypeContentTypeText

public long getArchetypeContentTypeText()

getArchetypeContentTypeImage

public long getArchetypeContentTypeImage()

getArchetypeContentTypeApplication

public long getArchetypeContentTypeApplication()

getArchetypeContentTypeOthers

public long getArchetypeContentTypeOthers()

getRequests
```
public long getRequests()
```

getResources
```
public long getResources()
```

getBytes
```
public long getBytes()
```

getReceivedURLs
```
public long getReceivedURLs()
```

getDuplicates
```
public long getDuplicates()
```

getDuplicatePercentage

public double getDuplicatePercentage()

getReadyURLs
```
public long getReadyURLs()
```

getFetchingThreadWaits
```
public long getFetchingThreadWaits()
```

getFetchingThreadTotalWaitTime

public long getFetchingThreadTotalWaitTime()

getURLsInQueues
```
public long getURLsInQueues()
```

getURLsInQueuesPercentage

public double getURLsInQueuesPercentage()

getQueueDistribution
```
public int[] getQueueDistribution()
```

getUnresolved
```
public long getUnresolved()
```

getBroken
```
public long getBroken()
```

getEntryAverage
```
public double getEntryAverage()
```

getEntryMax
```
public double getEntryMax()
```

getEntryMin
```
public double getEntryMin()
```

getEntryVariance
```
public double getEntryVariance()
```

getVisitStates
```
public int getVisitStates()
```

getResolvedVisitStates
```
public long getResolvedVisitStates()
```

getIPOnWorkbench
```
public long getIPOnWorkbench()
```

getVisitStatesOnWorkbench

public long getVisitStatesOnWorkbench()

getToDoSize
```
public long getToDoSize()
```

getActiveFecthingThreads
```
public int getActiveFecthingThreads()
```

getReadyToParse
```
public int getReadyToParse()
```

getUnknownHosts
```
public int getUnknownHosts()
```

getBrokenVisitStates
```
public long getBrokenVisitStates()
```

getBrokenVisitStatesOnWorkbench

public long getBrokenVisitStatesOnWorkbench()

getWaitingVisitStates
```
public int getWaitingVisitStates()
```

getVisitStatesOnDisk
```
public long getVisitStatesOnDisk()
```

getRequiredFrontSize
```
public long getRequiredFrontSize()
```

main

public static void main(java.lang.String[] arg)
                 throws java.lang.Exception

Throws:: java.lang.Exception

Class Agent

Nested Class Summary

Nested classes/interfaces inherited from class it.unimi.dsi.jai4j.jgroups.JGroupsJobManager

Field Summary

Fields inherited from class it.unimi.dsi.jai4j.jgroups.JGroupsJobManager

Constructor Summary

Method Summary

Methods inherited from class it.unimi.dsi.jai4j.jgroups.JGroupsJobManager

Methods inherited from class java.lang.Object

Field Detail

JMX_REMOTE_PORT_SYSTEM_PROPERTY

JGROUPS_CONFIGURATION_PROPERTY_NAME

messageThread

quickMessageThread

Constructor Detail

Agent

Method Detail

fromString

toByteArray

fromByteArray

getKnownCount

stop

pause

resume

flush

addBlackListedIPv4

addBlackListedHost

getManager

setDnsThreads

getDnsThreads

setFetchingThreads

getFetchingThreads

setParsingThreads

getParsingThreads

setFetchFilter

getFetchFilter

setScheduleFilter

getScheduleFilter

setParseFilter

getParseFilter

setFollowFilter

getFollowFilter

setStoreFilter

getStoreFilter

setResponseBodyMaxByteSize

getResponseBodyMaxByteSize

setKeepAliveTime

getKeepAliveTime

setSchemeAuthorityDelay

getUrlDelay

setIpDelay

getIpDelay

setMaxUrls

getMaxUrls

setSocketTimeout

getSocketTimeout

setConnectionTimeout

getConnectionTimeout

setRobotsExpiration

getRobotsExpiration

setWorkbenchMaxByteSize

getWorkbenchMaxByteSize

setUrlCacheMaxByteSize

getUrlCacheMaxByteSize

getWorkbenchByteSize

getStoreSize

getArchetypes

getArchetypesOther

getArchetypes1xx

getArchetypes2xx

getArchetypes3xx

getArchetypes4xx

getArchetypes5xx

getArchetypeOutdegree

getArchetypeExternalOutdegree

getArchetypeContentLength

getArchetypeContentTypeText

getArchetypeContentTypeImage

getArchetypeContentTypeApplication

getArchetypeContentTypeOthers