Package it.unimi.di.law.bubing.util

Class FetchData

java.lang.Object

it.unimi.di.law.bubing.util.FetchData

All Implemented Interfaces:

URIResponse, java.io.Closeable, java.lang.AutoCloseable

public class FetchData
extends java.lang.Object
implements URIResponse, java.io.Closeable

Response of a HTTP request. At construction, data needed to issue requests are set up; in particular, an InspectableFileCachedInputStream is instantiated that will be later used to read data from the socket and to store them for the later stages (parsing, storing etc.): note that the memory needed for the buffer of the InspectableFileCachedInputStream, as well as the associated overflow file, are created at construction time, and will be disposed only by calling close().

After construction, the fetch(URI, HttpClient, RequestConfig, VisitState, boolean) method is used to issue the request; after that, all data obtained as a response are available. All data is available until disposal, or until another call to fetch(URI, HttpClient, RequestConfig, VisitState, boolean).

Note that since this object will be populated by one thread and used by another all fields must be either finalfinal or volatile.

Field Summary

Fields

Modifier and Type	Field	Description
BinaryParser	binaryParser	The binary parser associated with this fetched response.
long	endTime	System.currentTimeMillis() when the GET request was completed.
java.lang.Throwable	exception	The exception thrown in case of a failed fetch, or null.
boolean	inUse	If true, this istance has been enqueued to the list of results and we are waiting for the signal of the ParsingThread that is analyzing it.
static int	OVERFLOW_FILES_RANDOM_PATH_ELEMENTS	The number of path elements for the hierarchical overflow files (see Util.createHierarchicalTempFile(File, int, String, String)).
protected org.apache.http.HttpResponse	response	The response from Apache Http Components returned during the last fetch.
boolean	robots	Whether we are fecthing a robots file.
long	startTime	System.currentTimeMillis() when the GET request was issued.
protected boolean	truncated	True if the last fetch was truncated because of exceedingly long response body.
protected java.net.URI	url	The BUbiNG URL associated with this request.
VisitState	visitState	The visit state associated with this request.

Constructor Summary

Constructors

Constructor	Description
FetchData(RuntimeConfiguration rc)	Creates a fetched response according to the given properties.

Method Summary

Modifier and Type	Method	Description
void	abort()	Invokes AbstractExecutionAwareRequest.abort() on the underlying request.
void	close()	Disposes the underlying InspectableFileCachedInputStream.
byte[]	digest()	Get the digest
void	digest(byte[] digest)	Set the digest with a given value
void	fetch(java.net.URI url, org.apache.http.client.HttpClient httpClient, org.apache.http.client.config.RequestConfig requestConfig, VisitState visitState, boolean robots)	Fetches a given URL.
boolean	isDuplicate()	Get whether the current FetchData is duplicate or not
void	isDuplicate(boolean isDuplicate)	Mark the current FetchData as duplicated or not duplicated
long	length()	Returns (an approximation of) the length of the response (headers and body).
org.apache.http.HttpResponse	response()	Returns the response part.
java.lang.String	toString()
java.net.URI	uri()	Returns the URI part.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

OVERFLOW_FILES_RANDOM_PATH_ELEMENTS

public static final int OVERFLOW_FILES_RANDOM_PATH_ELEMENTS

The number of path elements for the hierarchical overflow files (see Util.createHierarchicalTempFile(File, int, String, String)).

See Also:

Constant Field Values

url

protected volatile java.net.URI url

The BUbiNG URL associated with this request.

visitState

public volatile VisitState visitState

The visit state associated with this request.

response

protected volatile org.apache.http.HttpResponse response

The response from Apache Http Components returned during the last fetch.

truncated

protected volatile boolean truncated

True if the last fetch was truncated because of exceedingly long response body.

startTime

public volatile long startTime

System.currentTimeMillis() when the GET request was issued.

endTime

public volatile long endTime

System.currentTimeMillis() when the GET request was completed.

exception

public volatile java.lang.Throwable exception

The exception thrown in case of a failed fetch, or null.

robots

public boolean robots

Whether we are fecthing a robots file.

binaryParser

public final BinaryParser binaryParser

The binary parser associated with this fetched response.

inUse

public volatile boolean inUse

If true, this istance has been enqueued to the list of results and we are waiting for the signal of the ParsingThread that is analyzing it.

Constructor Detail

FetchData

public FetchData(RuntimeConfiguration rc)
          throws java.security.NoSuchAlgorithmException,
                 java.lang.IllegalArgumentException,
                 java.io.IOException

Creates a fetched response according to the given properties.

Parameters:

rc - the runtime configuration.

Throws:

java.security.NoSuchAlgorithmException

java.lang.IllegalArgumentException

java.io.IOException

Method Detail

length

public long length()

Returns (an approximation of) the length of the response (headers and body).

We cannot compute easily the exact length because we must deduce it from headers and body length.

Returns:

(an approximation of) the length of the response content.

uri

public java.net.URI uri()

Description copied from interface: URIResponse

Returns the URI part.

Specified by:

uri in interface URIResponse

Returns:

the URI.

response

public org.apache.http.HttpResponse response()

Description copied from interface: URIResponse

Returns the response part.

Specified by:

response in interface URIResponse

Returns:

the response.

fetch

public void fetch(java.net.URI url,
                  org.apache.http.client.HttpClient httpClient,
                  org.apache.http.client.config.RequestConfig requestConfig,
                  VisitState visitState,
                  boolean robots)
           throws java.io.IOException

Fetches a given URL.

Parameters:

httpClient - the client that will be used to fetch url.

url - the URL to be used to populate this response.

visitState - the VisitState associated with url.

Throws:

java.io.IOException

digest

public void digest(byte[] digest)

Set the digest with a given value

Parameters:

digest - the value to be set for digest

digest

public byte[] digest()

Get the digest

Returns:

the digest of this FetchData

isDuplicate

public void isDuplicate(boolean isDuplicate)

Mark the current FetchData as duplicated or not duplicated

Parameters:

isDuplicate - a boolean value indicating whether the current FetchData should be marked as duplicated or not

isDuplicate

public boolean isDuplicate()

Get whether the current FetchData is duplicate or not

Returns:

true if this is duplicate or not

toString

public java.lang.String toString()

Overrides:

toString in class java.lang.Object

abort

public void abort()

Invokes AbstractExecutionAwareRequest.abort() on the underlying request.

close

public void close()
           throws java.io.IOException

Disposes the underlying InspectableFileCachedInputStream.

Specified by:

close in interface java.lang.AutoCloseable

Specified by:

close in interface java.io.Closeable

Throws:

java.io.IOException