Package it.unimi.dsi.law.warc.parser

Class Digester

java.lang.Object

it.unimi.dsi.law.warc.parser.Digester

All Implemented Interfaces:

Callback

public class Digester
extends Object
implements Callback

A callback computing the digest of a page.

The page is somewhat simplified before being passed (as a sequence of bytes obtained by breaking each character into the upper and lower byte) to MessageDigest.update(byte[]). All start/end tags are case-normalized, and all their content (except for the element-type name) is removed. An exception is made for SRC attribute of FRAME and IFRAME elements, as they are necessary to distinguish correctly framed pages without alternative text. The attributes will be resolved w.r.t. the URL associated to the page.

To avoid clashes between digests coming from different sites, you can optionally set a URL whose authority that will be used to update the digest before adding the actual text page. You can set the URL with url(URI). A good idea is to use the host name (or even the authority).

Field Summary

Fields inherited from interface it.unimi.dsi.parser.callback.Callback

EMPTY_CALLBACK_ARRAY

Constructor Summary

Constructors

Constructor	Description
Digester(String algorithm)	Creates a new callback using the given message digest.

Method Summary

Modifier and Type	Method	Description
boolean	cdata(Element element, char[] data, int offset, int length)
boolean	characters(char[] data, int offset, int length, boolean flowBroken)
void	configure(BulletParser parser)
byte[]	digest()	Returns the digest computed.
void	endDocument()
boolean	endElement(Element element)
void	startDocument()
boolean	startElement(Element element, Map<Attribute,MutableString> attributes)
void	url(URI uri)	Sets the URI that will be used to tune the next digest.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

Digester

public Digester(String algorithm)
         throws NoSuchAlgorithmException

Creates a new callback using the given message digest.

Parameters:

algorithm - a message digest algorithm (to be passed to MessageDigest.getInstance(java.lang.String)).

Throws:

NoSuchAlgorithmException

Method Details

configure

public void configure(BulletParser parser)

Specified by:

configure in interface Callback

digest

public byte[] digest()

Returns the digest computed.

Returns:

the digest computed.

url

public void url(URI uri)

Sets the URI that will be used to tune the next digest.

Parameters:

uri - a URI, or null for no URL.

startDocument

public void startDocument()

Specified by:

startDocument in interface Callback

startElement

public boolean startElement(Element element,
Map<Attribute,MutableString> attributes)

Specified by:

startElement in interface Callback

endElement

public boolean endElement(Element element)

Specified by:

endElement in interface Callback

characters

public boolean characters(char[] data,
int offset,
int length,
boolean flowBroken)

Specified by:

characters in interface Callback

cdata

public boolean cdata(Element element,
char[] data,
int offset,
int length)

Specified by:

cdata in interface Callback

endDocument

public void endDocument()

Specified by:

endDocument in interface Callback