Package it.unimi.di.law.bubing.parser

Class HTMLParser.DigestAppendable

java.lang.Object

it.unimi.di.law.bubing.parser.HTMLParser.DigestAppendable

All Implemented Interfaces:

java.lang.Appendable

Enclosing class:

HTMLParser<T>

public static final class HTMLParser.DigestAppendable
extends java.lang.Object
implements java.lang.Appendable

A class computing the digest of a page.

The page is somewhat simplified before being passed (as a sequence of bytes obtained by breaking each character into the upper and lower byte) to a Hasher. All start/end tags are case-normalized, and their whole content (except for the element-type name) is removed. An exception is made for SRC attribute of FRAME and IFRAME elements, as they are necessary to distinguish correctly framed pages without alternative text. The attributes will be resolved w.r.t. the URL associated to the page. Moreover, non-HTML tags are substituted with a special tag unknown.

For what concerns the text, all digits are substituted by a whitespace, and nonempty whitespace maximal sequences are coalesced to a single space. Tags are considered as a non-whitespace character.

To avoid clashes between digests coming from different sites, you can optionally set a URL (passed to the init(URI) method) whose scheme+authority will be used to update the digest before adding the actual text page.

Additionally, since BUbiNG 0.9.10 location redirect URLs (both from headers and from META elements), if present, are mixed in to avoid collapsing 3xx pages with boilerplate text.

Field Summary

Fields

Modifier and Type	Field	Description
protected byte[]	digest	The last returne digest, or null if init(URI) has been called but digest() hasn't.
protected static Reference2ObjectOpenHashMap<java.lang.String,byte[]>	endTags	Cached byte representations of all closing tags.
protected com.google.common.hash.Hasher	hasher	The hasher currently used to compute the digest.
protected com.google.common.hash.HashFunction	hashFunction	The message digest used to compute the digest.
protected boolean	lastAppendedWasSpace	True iff the last character appended was a space.
protected static Reference2ObjectOpenHashMap<java.lang.String,byte[]>	startTags	Cached byte representations of all opening tags.

Constructor Summary

Constructors

Constructor	Description
DigestAppendable(com.google.common.hash.HashFunction hashFunction)	Create a digest appendable using a given hash function.

Method Summary

Modifier and Type	Method	Description
java.lang.Appendable	append(char c)
java.lang.Appendable	append(java.lang.CharSequence csq)
java.lang.Appendable	append(java.lang.CharSequence csq, int start, int end)
byte[]	digest()
void	endTag(net.htmlparser.jericho.EndTag endTag)
void	init(java.net.URI url)	Initializes the digest computation.
void	startTag(net.htmlparser.jericho.StartTag startTag)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

startTags

protected static final Reference2ObjectOpenHashMap<java.lang.String,byte[]> startTags

Cached byte representations of all opening tags. The map must be queried using Jericho names.

endTags

protected static final Reference2ObjectOpenHashMap<java.lang.String,byte[]> endTags

Cached byte representations of all closing tags. The map must be queried using Jericho names.

hashFunction

protected final com.google.common.hash.HashFunction hashFunction

The message digest used to compute the digest.

hasher

protected com.google.common.hash.Hasher hasher

The hasher currently used to compute the digest.

lastAppendedWasSpace

protected boolean lastAppendedWasSpace

True iff the last character appended was a space.

digest

protected byte[] digest

The last returne digest, or null if init(URI) has been called but digest() hasn't.

Constructor Detail

DigestAppendable

public DigestAppendable(com.google.common.hash.HashFunction hashFunction)

Create a digest appendable using a given hash function.

Parameters:

hashFunction - the hash function used to digest.

Method Detail

init

public void init(java.net.URI url)

Initializes the digest computation.

Parameters:

url - a URL, or null for no URL. In the former case, the host name will be used to initialize the digest.

append

public java.lang.Appendable append(java.lang.CharSequence csq,
                                   int start,
                                   int end)

Specified by:

append in interface java.lang.Appendable

append

public java.lang.Appendable append(char c)

Specified by:

append in interface java.lang.Appendable

append

public java.lang.Appendable append(java.lang.CharSequence csq)

Specified by:

append in interface java.lang.Appendable

digest

public byte[] digest()

startTag

public void startTag(net.htmlparser.jericho.StartTag startTag)

endTag

public void endTag(net.htmlparser.jericho.EndTag endTag)