Package it.unimi.di.law.bubing.frontier

Class ParsingThread.FrontierEnqueuer

java.lang.Object

it.unimi.di.law.bubing.frontier.ParsingThread.FrontierEnqueuer

Enclosing class:

ParsingThread

protected static final class ParsingThread.FrontierEnqueuer
extends java.lang.Object

A small gadget used to insert links in the frontier. It should be initialized specifying URI and scheme/authority of the page being visited and the robot filter to be applied. Then, one or more URLs can be enqueued: the actual enqueuing takes place only if the URL passes both the schedule and the robots filter.

Field Summary

Fields

Modifier and Type	Field	Description
int	outlinks
int	scheduledLinks

Constructor Summary

Constructors

Constructor	Description
FrontierEnqueuer(Frontier frontier, RuntimeConfiguration rc)	Creates the enqueuer.

Method Summary

Modifier and Type	Method	Description
void	enqueue(java.net.URI url)	Enqueues the given URL, provided that it passes the schedule filter, its host is blacklisted.
void	init(java.net.URI uri, byte[] schemeAuthority, char[][] robotsFilter)	Initializes the enqueuer for parsing a page with a specific scheme+authority and robots filter.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

outlinks

public int outlinks

scheduledLinks

public int scheduledLinks

Constructor Detail

FrontierEnqueuer

public FrontierEnqueuer(Frontier frontier,
                        RuntimeConfiguration rc)

Creates the enqueuer.

Parameters:

frontier - the frontier instantiating the enqueuer.

rc - the configuration to be used.

Method Detail

init

public void init(java.net.URI uri,
                 byte[] schemeAuthority,
                 char[][] robotsFilter)

Initializes the enqueuer for parsing a page with a specific scheme+authority and robots filter.

Parameters:

schemeAuthority - the scheme+authority of the page to be parsed.

robotsFilter - the robots filter of the (authority of the) page to be parsed.

enqueue

public void enqueue(java.net.URI url)

Enqueues the given URL, provided that it passes the schedule filter, its host is blacklisted. Moreover, if the scheme+authority is the same as the one of the page being parsed, we check that the URL respects the robots filter.

Parameters:

url - the URL to be enqueued.