Package it.unimi.di.law.bubing.util

Class URLRespectsRobots

java.lang.Object

it.unimi.di.law.bubing.util.URLRespectsRobots

public class URLRespectsRobots
extends java.lang.Object

A class providing static methods to parse robots.txt into arrays of char arrays and handle robot filtering.

Field Summary

Fields

Modifier and Type	Field	Description
static char[][]	EMPTY_ROBOTS_FILTER	A singleton empty robots filter.
static int	MAX_TO_STRING_ROBOTS	The maximum number of robots entries returned by Object.toString().

Method Summary

Modifier and Type	Method	Description
static boolean	apply(char[][] robotsFilter, java.net.URI url)	Checks whether a specified URL passes a specified robots filter.
static void	main(java.lang.String[] arg)
static char[][]	parseRobotsReader(java.io.Reader content, java.lang.String userAgent)	Parses the argument as if it were the content of a robots.txt file, and returns a sorted array of prefixes of URLs that the agent should not follow.
static char[][]	parseRobotsResponse(URIResponse robotsResponse, java.lang.String userAgent)	Parses a robots.txt file contained in a FetchData and returns the corresponding filter as an array of sorted prefixes.
static char[][]	toSortedPrefixFreeCharArrays(java.util.Set<java.lang.String> set)
static java.lang.String	toString(char[][] robotsFilter)	Prints gracefully a robot filter using at most 30 prefixes.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MAX_TO_STRING_ROBOTS

public static final int MAX_TO_STRING_ROBOTS

The maximum number of robots entries returned by Object.toString().

See Also:

Constant Field Values

EMPTY_ROBOTS_FILTER

public static final char[][] EMPTY_ROBOTS_FILTER

A singleton empty robots filter.

Method Detail

toSortedPrefixFreeCharArrays

public static char[][] toSortedPrefixFreeCharArrays(java.util.Set<java.lang.String> set)

parseRobotsReader

public static char[][] parseRobotsReader(java.io.Reader content,
                                         java.lang.String userAgent)
                                  throws java.io.IOException

Parses the argument as if it were the content of a robots.txt file, and returns a sorted array of prefixes of URLs that the agent should not follow.

Parameters:

content - the content of the robots.txt file.

userAgent - the string representing the user agent of interest.

Returns:

an array of character arrays, which are prefixes of the URLs not to follow, in sorted order.

Throws:

java.io.IOException

parseRobotsResponse

public static char[][] parseRobotsResponse(URIResponse robotsResponse,
                                           java.lang.String userAgent)
                                    throws java.io.IOException

Parses a robots.txt file contained in a FetchData and returns the corresponding filter as an array of sorted prefixes. HTTP statuses different from 2xx are logged. HTTP statuses of class 4xx generate an empty filter. HTTP statuses 2xx/3xx cause the tentative parsing of the request content. In the remaining cases we return null.

Parameters:

robotsResponse - the response containing robots.txt.

userAgent - the string representing the user agent of interest.

Returns:

an array of character arrays, which are prefixes of the URLs not to follow, in sorted order, or null

Throws:

java.io.IOException

apply

public static boolean apply(char[][] robotsFilter,
                            java.net.URI url)

Checks whether a specified URL passes a specified robots filter.

Parameters:

robotsFilter - a robot filter.

url - a URL to check against robotsFilter.

Returns:

true if url passes robotsFilter.

toString

public static java.lang.String toString(char[][] robotsFilter)

Prints gracefully a robot filter using at most 30 prefixes.

Parameters:

robotsFilter - a robots filter.

Returns:

a string describing the filter.

main

public static void main(java.lang.String[] arg)
                 throws java.io.IOException

Throws:

java.io.IOException