Package it.unimi.di.law.warc.io

Class ParallelBufferedWarcWriter

java.lang.Object

it.unimi.di.law.warc.io.ParallelBufferedWarcWriter

All Implemented Interfaces:

WarcWriter, java.io.Closeable, java.lang.AutoCloseable

public class ParallelBufferedWarcWriter
extends java.lang.Object
implements WarcWriter

A parallel Warc writer. Instances of this class use a number of internal buffer and WarcWriter instances to parallelize record compression.

Records will be written to the OutputStream provided at construction time (which we suggest to buffer heavily) by a flushing thread. There is no guarantee about the order in which the records will be output.

Note that for each thread there is an associated FastByteArrayOutputStream that will grow as needed to accommodate the output of the record.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
protected static class	ParallelBufferedWarcWriter.WriterPair

Field Summary

Fields

Modifier and Type	Field	Description
protected java.util.concurrent.ArrayBlockingQueue<ParallelBufferedWarcWriter.WriterPair>	emptyPairs	The queue of empty ParallelBufferedWarcWriter.WriterPair instances.
protected java.util.concurrent.ArrayBlockingQueue<ParallelBufferedWarcWriter.WriterPair>	filledPairs	The queue of filled ParallelBufferedWarcWriter.WriterPair instances; their content will be flushed to disk by the flushingThread.
protected it.unimi.di.law.warc.io.ParallelBufferedWarcWriter.FlushingThread	flushingThread	The thread that iteratively extracts filled @link ParallelBufferedWarcWriter.WriterPair} instances from filledPairs, dump them to outputStream and enqueue them to emptyPairs.
protected java.io.IOException	flushingThreadException	The exception throw by the flushingThread, if any, or null.
protected java.io.OutputStream	outputStream	The final output stream.

Constructor Summary

Constructors

Constructor	Description
ParallelBufferedWarcWriter(java.io.OutputStream outputStream, boolean compress)	Creates a Warc parallel output stream using 2×Runtime.availableProcessors() buffers.
ParallelBufferedWarcWriter(java.io.OutputStream outputStream, boolean compress, int numberOfBuffers)	Creates a Warc parallel output stream.

Method Summary

Modifier and Type	Method	Description
void	close()
void	write(WarcRecord record)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

emptyPairs

protected final java.util.concurrent.ArrayBlockingQueue<ParallelBufferedWarcWriter.WriterPair> emptyPairs

The queue of empty ParallelBufferedWarcWriter.WriterPair instances.

filledPairs

protected final java.util.concurrent.ArrayBlockingQueue<ParallelBufferedWarcWriter.WriterPair> filledPairs

The queue of filled ParallelBufferedWarcWriter.WriterPair instances; their content will be flushed to disk by the flushingThread.

flushingThread

protected final it.unimi.di.law.warc.io.ParallelBufferedWarcWriter.FlushingThread flushingThread

The thread that iteratively extracts filled @link ParallelBufferedWarcWriter.WriterPair} instances from filledPairs, dump them to outputStream and enqueue them to emptyPairs.

outputStream

protected final java.io.OutputStream outputStream

The final output stream.

flushingThreadException

protected volatile java.io.IOException flushingThreadException

The exception throw by the flushingThread, if any, or null.

Constructor Detail

ParallelBufferedWarcWriter

public ParallelBufferedWarcWriter(java.io.OutputStream outputStream,
                                  boolean compress)

Creates a Warc parallel output stream using 2×Runtime.availableProcessors() buffers.

Parameters:

outputStream - the final output stream.

compress - whether to write compressed records.

ParallelBufferedWarcWriter

public ParallelBufferedWarcWriter(java.io.OutputStream outputStream,
                                  boolean compress,
                                  int numberOfBuffers)

Creates a Warc parallel output stream.

Parameters:

outputStream - the final output stream.

compress - whether to write compressed records.

numberOfBuffers - the number of buffers.

Method Detail

write

public void write(WarcRecord record)
           throws java.io.IOException,
                  java.lang.InterruptedException

Specified by:

write in interface WarcWriter

Throws:

java.io.IOException

java.lang.InterruptedException

close

public void close()
           throws java.io.IOException

Specified by:

close in interface java.lang.AutoCloseable

Specified by:

close in interface java.io.Closeable

Throws:

java.io.IOException