org.apache.reef.io.data.loading.impl

Class InputFormatLoadingService<K,V>

java.lang.Object

org.apache.reef.io.data.loading.impl.InputFormatLoadingService<K,V>

All Implemented Interfaces:

DataLoadingService

public class InputFormatLoadingService<K,V>
extends Object
implements DataLoadingService

An implementation of DataLoadingService that uses the Hadoop InputFormat to find partitions of data and request resources.

The InputFormat is taken from the job configurations

The EvaluatorToPartitionStrategy is injected via Tang, in order to support different ways to map evaluators to data

Constructor Summary

Constructors

Constructor and Description
InputFormatLoadingService(EvaluatorToPartitionStrategy<org.apache.hadoop.mapred.InputSplit> evaluatorToPartitionStrategy, boolean inMemory, String inputFormatClass)

Method Summary

Methods

Modifier and Type	Method and Description
String	getComputeContextIdPrefix()
Configuration	getContextConfiguration(AllocatedEvaluator allocatedEvaluator)
int	getNumberOfPartitions() This method actually returns the number of splits in all partition of the data.
Configuration	getServiceConfiguration(AllocatedEvaluator allocatedEvaluator)
boolean	isComputeContext(ActiveContext context)
boolean	isDataLoadedContext(ActiveContext context) Distinguishes data loaded contexts from compute contexts.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

InputFormatLoadingService

@Inject
public InputFormatLoadingService(EvaluatorToPartitionStrategy<org.apache.hadoop.mapred.InputSplit> evaluatorToPartitionStrategy,
                                boolean inMemory,
                                String inputFormatClass)

Method Detail

getNumberOfPartitions

public int getNumberOfPartitions()

This method actually returns the number of splits in all partition of the data. We should probably need to rename it in the future

Specified by:

getNumberOfPartitions in interface DataLoadingService

Returns:

the number of partitions suggested by this DataSource.

getContextConfiguration

public Configuration getContextConfiguration(AllocatedEvaluator allocatedEvaluator)

Specified by:

getContextConfiguration in interface DataLoadingService

Returns:

the context configuration for the given Evaluator.

getServiceConfiguration

public Configuration getServiceConfiguration(AllocatedEvaluator allocatedEvaluator)

Specified by:

getServiceConfiguration in interface DataLoadingService

Returns:

the service configuration for the given Evaluator.

getComputeContextIdPrefix

public String getComputeContextIdPrefix()

Specified by:

getComputeContextIdPrefix in interface DataLoadingService

Returns:

Return the prefix to be used to enumerate context ids for compute requests fired other than the data load contexts.

isComputeContext

public boolean isComputeContext(ActiveContext context)

Specified by:

isComputeContext in interface DataLoadingService

Returns:

true if this is a computation context, false otherwise. (e.g. this is a data loading context).

isDataLoadedContext

public boolean isDataLoadedContext(ActiveContext context)

Description copied from interface: DataLoadingService

Distinguishes data loaded contexts from compute contexts.

Specified by:

isDataLoadedContext in interface DataLoadingService

Returns:

true if this context has been loaded with data.