Package org.intermine.dataloader

Class BatchingFetcher

java.lang.Object

org.intermine.dataloader.BaseEquivalentObjectFetcher

org.intermine.dataloader.HintingFetcher

org.intermine.dataloader.BatchingFetcher

All Implemented Interfaces:

EquivalentObjectFetcher

Direct Known Subclasses:

ParallelBatchingFetcher

public class BatchingFetcher
extends HintingFetcher

Class providing EquivalentObjectFetcher functionality that batches fetches to improve performance.

Author:

Matthew Wakeling

Field Summary

Fields

Modifier and Type	Field	Description
protected int	batchQueried
protected int	cacheMisses
protected DataTracker	dataTracker
protected java.util.Map<InterMineObject,java.util.Set<InterMineObject>>	equivalents
protected Source	source
protected long	timeSpentExecute
protected long	timeSpentPrefetchEquiv
protected long	timeSpentPrefetchTracker

Fields inherited from class org.intermine.dataloader.HintingFetcher

allPkClassesEmptyForClass, savedCounts, savedTimes

Fields inherited from class org.intermine.dataloader.BaseEquivalentObjectFetcher

idMap, lookupOs, model, summaryCallCounts, summaryCounts, summaryTimes

Constructor Summary

Constructors

Constructor	Description
BatchingFetcher(BaseEquivalentObjectFetcher fetcher, DataTracker dataTracker, Source source)	Constructor

Method Summary

Modifier and Type	Method	Description
protected boolean	canDoPkNow(PrimaryKey pk, ClassDescriptor cld, java.util.Map<PrimaryKey,ClassDescriptor> pksNotDone)	Returns whether this primary key can be fetched now.
void	close(Source source)	Close method - prints out summary data.
protected void	doPk(PrimaryKey pk, ClassDescriptor cld, java.util.Map<InterMineObject,java.util.Set<InterMineObject>> results, java.util.List<InterMineObject> objectsForCld, java.util.Set<java.lang.Integer> fetchedObjectIds)	Fetches equivalent objects for a particular primary key.
protected void	doPks(java.util.Map<PrimaryKey,ClassDescriptor> pksToDo, java.util.Map<InterMineObject,java.util.Set<InterMineObject>> results, java.util.Map<ClassDescriptor,java.util.List<InterMineObject>> cldToObjectsForCld, long time1)	Fetches data for the given primary keys.
protected void	getEquivalentsFor(java.util.List<ResultsRow<java.lang.Object>> batch)	Fetches the equivalent object information for a whole batch of objects.
protected void	getEquivalentsForObjects(java.util.List<FastPathObject> fpos)	Fetches the equivalent object information for a whole batch of objects.
ObjectStore	getNoseyObjectStore(ObjectStore os)	Returns an ObjectStore layered on top of the given ObjectStore, which reports to this fetcher which objects are being loaded.
java.util.Set<InterMineObject>	queryEquivalentObjects(InterMineObject obj, Source source)	Returns a Set of objects that are equivalent to the given object, according to the primary keys defined by the given Source.

Methods inherited from class org.intermine.dataloader.HintingFetcher

createPKQueriesForClass, createPKQueryForPK, getSummary

Methods inherited from class org.intermine.dataloader.BaseEquivalentObjectFetcher

createPKQuery, getIdMap, getLookupOs, getModel

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

equivalents

protected java.util.Map<InterMineObject,java.util.Set<InterMineObject>> equivalents

dataTracker

protected DataTracker dataTracker

source

protected Source source

batchQueried

protected int batchQueried

cacheMisses

protected int cacheMisses

timeSpentExecute

protected long timeSpentExecute

timeSpentPrefetchEquiv

protected long timeSpentPrefetchEquiv

timeSpentPrefetchTracker

protected long timeSpentPrefetchTracker

Constructor Detail

BatchingFetcher

public BatchingFetcher(BaseEquivalentObjectFetcher fetcher,
                       DataTracker dataTracker,
                       Source source)

Constructor

Parameters:

fetcher - another EquivalentObjectFetcher

dataTracker - a DataTracker object to pass prefetch instructions to

source - the data Source that is being loaded

Method Detail

getNoseyObjectStore

public ObjectStore getNoseyObjectStore(ObjectStore os)

Returns an ObjectStore layered on top of the given ObjectStore, which reports to this fetcher which objects are being loaded.

Parameters:

os - an ObjectStore

Returns:

an ObjectStore

close

public void close(Source source)

Close method - prints out summary data.

Overrides:

close in class HintingFetcher

Parameters:

source - the Source used

queryEquivalentObjects

public java.util.Set<InterMineObject> queryEquivalentObjects(InterMineObject obj,
                                                             Source source)
                                                      throws ObjectStoreException

Returns a Set of objects that are equivalent to the given object, according to the primary keys defined by the given Source.

Specified by:

queryEquivalentObjects in interface EquivalentObjectFetcher

Overrides:

queryEquivalentObjects in class HintingFetcher

Parameters:

obj - the Object to look for

source - the data Source

Returns:

a Set of InterMineObjects

Throws:

ObjectStoreException - if an error occurs

getEquivalentsFor

protected void getEquivalentsFor(java.util.List<ResultsRow<java.lang.Object>> batch)
                          throws ObjectStoreException

Fetches the equivalent object information for a whole batch of objects.

Parameters:

batch - the objects

Throws:

ObjectStoreException - if something goes wrong

getEquivalentsForObjects

protected void getEquivalentsForObjects(java.util.List<FastPathObject> fpos)
                                 throws ObjectStoreException

Fetches the equivalent object information for a whole batch of objects.

Parameters:

fpos - the objects to fetch equivalents for

Throws:

ObjectStoreException - if something goes wrong

doPks

protected void doPks(java.util.Map<PrimaryKey,ClassDescriptor> pksToDo,
                     java.util.Map<InterMineObject,java.util.Set<InterMineObject>> results,
                     java.util.Map<ClassDescriptor,java.util.List<InterMineObject>> cldToObjectsForCld,
                     long time1)
              throws ObjectStoreException

Fetches data for the given primary keys.

Parameters:

pksToDo - a Map of the primary keys to fetch

results - a Map to hold results that are to be added to the cache

cldToObjectsForCld - a Map of Lists of objects relevant to PrimaryKeys

time1 - the time that processing started

Throws:

ObjectStoreException - if something goes wrong

canDoPkNow

protected boolean canDoPkNow(PrimaryKey pk,
                             ClassDescriptor cld,
                             java.util.Map<PrimaryKey,ClassDescriptor> pksNotDone)

Returns whether this primary key can be fetched now.

Parameters:

pk - the PrimaryKey

cld - the ClassDescriptor that the PrimaryKey is in

pksNotDone - a Map of pks not yet fetched

Returns:

a boolean

doPk

protected void doPk(PrimaryKey pk,
                    ClassDescriptor cld,
                    java.util.Map<InterMineObject,java.util.Set<InterMineObject>> results,
                    java.util.List<InterMineObject> objectsForCld,
                    java.util.Set<java.lang.Integer> fetchedObjectIds)
             throws ObjectStoreException

Fetches equivalent objects for a particular primary key.

Parameters:

pk - the PrimaryKey

cld - the ClassDescriptor of the PrimaryKey

results - a Map to hold results that are to be added to the cache

objectsForCld - a List of objects relevant to this PrimaryKey

fetchedObjectIds - a Set to hold ids of objects that are fetched, to prefetch from the data tracker later

Throws:

ObjectStoreException - if something goes wrong