com.screenscraper.datamanager.skeleton

Class BasicDataManager

java.lang.Object

com.screenscraper.datamanager.skeleton.BasicDataManager

All Implemented Interfaces:

DataManager

Direct Known Subclasses:

SqlDataManager, XmlDataManager

public class BasicDataManager
extends java.lang.Object
implements DataManager

A DataManager contains the tree of DataNodes and also provides useful methods for adding data, writing, interacting with Schemas, etc.
The DataManager is the primary class for inserting, manipulating, and writing data to the underlying DataSource.
BasicDataManager implements many of the re-usable features of a DataManager, but is not intended to be an implemented class

Field Summary

Fields

Modifier and Type	Field and Description
protected CacheReaderWriter	cacher
protected CacheBlock	currentCacheBlock
protected java.util.Map<java.lang.String,DataNode>	currentDataNodes
protected EventEngine	eventEngine
protected java.util.Set<java.lang.String>	mergeable
protected RootNode	root
protected java.lang.Object	rootLock
protected DatabaseSchema	schema
protected java.util.Set<java.lang.String>	updateable

Constructor Summary

Constructors

Constructor and Description
BasicDataManager() Creates a DataManager tied to a screen-scraper session

Method Summary

Modifier and Type	Method and Description
void	addData(DataNode dataNode) Manually add a created DataNode to the DataManager for writing.
void	addData(java.lang.String schema, java.util.Map<java.lang.String,java.lang.Object> data) Add data to to the table represented by schema.
void	addData(java.lang.String schema, java.lang.String columnName, java.lang.Object value) Add data from the session variables to the table.
DataManagerEventListener	addDataAssertion(DataAssertion d) Adds a data assertion to be performed when the write is called.
DataManagerEventListener	addDataAssertion(DataAssertion d, DataManagerEventSource.EventFireTime when) Adds a DataAssertion to be run A DataAssertion is a special case of a DataManagerEvent that executes onSuccess if the assertion passes and onFail if the assertion fails
DataManagerEventListener	addEventListener(java.lang.String schema, DataManagerEventSource.EventFireTime when, DataManagerEventListener listener) Add an even listener.
DataManagerEventListener	addEventListener(java.lang.String schema, java.lang.String when, DataManagerEventListener listener) Adds an event listener
DataManagerEventListener	addEventListenerWithPriority(java.lang.String schema, DataManagerEventSource.EventFireTime when, DataManagerEventListener listener, int priority) Add an event listener with a specified execution priority
EventHandler<DataManagerSchemalessEvent>	addSchemalessEventListener(DataManagerSchemalessEvent.FireTime fireTime, EventHandler<DataManagerSchemalessEvent> listener) Adds a callback that will trigger at the specified time
EventHandler<DataManagerSchemalessEvent>	addSchemalessEventListenerWithPriority(DataManagerSchemalessEvent.FireTime fireTime, EventHandler<DataManagerSchemalessEvent> listener, int priority) Adds a callback that will trigger at the specified time
void	buildSchemas() Builds the schemas from a database.
void	cacheContent(java.lang.String name, java.lang.String content) Caches the current content and links it to the main record.
void	cacheContent(java.lang.String name, java.lang.String content, java.lang.String url) Caches the current content and links it to the main record.
void	cacheContent(java.lang.String name, java.lang.String content, java.lang.String url, java.lang.String contentType) Caches the current content and links it to the main record.
void	cacheContext(java.util.Map<java.lang.String,?> contextValues) Cache the current context, as passed in
void	cacheScrapeableFile(ScrapeableFile scrapeableFile) Caches the scrapeable file
void	cacheScrapeableFile(ScrapeableFile scrapeableFile, boolean includeHeaders) Caches the scrapeable file
static java.lang.String	catData(DataNode dataNode, java.util.Set<java.lang.String> columns) Concatenates the key value pairs of the specified columns in the DataNode
static java.lang.String	catData(java.util.SortedMap<java.lang.String,DataObject> m) Concatenates the key value pairs of the map
boolean	checkDataNode(java.lang.String schema) verifies that the data is ready to be inserted.
void	clearAllData() Clears out all the nodes of the DataManager.
void	clearData(java.lang.String schema) Clear out all the uncommitted data for the given schema
void	close() close should be called when you are done with the DataManager
boolean	commit(java.lang.String schema) commits the row of data corresponding to the schema, so the next time addData is called it will be in a new row.
protected void	connectTree(DataNode dataNode) Commit calls this method to create necessary parent nodes and connect this node to the active DataNode tree
void	enablePageAndContextCaching(java.lang.String mainSchema, CacheReaderWriter cacher) Enables page and context caching for the given main table
boolean	flush() writes the current data to the Database and clears out all the data.
static java.lang.String	generateUID(java.lang.String name) generates a 40 character UID comprised of 0-9a-z based on a SHA1 hash
CacheReaderWriter	getCacheReaderWriter() Returns the Cacher for the object
java.util.Set<java.lang.String>	getColumns(java.lang.String table) Retrieve the columns for a given schema
DataNode	getCurrentDataNode(java.lang.String schema) retrieves the current DataNode that is in scope for the schema
DatabaseSchema	getDatabaseSchema() Retrieve Database schema, for a Relational Database this contains all the information about the tables
DataNode	getNewDataNode(java.lang.String table) Returns a new data node from the provided schema identifier.
DataWriter	getNewDataWriter()
RootNode	getRoot() returns the root node of the DataNode tree.
RelationalSchema	getSchema(java.lang.String table) returns a schema with the given identifier
static java.lang.String	hashData(DataNode dataNode, java.util.Set<java.lang.String> columns) hashes the return of catData(n,columns) with a SHA1 hash
static java.lang.String	hashData(java.util.SortedMap<java.lang.String,DataObject> map) Hashes the return of catData(m) with a SHA1 hash
boolean	hasSchema(java.lang.String table) returns whether or not th
boolean	importSchemas(java.lang.String file)
void	removeEventListener(java.lang.String schema, DataManagerEventSource.EventFireTime when, DataManagerEventListener listener) Removes an event listener
void	removeSchemalessEventListener(DataManagerSchemalessEvent.FireTime fireTime, EventHandler<DataManagerSchemalessEvent> listener) Removes an event listener
void	rollback(java.lang.String schema) removes the DataNode corresponding to the schema prior to flush being called
DataManager	setDatabaseSchema(DatabaseSchema s) Manually set the DatabaseSchema.
void	setMergeEnabled(java.lang.String table, boolean merge) Sets all the nodes to be able to merge the values with the corresponding rows in the table when there is a primary key match.
void	setUpdateEnabled(java.lang.String table, boolean update) Sets all the nodes to be able to overwrite columns in the table with the new values when there is a primary key match.
boolean	write() writes out the root dataNode(without clearing the data structure) with the default DataWriter

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

rootLock

protected final java.lang.Object rootLock

root

protected RootNode root

eventEngine

@NotNull
protected final EventEngine eventEngine

cacher

@Nullable
protected CacheReaderWriter cacher

currentCacheBlock

@Nullable
protected CacheBlock currentCacheBlock

schema

@Nullable
protected DatabaseSchema schema

updateable

protected java.util.Set<java.lang.String> updateable

mergeable

protected java.util.Set<java.lang.String> mergeable

currentDataNodes

@NotNull
protected final java.util.Map<java.lang.String,DataNode> currentDataNodes

Constructor Detail

BasicDataManager

public BasicDataManager()

Creates a DataManager tied to a screen-scraper session

Method Detail

buildSchemas

public void buildSchemas()

Description copied from interface: DataManager

Builds the schemas from a database. This can take some time so if there are any tables in the database that are not required for the DataManager it is recommended to use buildSchemas(List<String> tables)

Specified by:

buildSchemas in interface DataManager

getNewDataWriter

@NotNull
 @InternalOnly
public DataWriter getNewDataWriter()

importSchemas

@InternalOnly
public boolean importSchemas(java.lang.String file)

getColumns

@NotNull
 @InternalOnly
public java.util.Set<java.lang.String> getColumns(java.lang.String table)

Description copied from interface: DataManager

Retrieve the columns for a given schema

Specified by:

getColumns in interface DataManager

Parameters:

table - The schema to get columns for

Returns:

A set of the columns in the given schema

getSchema

@NotNull
public RelationalSchema getSchema(@NotNull
                                            java.lang.String table)

returns a schema with the given identifier

Specified by:

getSchema in interface DataManager

Parameters:

table - The table for which to get a schema

Returns:

AbstractSchema corresponding the table

hasSchema

public boolean hasSchema(@NotNull
                         java.lang.String table)

Description copied from interface: DataManager

returns whether or not th

Specified by:

hasSchema in interface DataManager

Parameters:

table - The schema to check for

Returns:

True if the schema exists, false otherwise

setUpdateEnabled

public void setUpdateEnabled(java.lang.String table,
                             boolean update)

Sets all the nodes to be able to overwrite columns in the table with the new values when there is a primary key match.

Parameters:

table - The name of the table

update - Sets whether updates should occur for the table

setMergeEnabled

public void setMergeEnabled(java.lang.String table,
                            boolean merge)

Sets all the nodes to be able to merge the values with the corresponding rows in the table when there is a primary key match.

Parameters:

table - The name of the table

merge - Sets whether merging should occur for the table

addData

public void addData(java.lang.String schema,
                    java.util.Map<java.lang.String,java.lang.Object> data)

Add data to to the table represented by schema. Entries corresponding to columns defined in the schema will be added. addData can be called multiple times per row that you want to insert into the database, then when the row has all the data call commit(String schema) and the next time you call addData a new row will be started. Data is not actually written to the database until flush() is called.

Specified by:

addData in interface DataManager

Parameters:

schema - The schema to add data for

data - The data to add

addData

public void addData(java.lang.String schema,
                    java.lang.String columnName,
                    java.lang.Object value)

Add data from the session variables to the table. Column name matches are case insensitive

Specified by:

addData in interface DataManager

Parameters:

schema - The schema for which to add data

columnName - The name of the column in the database

value - The value to add

commit

public boolean commit(java.lang.String schema)

commits the row of data corresponding to the schema, so the next time addData is called it will be in a new row.

Specified by:

commit in interface DataManager

Parameters:

schema - The name of the schema for which to commit data

Returns:

the value of checkDataNode called on the corresponding DataNode

checkDataNode

public boolean checkDataNode(java.lang.String schema)

verifies that the data is ready to be inserted. If false is returned the cause can be retrieved via getMessage.

Specified by:

checkDataNode in interface DataManager

Parameters:

schema - The name of the schema for which to check the current node

Returns:

true if data meets constraints defined in the AbstractSchema

getCurrentDataNode

public DataNode getCurrentDataNode(java.lang.String schema)

retrieves the current DataNode that is in scope for the schema

Specified by:

getCurrentDataNode in interface DataManager

Parameters:

schema - The name of the schema for which to retrieve the current node

Returns:

current DataNode that is in scope for the schema

rollback

public void rollback(java.lang.String schema)

removes the DataNode corresponding to the schema prior to flush being called

Parameters:

schema - The name of the schema to rollback

connectTree

protected void connectTree(@NotNull
                           DataNode dataNode)

Commit calls this method to create necessary parent nodes and connect this node to the active DataNode tree

Parameters:

dataNode - Connects tree elements for this node

getNewDataNode

@NotNull
public DataNode getNewDataNode(@NotNull
                                         java.lang.String table)

Returns a new data node from the provided schema identifier. This should only be used if for some reason you need manual manipulation of the DataNode or DataNode tree. In general a DataManager.addData method should be used for adding data

Specified by:

getNewDataNode in interface DataManager

Parameters:

table - The name of the table for which to create a new node

Returns:

The new data node or null if the schema was invalid

addData

public void addData(@Nullable
                    DataNode dataNode)

Manually add a created DataNode to the DataManager for writing. This can be used in conjunction with DataManager.getNewDataNode for instances when manual manipulation of a DataNode is required. In general a DataManager.addData method should be used for adding data.

Specified by:

addData in interface DataManager

Parameters:

dataNode - The node to add

enablePageAndContextCaching

public void enablePageAndContextCaching(@Nullable
                                        java.lang.String mainSchema,
                                        @Nullable
                                        CacheReaderWriter cacher)

Enables page and context caching for the given main table

Parameters:

mainSchema - The name of the main schema (must have a integer or long value for the primary key)

cacher - The cacher to use

cacheScrapeableFile

public void cacheScrapeableFile(@NotNull
                                ScrapeableFile scrapeableFile)

Caches the scrapeable file

Parameters:

scrapeableFile - The scrapeable file to cache

cacheScrapeableFile

public void cacheScrapeableFile(@NotNull
                                ScrapeableFile scrapeableFile,
                                boolean includeHeaders)

Caches the scrapeable file

Parameters:

scrapeableFile - The scrapeable file to cache

includeHeaders - True if headers should be included in the cached content, false to exclude them

cacheContent

public void cacheContent(@NotNull
                         java.lang.String name,
                         @NotNull
                         java.lang.String content)

Caches the current content and links it to the main record. This does not cache the context

Parameters:

name - The name to use when caching the data

content - The content to cache

cacheContent

public void cacheContent(@NotNull
                         java.lang.String name,
                         @NotNull
                         java.lang.String content,
                         @Nullable
                         java.lang.String url)

Caches the current content and links it to the main record. This does not cache the context

Parameters:

name - The name to use when caching the data

content - The content to cache

url - The url the content came from.

cacheContent

public void cacheContent(@NotNull
                         java.lang.String name,
                         @NotNull
                         java.lang.String content,
                         @Nullable
                         java.lang.String url,
                         @Nullable
                         java.lang.String contentType)

Caches the current content and links it to the main record. This does not cache the context

Parameters:

name - The name to use when caching the data

content - The content to cache

url - The url the content came from.

contentType - The content type, such as application/json or text/html

cacheContext

public void cacheContext(@NotNull
                         java.util.Map<java.lang.String,?> contextValues)

Cache the current context, as passed in

Parameters:

contextValues - The values to save as context

getCacheReaderWriter

@Nullable
public CacheReaderWriter getCacheReaderWriter()

Returns the Cacher for the object

Returns:

The Cacher for the object

clearAllData

public void clearAllData()

Clears out all the nodes of the DataManager. This is called by flush and generally wouldn't be requited by the user. It is available for instances where when a certain condition is met you want to clear out all the information instead of writing.

Specified by:

clearAllData in interface DataManager

flush

public boolean flush()

Description copied from interface: DataManager

writes the current data to the Database and clears out all the data.

Specified by:

flush in interface DataManager

Returns:

true if flush was successful

write

@InternalOnly
public boolean write()

writes out the root dataNode(without clearing the data structure) with the default DataWriter

Returns:

True on success

getRoot

public RootNode getRoot()

returns the root node of the DataNode tree. This can be handed off to a DataFilter to do preprocessing before the tree is written to the database.

Specified by:

getRoot in interface DataManager

Returns:

RootNode of the DataNode tree

generateUID

@InternalOnly
public static java.lang.String generateUID(java.lang.String name)
                                                   throws java.security.NoSuchAlgorithmException

generates a 40 character UID comprised of 0-9a-z based on a SHA1 hash

Parameters:

name - A name to generate a UID for

Returns:

The generated UID

Throws:

java.security.NoSuchAlgorithmException - If the needed digest algorithm isn't available (this should never happen)

hashData

@NotNull
 @InternalOnly
public static java.lang.String hashData(java.util.SortedMap<java.lang.String,DataObject> map)

Hashes the return of catData(m) with a SHA1 hash

Parameters:

map - The map of data to hash

Returns:

31 character hash comprised of 0-9a-z based on a SHA1 hash

hashData

@NotNull
 @InternalOnly
public static java.lang.String hashData(@Nullable
                                                                  DataNode dataNode,
                                                                  @Nullable
                                                                  java.util.Set<java.lang.String> columns)

hashes the return of catData(n,columns) with a SHA1 hash

Parameters:

dataNode - The node for which to hash data

columns - The columns to hash

Returns:

40 character hash comprised of 0-9a-z based on a SHA1 hash

catData

@InternalOnly
public static java.lang.String catData(@Nullable
                                                      java.util.SortedMap<java.lang.String,DataObject> m)

Concatenates the key value pairs of the map

Parameters:

m - Sorted map to generate a cat String from.

Returns:

concatenated key-value string of Map

catData

@InternalOnly
public static java.lang.String catData(@Nullable
                                                      DataNode dataNode,
                                                      @Nullable
                                                      java.util.Set<java.lang.String> columns)

Concatenates the key value pairs of the specified columns in the DataNode

Parameters:

dataNode - The node for which to cat data

columns - The columns to cat

Returns:

concatenated key-value string of Map

close

public void close()

Description copied from interface: DataManager

close should be called when you are done with the DataManager

Specified by:

close in interface DataManager

clearData

public void clearData(java.lang.String schema)

Description copied from interface: DataManager

Clear out all the uncommitted data for the given schema

Specified by:

clearData in interface DataManager

Parameters:

schema - The name of the schema

setDatabaseSchema

@NotNull
public DataManager setDatabaseSchema(DatabaseSchema s)

Description copied from interface: DataManager

Manually set the DatabaseSchema. Use this for if you don't call buildSchemas for whatever reason

Specified by:

setDatabaseSchema in interface DataManager

Parameters:

s - The schema to set

Returns:

A reference to the DataManager

getDatabaseSchema

@Nullable
public DatabaseSchema getDatabaseSchema()

Description copied from interface: DataManager

Retrieve Database schema, for a Relational Database this contains all the information about the tables

Specified by:

getDatabaseSchema in interface DataManager

Returns:

The schema for the entire database this DataManager will write to

addDataAssertion

@NotNull
public DataManagerEventListener addDataAssertion(@NotNull
                                                           DataAssertion d)

Adds a data assertion to be performed when the write is called. A DataAssertion is a special case of a DataManagerEvent that executes onSuccess if the assertion passes and onFail if the assertion fails

Parameters:

d - The assertion to add

Returns:

The listener that will handle checking this assertion

addDataAssertion

@NotNull
public DataManagerEventListener addDataAssertion(@NotNull
                                                           DataAssertion d,
                                                           DataManagerEventSource.EventFireTime when)

Adds a DataAssertion to be run A DataAssertion is a special case of a DataManagerEvent that executes onSuccess if the assertion passes and onFail if the assertion fails

Parameters:

d - The assertion to add

when - The time to fire the assertion

Returns:

The listener that will handle checking this assertion

addSchemalessEventListener

@NotNull
public EventHandler<DataManagerSchemalessEvent> addSchemalessEventListener(@NotNull
                                                                                     DataManagerSchemalessEvent.FireTime fireTime,
                                                                                     @NotNull
                                                                                     EventHandler<DataManagerSchemalessEvent> listener)

Adds a callback that will trigger at the specified time

Parameters:

fireTime - The time at which to fire the event

listener - The listener to handle the event

Returns:

The listener that was added

addSchemalessEventListenerWithPriority

@NotNull
public EventHandler<DataManagerSchemalessEvent> addSchemalessEventListenerWithPriority(@NotNull
                                                                                                 DataManagerSchemalessEvent.FireTime fireTime,
                                                                                                 @NotNull
                                                                                                 EventHandler<DataManagerSchemalessEvent> listener,
                                                                                                 int priority)

Adds a callback that will trigger at the specified time

Parameters:

fireTime - The time at which to fire the event

listener - The listener to handle the event

priority - The priority, with lower numbers being higher priority

Returns:

The listener that was added

Since:

7.0.9a

removeSchemalessEventListener

public void removeSchemalessEventListener(@NotNull
                                          DataManagerSchemalessEvent.FireTime fireTime,
                                          @NotNull
                                          EventHandler<DataManagerSchemalessEvent> listener)

Removes an event listener

Parameters:

fireTime - The time at which the listener was set to fire

listener - The listener to remove

addEventListener

@NotNull
public DataManagerEventListener addEventListener(@NotNull
                                                           java.lang.String schema,
                                                           java.lang.String when,
                                                           DataManagerEventListener listener)

Adds an event listener

Parameters:

schema - The schema to add the listener to

when - When the event should fire

listener - The listener that will handle the event

Returns:

The listener that will handle the event

addEventListener

@NotNull
public DataManagerEventListener addEventListener(@NotNull
                                                           java.lang.String schema,
                                                           DataManagerEventSource.EventFireTime when,
                                                           DataManagerEventListener listener)

Description copied from interface: DataManager

Add an even listener. events are fired off during specific times

Specified by:

addEventListener in interface DataManager

Parameters:

schema - The schema the event will be added for

when - The time the event should fire

listener - The listener that will handle the event

Returns:

The listener that was added

addEventListenerWithPriority

@NotNull
public DataManagerEventListener addEventListenerWithPriority(@NotNull
                                                                       java.lang.String schema,
                                                                       DataManagerEventSource.EventFireTime when,
                                                                       DataManagerEventListener listener,
                                                                       int priority)

Description copied from interface: DataManager

Add an event listener with a specified execution priority

Specified by:

addEventListenerWithPriority in interface DataManager

Parameters:

schema - The schema the event will be added for

when - The time when the event should fire

listener - The listener that will handle the event

priority - The priority for the event, with lower numbers being higher priority

Returns:

The listener that was added

removeEventListener

public void removeEventListener(@NotNull
                                java.lang.String schema,
                                DataManagerEventSource.EventFireTime when,
                                DataManagerEventListener listener)

Removes an event listener

Parameters:

schema - The schema to remove the listener from

when - When the event would fire

listener - The listener that will be removed