com.screenscraper.datamanager.sql

Class SqlDataManager

java.lang.Object

com.screenscraper.datamanager.skeleton.BasicDataManager

com.screenscraper.datamanager.sql.SqlDataManager

All Implemented Interfaces:

DataManager

public class SqlDataManager
extends BasicDataManager
implements DataManager

A DataManager specific to sql databases connected to with a jdbc driver via a BasicDataSource

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	SqlDataManager.SqlDataManagerLogger Wrapper for a log4j logger

Field Summary

Fields

Modifier and Type	Field and Description
(package private) java.util.concurrent.ThreadPoolExecutor	executor
SqlDataManager.SqlDataManagerLogger	log The log object used by the data manager.
(package private) ScrapingSession	session

Fields inherited from class com.screenscraper.datamanager.skeleton.BasicDataManager

cacher, currentCacheBlock, currentDataNodes, eventEngine, mergeable, root, rootLock, schema, updateable

Constructor Summary

Constructors

Constructor and Description
SqlDataManager(javax.sql.DataSource dataSource) Creates an SqlDataManager that uses the given data source to connect to the database
SqlDataManager(javax.sql.DataSource dataSource, ScrapingSession session) Creates an SqlDataManager that uses the given data source to connect to the database

Method Summary

Modifier and Type	Method and Description
SchemaForeignKey	addForeignKey(java.lang.String childTable, java.lang.String childColumn, java.lang.String parentTable, java.lang.String parentColumn) Add a foreign key to the schemas.
void	addSessionVariablesOnCommit(boolean val) Sets whether or not to add session variables on commit
void	awaitCompletionOfPendingWrites() Blocks until all pending and active writes have completed from all threads.
void	awaitCompletionOfPendingWrites(long timeout, java.util.concurrent.TimeUnit unit) Blocks until all pending and active writes have completed from all threads.
void	buildSchemas() Builds the schemas from a database.
void	buildSchemas(boolean useOldSchemaIfKnown) Builds the schemas from a database.
void	buildSchemas(java.util.List<java.lang.String> tables) Builds the schemas from a database including only tables
void	buildSchemas(java.lang.String schemaName) Builds the schemas from a database.
void	buildSchemas(java.lang.String schemaName, boolean useOldSchemaIfKnown) Builds the schemas from a database.
void	buildSchemas(java.lang.String schemaName, java.util.List<java.lang.String> tables) Builds the schemas from a database including only tables
void	cacheContext() Cache the current context, using the session to get all SEARCH_ values for the context
boolean	checkColumnLength(DataNode n, java.lang.String column)
boolean	checkDataNode(java.lang.String schema) verifies that the data is ready to be inserted.
boolean	checkNullValue(DataNode n, java.lang.String column)
void	clearSessionVariables(java.lang.String table) Clears session variables that match fields in the given table
void	clearSessionVariablesOnCommit(boolean val) Sets whether or not to clear session variables on commit
void	clearThreadLocals() Clears all thread locals for the object
void	close() close should be called when you are done with the DataManager
boolean	commit(java.lang.String table) commits the row of data corresponding to the schema, so the next time addData is called it will be in a new row.
boolean	constraintMet(DataNode n, SchemaKey k)
void	enableConnectionMonitoring() Tells this data manager to log out information about connections it has given out that have been open for a long time.
void	enablePageAndContextCaching(java.lang.String cacheTableName, java.util.Collection<java.lang.String> scrapeableFilesToCache, boolean autoCacheDataRecords, boolean keepMostRecentPageOnly) Deprecated.
boolean	fkConstraintsMet(DataNode n)
boolean	flush() writes the current data to the Database and clears out all the data.
boolean	flushSameThread() This method is for when you have multi-threaded database writes enabled, but you need certain writes to be inline, for example if you need to run a query afterwards that depends on the data you just wrote being current.
java.sql.Connection	getActiveConnection() Returns the connection currently writing to the database for the active thread
CacheLoader	getCacheLoader() Deprecated.
CacheLoader	getCacheLoader(java.lang.String additionalWhereConstraints) Deprecated.
java.sql.Connection	getConnection() get a direct connection to the database from the Connection pool for specific queries When the connection is done being used call close() on it to return it to the pool.
java.sql.Connection	getConnection(java.lang.String name) get a direct connection to the database from the Connection pool for specific queries When the connection is done being used call close() on it to return it to the pool.
java.lang.String	getConnectionURL() Returns the URL used to connection to the database for this SqlDataManager, or the string "Unknown" if the url wasn't determinable
int	getCurrentActiveWriters() Returns the current number of writing tasks.
DataManagerDataSource	getDataSource() Returns the DataSource for the object.
DataObject	getLastAutoIncrementValue(java.lang.String table) returns the value of the autoincrement key for the table for the last commit, or null if none is set
int	getMaxActiveWritersSoFar() Returns the maximum number of writing tasks that have occurred at any given time.
int	getMaxBackgroundWriterThreads() Returns the maximum number of background writer threads that will be used at any given time
int	getMaxQueuedWritesSoFar() Returns the maximum number of writing tasks that have been in queue at any given time.
DataWriter	getNewDataWriter() Gets a new data writer for writing to the database
java.util.Set<java.lang.String>	getPrimaryKeys(java.lang.String table) return a set of the columns which are primary keys defined in the SqlSchema
int	getQueuedWrites() Returns the current number of writing tasks that are in queue but not running.
java.lang.String	getSchemaName() Returns the SchemaName for the object (this is largely for PostgreSQL where you can have a single database with multiple schemas, each with tables in them)
java.util.Set<RelationalSchema>	getSchemas() Helper method for backward compatibility to retrieve the relational (table) schemas
boolean	isSqlAutoCommit() Returns whether or not this data manager should use auto commit (each statement is it's own transaction)
boolean	keyConstraintsMet(DataNode n)
boolean	keyConstraintsMet(DataNode n, java.lang.String column)
void	removeEventListener(java.lang.String schema, DataManagerEventSource.EventFireTime when, DataManagerEventListener listener) Removes an event listener
void	setActiveConnection(java.sql.Connection connection) Sets the connection currently writing to the database for the active thread
void	setAutoCloseAfterScrapeEnds() Tells the data manager to close when the scrape ends.
void	setAutoIncrementSessionDuplicateRecords(ScrapingSession session, java.lang.String schema, DataManagerEventSource.EventFireTime fireTime) Increments the number of duplicate records saved value on the session when the given fire time occurs for the given schema
void	setAutoIncrementSessionDuplicateRecords(java.lang.String schema, DataManagerEventSource.EventFireTime fireTime) Increments the number of duplicate records saved value on the session when the given fire time occurs for the given schema
void	setAutoIncrementSessionErrorRecords(ScrapingSession session, java.lang.String schema, DataManagerEventSource.EventFireTime fireTime) Increments the number of error records saved value on the session when the given fire time occurs for the given schema
void	setAutoIncrementSessionErrorRecords(java.lang.String schema, DataManagerEventSource.EventFireTime fireTime) Increments the number of error records saved value on the session when the given fire time occurs for the given schema
void	setAutoIncrementSessionNewRecords(ScrapingSession session, java.lang.String schema, DataManagerEventSource.EventFireTime fireTime) Increments the number of new records saved value on the session when the given fire time occurs for the given schema
void	setAutoIncrementSessionNewRecords(java.lang.String schema, DataManagerEventSource.EventFireTime fireTime) Increments the number of new records saved value on the session when the given fire time occurs for the given schema
void	setAutoIncrementSessionUpdatedOrDuplicateRecordsByFields(ScrapingSession session, java.lang.String schema, java.lang.String... ignoreFieldsForDuplicateCheck) Increments the number of updated or duplicate records saved value on the session when the a record is being written.
void	setAutoIncrementSessionUpdatedOrDuplicateRecordsByFields(java.lang.String schema, java.lang.String... ignoreFieldsForDuplicateCheck) Increments the number of updated or duplicate records saved value on the session when the a record is being written.
void	setAutoIncrementSessionUpdatedRecords(ScrapingSession session, java.lang.String schema, DataManagerEventSource.EventFireTime fireTime) Increments the number of updated records saved value on the session when the given fire time occurs for the given schema
void	setAutoIncrementSessionUpdatedRecords(java.lang.String schema, DataManagerEventSource.EventFireTime fireTime) Increments the number of updated records saved value on the session when the given fire time occurs for the given schema
void	setAutoManyToMany(boolean enable) toggles automatically generating many to many relationship nodes based on schema information.
void	setBasicAutoIncrementRecordsByWriteType(ScrapingSession session, java.lang.String schema, java.lang.String... ignoreFieldsForDuplicateCheck) Sets up basic new/updated/duplicate/error record count increment handlers for the given schema.
void	setBasicAutoIncrementRecordsByWriteType(java.lang.String schema, java.lang.String... ignoreFieldsForDuplicateCheck) Sets up basic new/updated/duplicate/error record count increment handlers for the given schema.
DataManagerDataSource	setDataSource(javax.sql.DataSource newDataSource) Sets a new data source for the data manager.
void	setDateFormat(java.lang.String table, java.lang.String column, java.time.format.DateTimeFormatter inputDateFormat) Set the input date format for a date/time/timestamp column so that the database can convert a String input it into an appropriate value for the database
void	setDateFormat(java.lang.String table, java.lang.String column, java.text.SimpleDateFormat inputDateFormat) Set the input date format for a date/time/timestamp column so that the database can convert a String input it into an appropriate value for the database
void	setGlobalMergeEnabled(boolean merge) Sets all the nodes to be able to merge the values with the corresponding rows in the database when there is a primary key match.
void	setGlobalUpdateEnabled(boolean update) Sets all the nodes to be able to overwrite columns in the database with the new values when there is a primary key match.
void	setLastAutoIncrementValue(java.lang.String schemaName, DataObject value) Manually sets the last auto increment value for a schema.
void	setLastAutoIncrementValue(java.lang.String schemaName, java.lang.Object value) Manually sets the last auto increment value for a schema.
void	setLoggingLevel(org.apache.log4j.Level level) Sets the logging level
void	setLoggingLevelDebug() Shortcut method for setting the log level to debug
void	setLoggingLevelError() Shortcut method for setting the log level to error
void	setLoggingLevelInfo() Shortcut method for setting the log level to info
void	setLoggingLevelTrace() Shortcut method for setting the log level to trace
void	setLoggingLevelWarn() Shortcut method for setting the log level to warn
protected void	setManyToMany(DataNode dataNode) If a many to many relationship between this node and another in the set of current datanodes is possible and doesn't already exist, that node will be created.
void	setMultiThreadWrite(int numThreads) enables database writing to be done in its own thread.
void	setSchemaName(java.lang.String schemaName) Sets the SchemaName for the object (this is largely for PostgreSQL where you can have a single database with multiple schemas, each with tables in them)
void	setSingleThreadWrite() disables database writing to be done in its own thread.
void	setSqlAutoCommit(boolean sqlAutoCommit) Sets whether or not this data manager should use auto commit (each statement is it's own transaction)
(package private) boolean	validateData(DataNode n)

Methods inherited from class com.screenscraper.datamanager.skeleton.BasicDataManager

addData, addData, addData, addDataAssertion, addDataAssertion, addEventListener, addEventListener, addEventListenerWithPriority, addSchemalessEventListener, addSchemalessEventListenerWithPriority, cacheContent, cacheContent, cacheContent, cacheContext, cacheScrapeableFile, cacheScrapeableFile, catData, catData, clearAllData, clearData, connectTree, enablePageAndContextCaching, generateUID, getCacheReaderWriter, getColumns, getCurrentDataNode, getDatabaseSchema, getNewDataNode, getRoot, getSchema, hashData, hashData, hasSchema, importSchemas, removeSchemalessEventListener, rollback, setDatabaseSchema, setMergeEnabled, setUpdateEnabled, write

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.screenscraper.datamanager.DataManager

addData, addData, addData, addEventListener, addEventListenerWithPriority, clearAllData, clearData, getColumns, getCurrentDataNode, getDatabaseSchema, getNewDataNode, getRoot, getSchema, hasSchema, setDatabaseSchema

Field Detail

log

@NotNull
public final SqlDataManager.SqlDataManagerLogger log

The log object used by the data manager. It's public to allow access elsewhere

executor

@NotNull
final java.util.concurrent.ThreadPoolExecutor executor

session

@Nullable
final ScrapingSession session

Constructor Detail

SqlDataManager

@InternalOnly
public SqlDataManager(@NotNull
                                     javax.sql.DataSource dataSource)

Creates an SqlDataManager that uses the given data source to connect to the database

Parameters:

dataSource - The data source to use (cannot be null)

Throws:

java.lang.IllegalArgumentException - If the given data source is null

SqlDataManager

public SqlDataManager(@NotNull
                      javax.sql.DataSource dataSource,
                      @Nullable
                      ScrapingSession session)

Creates an SqlDataManager that uses the given data source to connect to the database

Parameters:

dataSource - The data source to use (cannot be null)

session - The scraping session the data manager will be running under

Throws:

java.lang.IllegalArgumentException - If the given data source is null

Method Detail

getDataSource

@NotNull
public DataManagerDataSource getDataSource()

Returns the DataSource for the object. This is the wrapped source the data manager uses internally so it can ensure all statements and result sets are closed properly when the connection is closed

Returns:

The DataSource for the object. This is the wrapped source the data manager uses internally

setDataSource

@NotNull
public DataManagerDataSource setDataSource(@NotNull
                                                     javax.sql.DataSource newDataSource)

Sets a new data source for the data manager. This is for advanced use (perhaps use one source to build schemas then swap out for a new source that's shared between multiple database AND is setup to automatically switch to the correct database when getting a connection)

Parameters:

newDataSource - The new data source to use

Returns:

The previous data source (so it can be closed or reused)

Since:

7.0.11a

enableConnectionMonitoring

public void enableConnectionMonitoring()

Tells this data manager to log out information about connections it has given out that have been open for a long time. This usually indicates a bug in the code

setLoggingLevelTrace

public void setLoggingLevelTrace()

Shortcut method for setting the log level to trace

setLoggingLevelDebug

public void setLoggingLevelDebug()

Shortcut method for setting the log level to debug

setLoggingLevelInfo

public void setLoggingLevelInfo()

Shortcut method for setting the log level to info

setLoggingLevelWarn

public final void setLoggingLevelWarn()

Shortcut method for setting the log level to warn

setLoggingLevelError

public void setLoggingLevelError()

Shortcut method for setting the log level to error

setLoggingLevel

public void setLoggingLevel(org.apache.log4j.Level level)

Sets the logging level

Parameters:

level - The new logging level

addSessionVariablesOnCommit

public void addSessionVariablesOnCommit(boolean val)

Sets whether or not to add session variables on commit

Parameters:

val - Whether or not to add session variables on commit

isSqlAutoCommit

public boolean isSqlAutoCommit()

Returns whether or not this data manager should use auto commit (each statement is it's own transaction)

Returns:

True if this data manager should use auto commit (each statement is it's own transaction)

setSqlAutoCommit

public void setSqlAutoCommit(boolean sqlAutoCommit)

Sets whether or not this data manager should use auto commit (each statement is it's own transaction)

Parameters:

sqlAutoCommit - True if this data manager should use auto commit (each statement is it's own transaction)

setActiveConnection

@InternalOnly
public void setActiveConnection(@Nullable
                                               java.sql.Connection connection)

Sets the connection currently writing to the database for the active thread

Parameters:

connection - The connection currently writing to the database for the active thread

getActiveConnection

@Nullable
public java.sql.Connection getActiveConnection()

Returns the connection currently writing to the database for the active thread

Returns:

The connection currently writing to the database for the active thread

clearThreadLocals

public void clearThreadLocals()

Clears all thread locals for the object

clearSessionVariablesOnCommit

public void clearSessionVariablesOnCommit(boolean val)

Sets whether or not to clear session variables on commit

Parameters:

val - Whether to clear values or not

clearSessionVariables

@InternalOnly
public void clearSessionVariables(@NotNull
                                                 java.lang.String table)

Clears session variables that match fields in the given table

Parameters:

table - The table for which to clear values

removeEventListener

public void removeEventListener(java.lang.String schema,
                                DataManagerEventSource.EventFireTime when,
                                DataManagerEventListener listener)

Removes an event listener

Overrides:

removeEventListener in class BasicDataManager

Parameters:

schema - The schema to remove the listener from

when - When the event would fire

listener - The listener that will be removed

getPrimaryKeys

@NotNull
 @InternalOnly
public java.util.Set<java.lang.String> getPrimaryKeys(java.lang.String table)

return a set of the columns which are primary keys defined in the SqlSchema

Parameters:

table - The name of the table

Returns:

A set of column names that form the primary key

addForeignKey

@NotNull
public SchemaForeignKey addForeignKey(@NotNull
                                                java.lang.String childTable,
                                                java.lang.String childColumn,
                                                java.lang.String parentTable,
                                                java.lang.String parentColumn)

Add a foreign key to the schemas. This is used if the underlying database engine does not support built in foreign key constraints, or if for some reason the tables do not use foreign key constraints but you want to relate data in the DataManager anyway

Parameters:

childTable - The name of the child table

childColumn - The column on the child table

parentTable - The name of the parent table

parentColumn - The name of the column on the parent table

Returns:

SchemaForeignKey that was generated and added to the child schema

setDateFormat

public void setDateFormat(@NotNull
                          java.lang.String table,
                          java.lang.String column,
                          java.text.SimpleDateFormat inputDateFormat)

Set the input date format for a date/time/timestamp column so that the database can convert a String input it into an appropriate value for the database

Parameters:

table - The name of the table

column - The field to set the format for

inputDateFormat - The date format for the given table / field

setDateFormat

public void setDateFormat(@NotNull
                          java.lang.String table,
                          java.lang.String column,
                          java.time.format.DateTimeFormatter inputDateFormat)

Set the input date format for a date/time/timestamp column so that the database can convert a String input it into an appropriate value for the database

Parameters:

table - The name of the table

column - The field to set the format for

inputDateFormat - The date format for the given table / field

getLastAutoIncrementValue

public DataObject getLastAutoIncrementValue(java.lang.String table)

returns the value of the autoincrement key for the table for the last commit, or null if none is set

Parameters:

table - The name of the table

Returns:

The last autoincrement value. This is from the last successful write, even if an error occurred after that write

buildSchemas

public void buildSchemas()

Builds the schemas from a database. This can take some time so if there are any tables in the database that are not required for the DataManager it is recommended to use buildSchemas(List<String> tables)

Specified by:

buildSchemas in interface DataManager

Overrides:

buildSchemas in class BasicDataManager

buildSchemas

public void buildSchemas(@Nullable
                         java.lang.String schemaName)

Builds the schemas from a database. This can take some time so if there are any tables in the database that are not required for the DataManager it is recommended to use buildSchemas(List)

Parameters:

schemaName - The name of the schema to parse, or null to parse all. For some database types this matters

buildSchemas

public void buildSchemas(boolean useOldSchemaIfKnown)

Builds the schemas from a database. This can take some time so if there are any tables in the database that are not required for the DataManager it is recommended to use buildSchemas(List)

Parameters:

useOldSchemaIfKnown - True to indicate we should use a schema we parsed before, if known. False to rebuild the schema regardless of whether or not it has been done before.

buildSchemas

public void buildSchemas(@Nullable
                         java.lang.String schemaName,
                         boolean useOldSchemaIfKnown)

Builds the schemas from a database. This can take some time so if there are any tables in the database that are not required for the DataManager it is recommended to use buildSchemas(List)

Parameters:

schemaName - The name of the schema to parse, or null to parse all. For some database types this matters

useOldSchemaIfKnown - True to indicate we should use a schema we parsed before, if known. False to rebuild the schema regardless of whether or not it has been done before.

buildSchemas

public void buildSchemas(@NotNull
                         java.util.List<java.lang.String> tables)
                  throws java.lang.Exception

Builds the schemas from a database including only tables

Parameters:

tables - A list of table names to build schemas for

Throws:

java.lang.Exception - On failure to build the schemas

buildSchemas

public void buildSchemas(@Nullable
                         java.lang.String schemaName,
                         @NotNull
                         java.util.List<java.lang.String> tables)
                  throws java.lang.Exception

Builds the schemas from a database including only tables

Parameters:

schemaName - The name of the schema to parse, or null to parse all. For some database types this matters

tables - A list of table names to build schemas for

Throws:

java.lang.Exception - On failure to build the schemas

getNewDataWriter

@NotNull
 @InternalOnly
public DataWriter getNewDataWriter()

Gets a new data writer for writing to the database

Overrides:

getNewDataWriter in class BasicDataManager

setLastAutoIncrementValue

public void setLastAutoIncrementValue(@NotNull
                                      java.lang.String schemaName,
                                      java.lang.Object value)

Manually sets the last auto increment value for a schema. Currently only supports setting the value manually if there's a single field for the primary key

Parameters:

schemaName - The name of the schema

value - The value to set as the last auto increment value

Since:

7.0.9a

setLastAutoIncrementValue

public void setLastAutoIncrementValue(@NotNull
                                      java.lang.String schemaName,
                                      DataObject value)

Manually sets the last auto increment value for a schema.

Parameters:

schemaName - The name of the schema

value - The value to set as the last auto increment value

Since:

7.0.9a

getSchemas

@NotNull
 @InternalOnly
public java.util.Set<RelationalSchema> getSchemas()

Helper method for backward compatibility to retrieve the relational (table) schemas

Returns:

A set of all the schemas available to the database

getSchemaName

@Nullable
public java.lang.String getSchemaName()

Returns the SchemaName for the object (this is largely for PostgreSQL where you can have a single database with multiple schemas, each with tables in them)

Returns:

The SchemaName for the object

setSchemaName

public void setSchemaName(@Nullable
                          java.lang.String schemaName)

Sets the SchemaName for the object (this is largely for PostgreSQL where you can have a single database with multiple schemas, each with tables in them)

Parameters:

schemaName - The new value for SchemaName

getConnection

public java.sql.Connection getConnection()
                                  throws java.sql.SQLException

get a direct connection to the database from the Connection pool for specific queries When the connection is done being used call close() on it to return it to the pool.

Returns:

Sql Connection

Throws:

java.sql.SQLException - If there is an error getting a connection

getConnection

public java.sql.Connection getConnection(java.lang.String name)
                                  throws java.sql.SQLException

get a direct connection to the database from the Connection pool for specific queries When the connection is done being used call close() on it to return it to the pool.

Parameters:

name - A name to help debug abandoned connections

Returns:

Sql Connection

Throws:

java.sql.SQLException - If no connection could be retrieved

setGlobalUpdateEnabled

public void setGlobalUpdateEnabled(boolean update)

Sets all the nodes to be able to overwrite columns in the database with the new values when there is a primary key match.

Parameters:

update - Whether updating should be enabled or not

setGlobalMergeEnabled

public void setGlobalMergeEnabled(boolean merge)

Sets all the nodes to be able to merge the values with the corresponding rows in the database when there is a primary key match.

Parameters:

merge - Whether merging should be enabled or not

checkDataNode

@InternalOnly
public boolean checkDataNode(@NotNull
                                            java.lang.String schema)

verifies that the data is ready to be inserted. If false is returned the cause can be retrieved via getMessage.

Specified by:

checkDataNode in interface DataManager

Overrides:

checkDataNode in class BasicDataManager

Parameters:

schema - The schema

Returns:

true if data meets constraints defined in the AbstractSchema

setAutoManyToMany

public void setAutoManyToMany(boolean enable)

toggles automatically generating many to many relationship nodes based on schema information. If a table has foreign keys to two or more tables, those tables have been added, and all other constraints are met, the many to many node will be generated and then written to the database when flush is called

Parameters:

enable - True to enable many to many relations being automatically detected.

setMultiThreadWrite

public void setMultiThreadWrite(int numThreads)

enables database writing to be done in its own thread. numThreads is the maximum number of threads that will be spawned. numThreads cannot exceed 100. See setSingleThreadWrite() to set no background threads. Calling that method overwrites what this method set.

Parameters:

numThreads - The number of threads used to write

getMaxBackgroundWriterThreads

public int getMaxBackgroundWriterThreads()

Returns the maximum number of background writer threads that will be used at any given time

Returns:

The maximum number of background writer threads that are allowed, or 0 to indicate this is a single thread writer

Since:

7.0.9a

setSingleThreadWrite

public void setSingleThreadWrite()

disables database writing to be done in its own thread. Writes will happen in the current thread instead of background threads (this is the default). See setMultiThreadWrite(int) to set background threads. Calling that method overwrites what this method set.

close

public void close()

Description copied from interface: DataManager

close should be called when you are done with the DataManager

Specified by:

close in interface DataManager

Overrides:

close in class BasicDataManager

setAutoCloseAfterScrapeEnds

public void setAutoCloseAfterScrapeEnds()

Tells the data manager to close when the scrape ends. Calling this method will close the data manager automatically when the scrape ends. If the session was stopped, it will clear the data that is set to be written, otherwise it will call flush before closing.

setAutoIncrementSessionNewRecords

public void setAutoIncrementSessionNewRecords(@NotNull
                                              java.lang.String schema,
                                              DataManagerEventSource.EventFireTime fireTime)

Increments the number of new records saved value on the session when the given fire time occurs for the given schema

Parameters:

schema - The schema

fireTime - The fire time

Since:

7.0.12a

setAutoIncrementSessionNewRecords

public void setAutoIncrementSessionNewRecords(@NotNull
                                              ScrapingSession session,
                                              @NotNull
                                              java.lang.String schema,
                                              DataManagerEventSource.EventFireTime fireTime)

Increments the number of new records saved value on the session when the given fire time occurs for the given schema

Parameters:

session - The session on which to increment records (useful for parent/child scrapes)

schema - The schema

fireTime - The fire time

Since:

7.0.12a

setAutoIncrementSessionDuplicateRecords

public void setAutoIncrementSessionDuplicateRecords(@NotNull
                                                    java.lang.String schema,
                                                    DataManagerEventSource.EventFireTime fireTime)

Increments the number of duplicate records saved value on the session when the given fire time occurs for the given schema

Parameters:

schema - The schema

fireTime - The fire time

Since:

7.0.12a

setAutoIncrementSessionDuplicateRecords

public void setAutoIncrementSessionDuplicateRecords(@NotNull
                                                    ScrapingSession session,
                                                    @NotNull
                                                    java.lang.String schema,
                                                    DataManagerEventSource.EventFireTime fireTime)

Increments the number of duplicate records saved value on the session when the given fire time occurs for the given schema

Parameters:

session - The session on which to increment records (useful for parent/child scrapes)

schema - The schema

fireTime - The fire time

Since:

7.0.12a

setAutoIncrementSessionUpdatedRecords

public void setAutoIncrementSessionUpdatedRecords(@NotNull
                                                  java.lang.String schema,
                                                  DataManagerEventSource.EventFireTime fireTime)

Increments the number of updated records saved value on the session when the given fire time occurs for the given schema

Parameters:

schema - The schema

fireTime - The fire time

Since:

7.0.12a

setAutoIncrementSessionUpdatedRecords

public void setAutoIncrementSessionUpdatedRecords(@NotNull
                                                  ScrapingSession session,
                                                  @NotNull
                                                  java.lang.String schema,
                                                  DataManagerEventSource.EventFireTime fireTime)

Increments the number of updated records saved value on the session when the given fire time occurs for the given schema

Parameters:

session - The session on which to increment records (useful for parent/child scrapes)

schema - The schema

fireTime - The fire time

Since:

7.0.12a

setAutoIncrementSessionUpdatedOrDuplicateRecordsByFields

public void setAutoIncrementSessionUpdatedOrDuplicateRecordsByFields(@NotNull
                                                                     java.lang.String schema,
                                                                     @NotNull
                                                                     java.lang.String... ignoreFieldsForDuplicateCheck)

Increments the number of updated or duplicate records saved value on the session when the a record is being written. Uses the ignore fields to determine if a write is an update or a duplicate (if only fields in ignore fields are being written, this is a duplicate, otherwise it is considered an update)

Parameters:

schema - The schema

ignoreFieldsForDuplicateCheck - The fields to ignore for duplicate counts. If only fields from the ignore fields are being written, this is considered to be a duplicate, otherwise it is an update

Since:

7.0.12a

setAutoIncrementSessionUpdatedOrDuplicateRecordsByFields

public void setAutoIncrementSessionUpdatedOrDuplicateRecordsByFields(@NotNull
                                                                     ScrapingSession session,
                                                                     @NotNull
                                                                     java.lang.String schema,
                                                                     @NotNull
                                                                     java.lang.String... ignoreFieldsForDuplicateCheck)

Increments the number of updated or duplicate records saved value on the session when the a record is being written. Uses the ignore fields to determine if a write is an update or a duplicate (if only fields in ignore fields are being written, this is a duplicate, otherwise it is considered an update)

Parameters:

session - The session on which to increment records (useful for parent/child scrapes)

schema - The schema

ignoreFieldsForDuplicateCheck - The fields to ignore for duplicate counts. If only fields from the ignore fields are being written, this is considered to be a duplicate, otherwise it is an update

Since:

7.0.12a

setBasicAutoIncrementRecordsByWriteType

public void setBasicAutoIncrementRecordsByWriteType(@NotNull
                                                    java.lang.String schema,
                                                    @NotNull
                                                    java.lang.String... ignoreFieldsForDuplicateCheck)

Sets up basic new/updated/duplicate/error record count increment handlers for the given schema.

Parameters:

schema - The schema

ignoreFieldsForDuplicateCheck - The fields to ignore for duplicate counts. If only fields from the ignore fields are being written, this is considered to be a duplicate, otherwise it is an update (for records being updated in the database)

Since:

7.0.12a

setAutoIncrementSessionErrorRecords

public void setAutoIncrementSessionErrorRecords(@NotNull
                                                java.lang.String schema,
                                                DataManagerEventSource.EventFireTime fireTime)

Increments the number of error records saved value on the session when the given fire time occurs for the given schema

Parameters:

schema - The schema

fireTime - The fire time

Since:

7.0.12a

setAutoIncrementSessionErrorRecords

public void setAutoIncrementSessionErrorRecords(@NotNull
                                                ScrapingSession session,
                                                @NotNull
                                                java.lang.String schema,
                                                DataManagerEventSource.EventFireTime fireTime)

Increments the number of error records saved value on the session when the given fire time occurs for the given schema

Parameters:

session - The session on which to increment records (useful for parent/child scrapes)

schema - The schema

fireTime - The fire time

Since:

7.0.12a

setBasicAutoIncrementRecordsByWriteType

public void setBasicAutoIncrementRecordsByWriteType(@NotNull
                                                    ScrapingSession session,
                                                    @NotNull
                                                    java.lang.String schema,
                                                    @NotNull
                                                    java.lang.String... ignoreFieldsForDuplicateCheck)

Sets up basic new/updated/duplicate/error record count increment handlers for the given schema.

Parameters:

session - The session on which to increment records (useful for parent/child scrapes)

schema - The schema

ignoreFieldsForDuplicateCheck - The fields to ignore for duplicate counts. If only fields from the ignore fields are being written, this is considered to be a duplicate, otherwise it is an update (for records being updated in the database)

Since:

7.0.12a

flush

public boolean flush()

writes the current data to the Database and clears out all the data.

Specified by:

flush in interface DataManager

Overrides:

flush in class BasicDataManager

Returns:

true if write is successful, if multithreaded writes are enabled it always returns true as there is no callback function.

awaitCompletionOfPendingWrites

public void awaitCompletionOfPendingWrites()
                                    throws java.lang.InterruptedException

Blocks until all pending and active writes have completed from all threads. Note this should not be called from inside an event on the data manager or it will cause a deadlock since the current thread can't complete writing while waiting for the write to complete!

Throws:

java.lang.InterruptedException - If the thread was interrupted while waiting

Since:

7.0.12a

awaitCompletionOfPendingWrites

public void awaitCompletionOfPendingWrites(long timeout,
                                           java.util.concurrent.TimeUnit unit)
                                    throws java.lang.InterruptedException,
                                           java.util.concurrent.TimeoutException

Blocks until all pending and active writes have completed from all threads. Note this should not be called from inside an event on the data manager or it will cause a deadlock until the timeout since the current thread can't complete writing while waiting for the write to complete!

Parameters:

timeout - The amount of time to wait (negative for infinite)

unit - The unit of the amount to wait

Throws:

java.lang.InterruptedException - If the thread was interrupted while waiting

java.util.concurrent.TimeoutException - If the time elapsed before the pending writes completed

Since:

7.0.12a

setManyToMany

protected void setManyToMany(@NotNull
                             DataNode dataNode)

If a many to many relationship between this node and another in the set of current datanodes is possible and doesn't already exist, that node will be created. This makes it so the end developer doesn't have to call an empty addData to create the relationship. If there are any other constraints that aren't met the node won't be added because an addData would be required anyway to complete the constraints.

Parameters:

dataNode - The node on which to set many to many

commit

public boolean commit(java.lang.String table)

commits the row of data corresponding to the schema, so the next time addData is called it will be in a new row.

Specified by:

commit in interface DataManager

Overrides:

commit in class BasicDataManager

Parameters:

table - The table to commit

Returns:

the value of checkDataNode called on the corresponding DataNode

flushSameThread

public boolean flushSameThread()

This method is for when you have multi-threaded database writes enabled, but you need certain writes to be inline, for example if you need to run a query afterwards that depends on the data you just wrote being current.

Returns:

True iff we were able to begin the write, false otherwise. This will return true even if a node fails to write but the write was attempted.

validateData

@InternalOnly
boolean validateData(@NotNull
                                    DataNode n)

keyConstraintsMet

@InternalOnly
public boolean keyConstraintsMet(@NotNull
                                                DataNode n)

keyConstraintsMet

@InternalOnly
public boolean keyConstraintsMet(@NotNull
                                                DataNode n,
                                                @Nullable
                                                java.lang.String column)

constraintMet

@InternalOnly
public boolean constraintMet(@NotNull
                                            DataNode n,
                                            @Nullable
                                            SchemaKey k)

checkNullValue

@InternalOnly
public boolean checkNullValue(@NotNull
                                             DataNode n,
                                             java.lang.String column)

fkConstraintsMet

@InternalOnly
public boolean fkConstraintsMet(@NotNull
                                               DataNode n)

checkColumnLength

@InternalOnly
public boolean checkColumnLength(@NotNull
                                                DataNode n,
                                                java.lang.String column)

enablePageAndContextCaching

@InternalOnly
 @Deprecated
public void enablePageAndContextCaching(@NotNull
                                                                     java.lang.String cacheTableName,
                                                                     @Nullable
                                                                     java.util.Collection<java.lang.String> scrapeableFilesToCache,
                                                                     boolean autoCacheDataRecords,
                                                                     boolean keepMostRecentPageOnly)

Deprecated.

Tells the data manager that you want to store a copy of the pages and context associated with the main records in a table. This method should only be called once. This method should only be called after the schema is built, and will use the schema to determine the other tables and relations needed for caching.

The cache tables should have the following schema (an exception will be thrown if they schema isn't correct)

Record cache table

Field descriptions

Field	Data Type
main_pk	Same as the unique single field the context will relate to, usually an unsigned autoincrement integer
hash	varchar (80)
name	varchar
type	tinyint (See CacheType for the values)
cache_time	timestamp
compression	tinyint
cached_data	blob

Additional Constraints

There should be a foreign key from main_pk to the primary key in the main record table.

There should also be a primary key on main_pk, name, hash, type although that is not required (but will make things nicer).

It is also recommended if you plan to run queries on the data that you have an index on type and name

Parameters:

cacheTableName - The name of the cache table. This should be the table name that will store the cached content as opposed to the name of the table where the data you are extracting will be stored.

scrapeableFilesToCache - A set of scrapeable file names that should be cached automatically for this scrape. These will be cached anytime data is added to the data manager AND the current scrapeable file is one of those named.

autoCacheDataRecords - True if anytime data is added (not committed) for the main table the current data record (if there is a dataRecord in the script) should be cached as well.

keepMostRecentPageOnly - True if only the most recent copy of a given cached page (by name). Basically this means if a page is cached, but one with the same name already exists, the new version will overwrite the old version

Throws:

java.lang.IllegalArgumentException - If the cache table doesn't exist in the parsed schemas

java.lang.IllegalStateException - If the cache table has an invalid format

getCacheLoader

@NotNull
 @InternalOnly
 @Deprecated
public CacheLoader getCacheLoader()

Deprecated.

Returns a cache loader that can be used to view the cached values associated with a record

Returns:

A cache loader that can be used to view the cached values associated with a record

getCacheLoader

@NotNull
 @InternalOnly
 @Deprecated
public CacheLoader getCacheLoader(@Nullable
                                                                          java.lang.String additionalWhereConstraints)

Deprecated.

Returns a cache loader that can be used to view the cached values associated with a record. This also allows you to pass in additional constraints for the where clause of the lookup queries.

Parameters:

additionalWhereConstraints - Additional where constraints for the lookup. This is useful if you are storing multiple sources in a single database. For example, if you have a scrape table, and then the main records have a foreign key to that table to indicate which scrape the data is actually from.

Returns:

A cache loader that can be used to view the cached values associated with a record

cacheContext

public void cacheContext()

Cache the current context, using the session to get all SEARCH_ values for the context

getCurrentActiveWriters

public int getCurrentActiveWriters()

Returns the current number of writing tasks. Note that this value may be incorrect by the time it is returned, and is for debugging purposes only

Returns:

The current number of writing tasks.

getMaxActiveWritersSoFar

public int getMaxActiveWritersSoFar()

Returns the maximum number of writing tasks that have occurred at any given time. Note that this value may be incorrect by the time it is returned, and is for debugging purposes only

Returns:

The maximum number of writing tasks that have occurred at any given time.

getQueuedWrites

public int getQueuedWrites()

Returns the current number of writing tasks that are in queue but not running. Note that this value may be incorrect by the time it is returned, and is for debugging purposes only

Returns:

The current number of writing tasks that are in queue but not running

getMaxQueuedWritesSoFar

public int getMaxQueuedWritesSoFar()

Returns the maximum number of writing tasks that have been in queue at any given time. Note that this value may be incorrect by the time it is returned, and is for debugging purposes only

Returns:

The maximum number of writing tasks that have been in queue at any given time.

getConnectionURL

@NotNull
public java.lang.String getConnectionURL()

Returns the URL used to connection to the database for this SqlDataManager, or the string "Unknown" if the url wasn't determinable

Returns:

The URL used to connection to the database for this SqlDataManager, or the string "Unknown" if the url wasn't determinable