com.screenscraper.scraper

Interface RunningScrapingSession

All Known Implementing Classes:

ScrapingSession

public interface RunningScrapingSession

Field Summary

Fields

Modifier and Type	Field and Description
static org.apache.log4j.Logger	log Used for logging.

Method Summary

Modifier and Type	Method and Description
void	addToNumDuplicateRecordsScraped(int numRecordsScraped) Adds to the number of duplicate records scraped.
void	addToNumErrorRecordsScraped(int numRecordsScraped) Adds to the number of error records scraped.
void	addToNumNewRecordsScraped(int numRecordsScraped) Adds to the number of new records scraped.
void	addToNumRecordsScraped(int numRecordsScraped) Adds to the number of records scraped.
void	addToNumRecordsScraped(java.lang.Object numRecordsScraped) Adds to the number of records scraped.
void	addToNumUpdatedRecordsScraped(int numRecordsScraped) Adds to the number of updated records scraped.
void	addToVariable(java.lang.String variable, int value) Adds a value to a session variable.
void	breakpoint() Performs a breakpoint.
void	clearCookies() Clears all cookies.
boolean	downloadFile(java.lang.String urlString, java.lang.String fileName) A utility method for downloading a resource and writing it to a file.
void	executeScript(java.lang.String scriptName) Executes the named script.
long	getElapsedRunningTime() Gets the number of milliseconds that have passed since the scraping session started.
java.lang.String	getErrorMessage() Gets the error message that should be persisted to the database once the scraping session ends.
boolean	getFatalErrorOccurred() Gets the flag used to indicate whether or not a fatal error occurred while running this scraping session.
int	getNumDuplicateRecordsScraped() Gets the number of duplicate records scraped for the primary data set corresponding to this scraping session.
int	getNumErrorRecordsScraped() Gets the number of error records scraped for the primary data set corresponding to this scraping session.
int	getNumNewRecordsScraped() Gets the number of new records scraped for the primary data set corresponding to this scraping session.
int	getNumRecordsScraped() Gets the number of records scraped for the primary data set corresponding to this scraping session.
int	getNumScriptsOnStack() Gets the number of ScriptContext objects on the stack.
int	getNumUpdatedRecordsScraped() Gets the number of duplicate records scraped for the primary data set corresponding to this scraping session.
ScrapingSessionNotifiable	getScrapingSessionNotifiable() Gets the object the scraping session will use to notify its progress of.
java.lang.Object	getVariable(java.lang.String variable) Gets the value of a session variable.
void	log(java.lang.Object message) Logs a message for the session
void	logCurrentDateAndTime() Logs the current date and time.
void	logCurrentTime() Logs the current time.
void	logElapsedRunningTime() Logs the number of days, hours, minutes, and seconds from the time the scraping session started.
void	pause(long time) Indicates that the scraping session should pause for the given number of milliseconds.
void	popScriptContext() Pops the top ScriptContext object off the stack.
void	pushScriptContext(ScriptContext scriptContext) Pushes a ScriptContext object on to the stack.
void	saveVariables(java.lang.String fileToSaveTo) Writes out the current session variables to a file.
void	scrapeFile(java.lang.String scrapeableFileIdentifier) Queues up a ScrapeableFile to be scraped.
void	sendDataToClient(java.lang.String key, java.lang.Object value) Queues up an element of data to be sent to a client.
void	sendMail(java.lang.String subject, java.lang.String body, java.lang.String recipients, java.lang.String attachments, java.lang.String headers) Sends mail to a pre-specified SMTP server.
void	setCookie(java.lang.String domain, java.lang.String name, java.lang.String value) Allows a cookie to be set in the current session state.
void	setErrorMessage(java.lang.String errorMessage) Sets the error message that should be persisted to the database once the scraping session ends.
void	setFatalErrorOccurred(boolean fatalErrorOccurred) Sets the flag used to indicate whether or not a fatal error occurred while running this scraping session.
void	setNumDuplicateRecordsScraped(int numRecordsScraped) Sets the number of duplicate records scraped.
void	setNumErrorRecordsScraped(int numRecordsScraped) Sets the number of error records scraped.
void	setNumNewRecordsScraped(int numRecordsScraped) Sets the number of new records scraped.
void	setNumRecordsScraped(int numRecordsScraped) Sets the number of records scraped for the primary data set corresponding to this scraping session.
void	setNumRecordsScraped(java.lang.Object numRecordsScraped) Sets the number of records scraped for the primary data set corresponding to this scraping session.
void	setNumUpdatedRecordsScraped(int numRecordsScraped) Sets the number of duplicate records scraped.
void	setVariable(java.lang.String variable, java.lang.Object value) Sets the value of a session variable
void	stopScraping() Indicates that the scraping session should stop scraping as soon as possible.
void	stopScraping(StopScrapingCallType callType, java.lang.String explanation) Indicates that the scraping session should stop scraping as soon as possible.
void	terminateAllAnonymousProxies() Stops all of the currently running proxy instances

Field Detail

log

static final org.apache.log4j.Logger log

Used for logging.

Method Detail

getVariable

@Nullable
java.lang.Object getVariable(java.lang.String variable)

Gets the value of a session variable.

Parameters:

variable - The name of the variable

Returns:

The value of the variable.

setVariable

void setVariable(@NotNull
                 java.lang.String variable,
                 @Nullable
                 java.lang.Object value)

Sets the value of a session variable

Parameters:

variable - The variable.

value - The value of the variable.

addToVariable

void addToVariable(java.lang.String variable,
                   int value)
            throws java.lang.Exception

Adds a value to a session variable.

Parameters:

variable - The name of the session variable.

value - The value to add.

Throws:

java.lang.Exception

scrapeFile

void scrapeFile(java.lang.String scrapeableFileIdentifier)
         throws java.lang.Exception

Queues up a ScrapeableFile to be scraped.

Parameters:

scrapeableFileIdentifier - The identifier of the file to be scraped.

Throws:

java.lang.Exception

stopScraping

void stopScraping()

Indicates that the scraping session should stop scraping as soon as possible.

stopScraping

void stopScraping(@NotNull
                  StopScrapingCallType callType,
                  @Nullable
                  java.lang.String explanation)

Indicates that the scraping session should stop scraping as soon as possible.

Parameters:

callType - The type of stop being issued

explanation - An explanation string for the stop

log

void log(java.lang.Object message)

Logs a message for the session

Parameters:

message - The message to be logged.

pause

void pause(long time)
    throws java.lang.Exception

Indicates that the scraping session should pause for the given number of milliseconds.

Parameters:

time - The amount of time to pause, in milliseconds.

Throws:

java.lang.Exception

downloadFile

boolean downloadFile(java.lang.String urlString,
                     java.lang.String fileName)
              throws java.lang.Exception

A utility method for downloading a resource and writing it to a file.

Parameters:

urlString - The Uniform Resource Locator of the resource.

fileName - The full path that the downloaded resource should be written to. Example "C:\\downloaded.gif". The "\" character should be included in the path twice for each instance of it's use in order for it to be not treated as an escape character.

Throws:

java.lang.Exception

breakpoint

void breakpoint()
         throws java.lang.Exception

Performs a breakpoint.

Throws:

java.lang.Exception

sendMail

void sendMail(java.lang.String subject,
              java.lang.String body,
              java.lang.String recipients,
              java.lang.String attachments,
              java.lang.String headers)
       throws java.lang.Exception

Sends mail to a pre-specified SMTP server.

Parameters:

subject - The subject line of the e-mail.

body - The text of the body of the e-mail.

recipients - A comma separated list of recipients.

attachments - A comma separated list of local files to be attached.

headers - Additional headers one would like to add to the e-mail.

Throws:

java.lang.Exception

saveVariables

void saveVariables(java.lang.String fileToSaveTo)
            throws java.lang.Exception

Writes out the current session variables to a file.

Parameters:

fileToSaveTo - The file to save the variables to.

Throws:

java.lang.Exception

setCookie

void setCookie(java.lang.String domain,
               java.lang.String name,
               java.lang.String value)
        throws java.lang.Exception

Allows a cookie to be set in the current session state.

Parameters:

domain - The domain of the cookie.

name - The name of the cookie.

value - The value the cookie should hold.

Throws:

java.lang.Exception

clearCookies

void clearCookies()
           throws java.lang.Exception

Clears all cookies.

Throws:

java.lang.Exception

executeScript

void executeScript(java.lang.String scriptName)
            throws java.lang.Exception

Executes the named script.

Parameters:

scriptName - The name of the script to execute.

Throws:

java.lang.Exception

setNumRecordsScraped

void setNumRecordsScraped(java.lang.Object numRecordsScraped)
                   throws java.lang.Exception

Sets the number of records scraped for the primary data set corresponding to this scraping session.

Parameters:

numRecordsScraped - The number of records scraped.

Throws:

java.lang.Exception

addToNumRecordsScraped

void addToNumRecordsScraped(java.lang.Object numRecordsScraped)
                     throws java.lang.Exception

Adds to the number of records scraped.

Parameters:

numRecordsScraped - The number of records to add.

Throws:

java.lang.Exception

addToNumRecordsScraped

void addToNumRecordsScraped(int numRecordsScraped)
                     throws java.lang.Exception

Adds to the number of records scraped.

Parameters:

numRecordsScraped - The number of records scraped.

Throws:

java.lang.Exception

addToNumNewRecordsScraped

void addToNumNewRecordsScraped(int numRecordsScraped)

Adds to the number of new records scraped.

Parameters:

numRecordsScraped - The number of new records scraped to add.

addToNumDuplicateRecordsScraped

void addToNumDuplicateRecordsScraped(int numRecordsScraped)

Adds to the number of duplicate records scraped.

Parameters:

numRecordsScraped - The number of duplicate records scraped to add.

addToNumUpdatedRecordsScraped

void addToNumUpdatedRecordsScraped(int numRecordsScraped)

Adds to the number of updated records scraped.

Parameters:

numRecordsScraped - The number of updated records scraped to add.

Since:

7.0.12a

addToNumErrorRecordsScraped

void addToNumErrorRecordsScraped(int numRecordsScraped)

Adds to the number of error records scraped.

Parameters:

numRecordsScraped - The number of error records scraped to add.

setNumRecordsScraped

void setNumRecordsScraped(int numRecordsScraped)
                   throws java.lang.Exception

Sets the number of records scraped for the primary data set corresponding to this scraping session.

Parameters:

numRecordsScraped - The number of records scraped.

Throws:

java.lang.Exception

getFatalErrorOccurred

boolean getFatalErrorOccurred()
                       throws java.lang.Exception

Gets the flag used to indicate whether or not a fatal error occurred while running this scraping session. This is used in the web interface and to determine whether or not the scraping session should be stopped prematurely.

Returns:

A boolean.

Throws:

java.lang.Exception

setFatalErrorOccurred

void setFatalErrorOccurred(boolean fatalErrorOccurred)
                    throws java.lang.Exception

Sets the flag used to indicate whether or not a fatal error occurred while running this scraping session. This is used in the web interface and to determine whether or not the scraping session should be stopped prematurely.

Parameters:

fatalErrorOccurred - A boolean.

Throws:

java.lang.Exception

getErrorMessage

@NotNull
java.lang.String getErrorMessage()
                                    throws java.lang.Exception

Gets the error message that should be persisted to the database once the scraping session ends.

Returns:

One or more error messages.

Throws:

java.lang.Exception

setErrorMessage

void setErrorMessage(java.lang.String errorMessage)
              throws java.lang.Exception

Sets the error message that should be persisted to the database once the scraping session ends.

Parameters:

errorMessage - The error message to set.

Throws:

java.lang.Exception

getElapsedRunningTime

long getElapsedRunningTime()
                    throws java.lang.Exception

Gets the number of milliseconds that have passed since the scraping session started.

Returns:

The number of milliseconds.

Throws:

java.lang.Exception

logElapsedRunningTime

void logElapsedRunningTime()
                    throws java.lang.Exception

Logs the number of days, hours, minutes, and seconds from the time the scraping session started.

Throws:

java.lang.Exception

logCurrentTime

void logCurrentTime()
             throws java.lang.Exception

Logs the current time.

Throws:

java.lang.Exception

logCurrentDateAndTime

void logCurrentDateAndTime()
                    throws java.lang.Exception

Logs the current date and time.

Throws:

java.lang.Exception

getNumRecordsScraped

int getNumRecordsScraped()
                  throws java.lang.Exception

Gets the number of records scraped for the primary data set corresponding to this scraping session.

Returns:

The number of records.

Throws:

java.lang.Exception

setNumNewRecordsScraped

void setNumNewRecordsScraped(int numRecordsScraped)
                      throws java.lang.Exception

Sets the number of new records scraped.

Parameters:

numRecordsScraped - The number of new records scraped.

Throws:

java.lang.Exception

setNumDuplicateRecordsScraped

void setNumDuplicateRecordsScraped(int numRecordsScraped)
                            throws java.lang.Exception

Sets the number of duplicate records scraped.

Parameters:

numRecordsScraped - The number of duplicate records scraped.

Throws:

java.lang.Exception

setNumUpdatedRecordsScraped

void setNumUpdatedRecordsScraped(int numRecordsScraped)
                          throws java.lang.Exception

Sets the number of duplicate records scraped.

Parameters:

numRecordsScraped - The number of duplicate records scraped.

Throws:

java.lang.Exception

setNumErrorRecordsScraped

void setNumErrorRecordsScraped(int numRecordsScraped)
                        throws java.lang.Exception

Sets the number of error records scraped.

Parameters:

numRecordsScraped - The number of error records scraped.

Throws:

java.lang.Exception

getNumNewRecordsScraped

int getNumNewRecordsScraped()
                     throws java.lang.Exception

Gets the number of new records scraped for the primary data set corresponding to this scraping session.

Returns:

The number of records.

Throws:

java.lang.Exception

getNumDuplicateRecordsScraped

int getNumDuplicateRecordsScraped()
                           throws java.lang.Exception

Gets the number of duplicate records scraped for the primary data set corresponding to this scraping session.

Returns:

The number of records.

Throws:

java.lang.Exception

getNumUpdatedRecordsScraped

int getNumUpdatedRecordsScraped()
                         throws java.lang.Exception

Gets the number of duplicate records scraped for the primary data set corresponding to this scraping session.

Returns:

The number of records.

Throws:

java.lang.Exception

Since:

7.0.12a

getNumErrorRecordsScraped

int getNumErrorRecordsScraped()
                       throws java.lang.Exception

Gets the number of error records scraped for the primary data set corresponding to this scraping session.

Returns:

The number of records.

Throws:

java.lang.Exception

sendDataToClient

void sendDataToClient(java.lang.String key,
                      java.lang.Object value)
               throws java.lang.Exception

Queues up an element of data to be sent to a client.

Parameters:

key - The key identifier of the data.

value - The value portion of the data.

Throws:

java.lang.Exception

terminateAllAnonymousProxies

void terminateAllAnonymousProxies()
                           throws java.lang.Exception

Stops all of the currently running proxy instances

Throws:

java.lang.Exception

pushScriptContext

void pushScriptContext(ScriptContext scriptContext)

Pushes a ScriptContext object on to the stack.

Parameters:

scriptContext - A ScriptContext object.

popScriptContext

void popScriptContext()

Pops the top ScriptContext object off the stack.

getNumScriptsOnStack

int getNumScriptsOnStack()

Gets the number of ScriptContext objects on the stack.

Returns:

The number.

getScrapingSessionNotifiable

@Nullable
ScrapingSessionNotifiable getScrapingSessionNotifiable()

Gets the object the scraping session will use to notify its progress of.

Returns:

The object.