com.screenscraper.util.retry

Class StandardRetryPolicy

java.lang.Object

com.screenscraper.util.retry.AbstractRetryPolicy

com.screenscraper.util.retry.StandardRetryPolicy

All Implemented Interfaces:

RetryPolicy

Direct Known Subclasses:

MultiRegexRetryPolicy

public class StandardRetryPolicy
extends AbstractRetryPolicy

Standard retry policy with a single regex to match and to not match

Field Summary

Fields inherited from class com.screenscraper.util.retry.AbstractRetryPolicy

errorOutputMap, maxNumberOfRetries, resetReferrerBeforeRescrape, resetSessionVariablesBeforeRescrape, scrapingSession, scriptContext, theScrapeableFile

Constructor Summary

Constructors

Constructor and Description
StandardRetryPolicy() Build a StandardRetryPolicy that doesn't do anything except retry the page

Method Summary

Modifier and Type	Method and Description
AbstractRetryPolicy	duplicate() Duplicates this retry policy, copying any needed values to the new AbstractRetryPolicy.
boolean	isError() Checks to see if the page loaded incorrectly
void	runOnAllAttemptsFailed() This will be called if all the retry attempts for the scrapeable file failed.
void	runOnError() Runs this code when the page had an error.
void	setCheckWasErrorOnRequest(boolean checkWasErrorOnRequest) Sets whether or not we should check if there was an error on the request by status code, of content length mismatch
void	setEndScrapeOnFailure(boolean endScrapeOnFailure) Sets whether or not the policy should stop the scrape if it fails to get a good page after the maximum number of retries is reached
void	setEndScrapeOnFailureCount(int numFailuresBeforeEnd) Sets whether or not the policy should stop the scrape if it fails to get a good page after the maximum number of retries is reached.
void	setEndScrapeOnScrapeableFileFailureCount(java.lang.String scrapeableFileName, int numFailuresBeforeEnd) Sets whether or not the policy should stop the scrape if it fails to get a good page after the maximum number of retries is reached.
void	setExtractorsMustMatch(boolean extractorsMustMatch) Sets whether or not extractor patterns must match for the page to be considered successfully loaded
void	setRunOnFail(java.lang.Runnable runnable) Sets a runnable that will run if the file has an error
void	setScriptOnFail(java.lang.String script) Sets the script that will run if there is an error on the page
void	setShouldFailPattern(java.lang.String pattern) Sets a Regex that shouldn't match.
void	setShouldMatchPattern(java.lang.String pattern) Sets a Regex that should match.
java.lang.String	toString()

Methods inherited from class com.screenscraper.util.retry.AbstractRetryPolicy

getErrorChecksMap, getMaxRetryAttempts, getScriptContext, resetReferrerBeforeRescrape, resetSessionVariablesBeforeRescrape, setMaxNumberOfRetries, setResetReferrerBeforeRescrape, setResetSessionVariablesBeforeRescrape, setScrapeableFile, setScrapingSession, setScriptContext, shouldLogErrors

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

StandardRetryPolicy

public StandardRetryPolicy()

Build a StandardRetryPolicy that doesn't do anything except retry the page

Method Detail

setScriptOnFail

public void setScriptOnFail(java.lang.String script)

Sets the script that will run if there is an error on the page

Parameters:

script - The name of the script to run

setRunOnFail

public void setRunOnFail(java.lang.Runnable runnable)

Sets a runnable that will run if the file has an error

Parameters:

runnable - The runnable to run on error

setShouldMatchPattern

public void setShouldMatchPattern(@NotNull
                                  java.lang.String pattern)

Sets a Regex that should match. If it fails to match, there is an error on the page

Parameters:

pattern - The Regex Pattern

setShouldFailPattern

public void setShouldFailPattern(@NotNull
                                 java.lang.String pattern)

Sets a Regex that shouldn't match. If it does, that signals an error

Parameters:

pattern - The Regex pattern

setExtractorsMustMatch

public void setExtractorsMustMatch(boolean extractorsMustMatch)

Sets whether or not extractor patterns must match for the page to be considered successfully loaded

Parameters:

extractorsMustMatch - True if extractors must match, false otherwise

setCheckWasErrorOnRequest

public void setCheckWasErrorOnRequest(boolean checkWasErrorOnRequest)

Sets whether or not we should check if there was an error on the request by status code, of content length mismatch

Parameters:

checkWasErrorOnRequest - True is we should check for an error on request, false otherwise

setEndScrapeOnFailure

public void setEndScrapeOnFailure(boolean endScrapeOnFailure)

Sets whether or not the policy should stop the scrape if it fails to get a good page after the maximum number of retries is reached

Parameters:

endScrapeOnFailure - True if the scrape should be stopped when the policy fails to fix the problem, false otherwise

setEndScrapeOnFailureCount

public void setEndScrapeOnFailureCount(int numFailuresBeforeEnd)

Sets whether or not the policy should stop the scrape if it fails to get a good page after the maximum number of retries is reached. This is used to set the number of failures allowed (of value of 1 means on the first failure end the scrape, 2 means not until the 2nd failure, etc...)

Parameters:

numFailuresBeforeEnd - The number of times the policy can fail to fix the issue before stopping the scrape (

Since:

7.0.9a

setEndScrapeOnScrapeableFileFailureCount

public void setEndScrapeOnScrapeableFileFailureCount(java.lang.String scrapeableFileName,
                                                     int numFailuresBeforeEnd)

Sets whether or not the policy should stop the scrape if it fails to get a good page after the maximum number of retries is reached. This is used to set the number of failures allowed (of value of 1 means on the first failure end the scrape, 2 means not until the 2nd failure, etc...). Note if a scrapeable file has a value (greater than 0) then this check is used instead of the one set by setEndScrapeOnFailureCount(int)

Parameters:

scrapeableFileName - The name of the scrapeable file

numFailuresBeforeEnd - The number of times this scrapeable file can error before the scrape is stopped (note a value of 0 or less does nothing)

Since:

7.0.9a

isError

public boolean isError()
                throws java.lang.Exception

Description copied from interface: RetryPolicy

Checks to see if the page loaded incorrectly

Specified by:

isError in interface RetryPolicy

Overrides:

isError in class AbstractRetryPolicy

Returns:

True on errors, false otherwise

Throws:

java.lang.Exception - If something goes wrong while executing this method

runOnError

public void runOnError()

Description copied from interface: RetryPolicy

Runs this code when the page had an error. This could include things such as rotating the proxy. This will be run AFTER resetting the referrer and any session variables. In this way you can manually adjust values if needed but not have the side effects from extractors that may have modified values from this scrapeable file

duplicate

public AbstractRetryPolicy duplicate()

Description copied from class: AbstractRetryPolicy

Duplicates this retry policy, copying any needed values to the new AbstractRetryPolicy. Duplicating a retry policy allows for safe multithreading. Returns null if the policy should not be duplicated for whatever reason.

Note that the copy can share internal references if necessary for the functionality of the policy. For example, if the policy is tracking total number of failures it may have a shared AtomicInteger reference for counting, which is passed in to the duplicate policy by reference. Therefore, duplicate isn't necessarily an independent duplicate. It should be noted though that a "duplicate" copy of the policy is used for each scrapeable file when called

Overrides:

duplicate in class AbstractRetryPolicy

Returns:

A new retry policy that performs the same functions as the current policy, or null if by nature of the policy it cannot be duplicated.

toString

@NotNull
public java.lang.String toString()

Overrides:

toString in class java.lang.Object

runOnAllAttemptsFailed

public void runOnAllAttemptsFailed()

Description copied from interface: RetryPolicy

This will be called if all the retry attempts for the scrapeable file failed. In other words, if the policy said to retry 25 times, after 25 failures this method will be called. Note that RetryPolicy.runOnError() will be called just before this, as it is called after each time the scrapeable file fails to load correctly, including the last time it fails to load.

This should only contain code that handles the final error. Any proxy rotating, cookie clearing, etc... should generally be done in the RetryPolicy.runOnError() method, especially since it will still be called after the final error.

Specified by:

runOnAllAttemptsFailed in interface RetryPolicy

Overrides:

runOnAllAttemptsFailed in class AbstractRetryPolicy