com.screenscraper.util.retry

Class MultiplePolicyRetryPolicy

java.lang.Object

com.screenscraper.util.retry.AbstractRetryPolicy

com.screenscraper.util.retry.MultiplePolicyRetryPolicy

All Implemented Interfaces:

RetryPolicy

public class MultiplePolicyRetryPolicy
extends AbstractRetryPolicy

Retry policy wrapper class. This is designed to make it easy to use multiple retry policies without having to create an entirely new policy from scratch. It allows for easily modifying an existing policy by adding additional constraints and/or actions to perform.

Version:

1.0

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	MultiplePolicyRetryPolicy.Operator

Field Summary

Fields inherited from class com.screenscraper.util.retry.AbstractRetryPolicy

errorOutputMap, maxNumberOfRetries, resetReferrerBeforeRescrape, resetSessionVariablesBeforeRescrape, scrapingSession, scriptContext, theScrapeableFile

Constructor Summary

Constructors

Constructor and Description
MultiplePolicyRetryPolicy(MultiplePolicyRetryPolicy.Operator operator, boolean forceRunAll, java.util.List<RetryPolicy> policies) Creates a retry policy based on the policies in the given list.
MultiplePolicyRetryPolicy(MultiplePolicyRetryPolicy.Operator operator, boolean forceRunAll, RetryPolicy... policies) Creates a retry policy based on the policies in the given array.
MultiplePolicyRetryPolicy(MultiplePolicyRetryPolicy.Operator operator, java.util.List<RetryPolicy> policies) Creates a retry policy based on the policies in the given list.
MultiplePolicyRetryPolicy(MultiplePolicyRetryPolicy.Operator operator, RetryPolicy... policies) Creates a retry policy based on the policies in the given array.

Method Summary

Modifier and Type	Method and Description
AbstractRetryPolicy	duplicate() Duplicates this retry policy, copying any needed values to the new AbstractRetryPolicy.
int	getMaxRetryAttempts() This method will return the minimum of any of it's contained policies' retry counts Return the maximum number of times this policy allows for a retry before terminating in an error
boolean	isError() Checks to see if the page loaded incorrectly
boolean	resetReferrerBeforeRescrape() Returns true if the referrer should be reset before attempting to rescrape the file, if there was an error.
boolean	resetSessionVariablesBeforeRescrape() Returns true if the session variables should be reset before attempting to rescrape the file, if there was an error.
void	runOnAllAttemptsFailed() This will be called if all the retry attempts for the scrapeable file failed.
void	runOnError() Runs this code when the page had an error.
void	setScrapeableFile(ScrapeableFile scrapeableFile) Sets the scrapeable file this policy will use to get the downloaded content
void	setScrapingSession(ScrapingSession session) Sets the scraping session for this policy to access, if need be
boolean	shouldLogErrors() Returns true if errors should be logged to the log/web interface when they occur

Methods inherited from class com.screenscraper.util.retry.AbstractRetryPolicy

getErrorChecksMap, getScriptContext, setMaxNumberOfRetries, setResetReferrerBeforeRescrape, setResetSessionVariablesBeforeRescrape, setScriptContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

MultiplePolicyRetryPolicy

public MultiplePolicyRetryPolicy(MultiplePolicyRetryPolicy.Operator operator,
                                 RetryPolicy... policies)

Creates a retry policy based on the policies in the given array. Policies will be applied in the order given

Parameters:

operator - The operator used to combine the error checks from each policy

policies - The policies to use together

MultiplePolicyRetryPolicy

public MultiplePolicyRetryPolicy(MultiplePolicyRetryPolicy.Operator operator,
                                 boolean forceRunAll,
                                 RetryPolicy... policies)

Creates a retry policy based on the policies in the given array. Policies will be applied in the order given

Parameters:

operator - The operator used to combine the error checks from each policy

forceRunAll - True if an error by this policy should run the error handling code for all sub policies, false otherwise

policies - The policies to use together

MultiplePolicyRetryPolicy

public MultiplePolicyRetryPolicy(MultiplePolicyRetryPolicy.Operator operator,
                                 java.util.List<RetryPolicy> policies)

Creates a retry policy based on the policies in the given list. Policies will be applied in the order given

Parameters:

operator - The operator used to combine the error checks from each policy

policies - The policies to use together

MultiplePolicyRetryPolicy

public MultiplePolicyRetryPolicy(MultiplePolicyRetryPolicy.Operator operator,
                                 boolean forceRunAll,
                                 @Nullable
                                 java.util.List<RetryPolicy> policies)

Creates a retry policy based on the policies in the given list. Policies will be applied in the order given

Parameters:

operator - The operator used to combine the error checks from each policy

forceRunAll - True if an error by this policy should run the error handling code for all sub policies, false otherwise

policies - The policies to use together

Method Detail

setScrapeableFile

public void setScrapeableFile(ScrapeableFile scrapeableFile)

Description copied from class: AbstractRetryPolicy

Sets the scrapeable file this policy will use to get the downloaded content

Overrides:

setScrapeableFile in class AbstractRetryPolicy

Parameters:

scrapeableFile - The scrapeable file to use

setScrapingSession

public void setScrapingSession(ScrapingSession session)

Description copied from class: AbstractRetryPolicy

Sets the scraping session for this policy to access, if need be

Overrides:

setScrapingSession in class AbstractRetryPolicy

Parameters:

session - The current scraping session

isError

public boolean isError()
                throws java.lang.Exception

Description copied from interface: RetryPolicy

Checks to see if the page loaded incorrectly

Specified by:

isError in interface RetryPolicy

Overrides:

isError in class AbstractRetryPolicy

Returns:

True on errors, false otherwise

Throws:

java.lang.Exception - If something goes wrong while executing this method

runOnError

public void runOnError()
                throws java.lang.Exception

Description copied from interface: RetryPolicy

Runs this code when the page had an error. This could include things such as rotating the proxy. This will be run AFTER resetting the referrer and any session variables. In this way you can manually adjust values if needed but not have the side effects from extractors that may have modified values from this scrapeable file

Throws:

java.lang.Exception - If something goes wrong while executing this method

resetSessionVariablesBeforeRescrape

public boolean resetSessionVariablesBeforeRescrape()

Description copied from interface: RetryPolicy

Returns true if the session variables should be reset before attempting to rescrape the file, if there was an error. This can be useful especially if extractors null session variables when they don't match, but the value is needed to rescrape the file. If this returns true, the session variables will be reset BEFORE calling RetryPolicy.runOnError()

Specified by:

resetSessionVariablesBeforeRescrape in interface RetryPolicy

Overrides:

resetSessionVariablesBeforeRescrape in class AbstractRetryPolicy

Returns:

True if session variables should be reset if there was an error, false otherwise.

resetReferrerBeforeRescrape

public boolean resetReferrerBeforeRescrape()

Description copied from interface: RetryPolicy

Returns true if the referrer should be reset before attempting to rescrape the file, if there was an error. This can be useful to reset so the referrer doesn't show the page you just requested. If this returns true, the referrer will be reset BEFORE calling RetryPolicy.runOnError()

Specified by:

resetReferrerBeforeRescrape in interface RetryPolicy

Overrides:

resetReferrerBeforeRescrape in class AbstractRetryPolicy

Returns:

True if the referrer should be reset if there was an error, false otherwise.

shouldLogErrors

public boolean shouldLogErrors()

Description copied from interface: RetryPolicy

Returns true if errors should be logged to the log/web interface when they occur

Specified by:

shouldLogErrors in interface RetryPolicy

Overrides:

shouldLogErrors in class AbstractRetryPolicy

Returns:

True if errors should be logged to the log/web interface when they occur

getMaxRetryAttempts

public int getMaxRetryAttempts()

This method will return the minimum of any of it's contained policies' retry counts Return the maximum number of times this policy allows for a retry before terminating in an error

Specified by:

getMaxRetryAttempts in interface RetryPolicy

Overrides:

getMaxRetryAttempts in class AbstractRetryPolicy

Returns:

The maximum number of times to allow the ScrapeableFile to be rescraped before resulting in an error

runOnAllAttemptsFailed

public void runOnAllAttemptsFailed()
                            throws java.lang.Exception

This will be called if all the retry attempts for the scrapeable file failed. In other words, if the policy said to retry 25 times, after 25 failures this method will be called. Note that RetryPolicy.runOnError() will be called just before this, as it is called after each time the scrapeable file fails to load correctly, including the last time it fails to load.

This should only contain code that handles the final error. Any proxy rotating, cookie clearing, etc... should generally be done in the RetryPolicy.runOnError() method, especially since it will still be called after the final error. This method will only call the runOnAllAttemptsFailed method for policies that show an error on the current check.

Specified by:

runOnAllAttemptsFailed in interface RetryPolicy

Overrides:

runOnAllAttemptsFailed in class AbstractRetryPolicy

Throws:

java.lang.Exception

duplicate

public AbstractRetryPolicy duplicate()

Description copied from class: AbstractRetryPolicy

Duplicates this retry policy, copying any needed values to the new AbstractRetryPolicy. Duplicating a retry policy allows for safe multithreading. Returns null if the policy should not be duplicated for whatever reason.

Note that the copy can share internal references if necessary for the functionality of the policy. For example, if the policy is tracking total number of failures it may have a shared AtomicInteger reference for counting, which is passed in to the duplicate policy by reference. Therefore, duplicate isn't necessarily an independent duplicate. It should be noted though that a "duplicate" copy of the policy is used for each scrapeable file when called

Overrides:

duplicate in class AbstractRetryPolicy

Returns:

A new retry policy that performs the same functions as the current policy, or null if by nature of the policy it cannot be duplicated.