public abstract class AbstractRetryPolicy extends java.lang.Object implements RetryPolicy
Modifier and Type | Field and Description |
---|---|
protected java.util.Map<java.lang.String,java.lang.Object> |
errorOutputMap
A map of failure results
|
protected int |
maxNumberOfRetries
The maximum number of attempts to retry before failing
|
protected boolean |
resetReferrerBeforeRescrape
Flag used to indicate whether or not the referrer should be reset if this policy shows an error on the page
|
protected boolean |
resetSessionVariablesBeforeRescrape
Flag used to indicate whether or not the session variables should be reset if this policy shows an error on the page
|
protected ScrapingSession |
scrapingSession
The scraping session this is running under
|
protected ScriptContext |
scriptContext
The script context, used to allow scrapeableFile to have a value in an executed script
|
protected ScrapeableFile |
theScrapeableFile
The scrapeable file to use
|
Modifier | Constructor and Description |
---|---|
protected |
AbstractRetryPolicy()
Default constructor, used to initialize the default values of all the variables
|
Modifier and Type | Method and Description |
---|---|
AbstractRetryPolicy |
duplicate()
Duplicates this retry policy, copying any needed values to the new AbstractRetryPolicy.
|
java.util.Map<java.lang.String,?> |
getErrorChecksMap()
Returns a map that can be used to output an error message to indicate what checks failed.
|
int |
getMaxRetryAttempts()
Return the maximum number of times this policy allows for a retry before terminating in an error
|
ScriptContext |
getScriptContext()
Get the current script context (as last set)
|
boolean |
isError()
Checks to see if the page loaded incorrectly
|
boolean |
resetReferrerBeforeRescrape()
Returns true if the referrer should be reset before attempting to rescrape the file, if there was an error.
|
boolean |
resetSessionVariablesBeforeRescrape()
Returns true if the session variables should be reset before attempting to rescrape the file, if there was an error.
|
void |
runOnAllAttemptsFailed()
This will be called if all the retry attempts for the scrapeable file failed.
|
void |
setMaxNumberOfRetries(int retries)
Sets the maximum number of retry attempts for this policy
|
void |
setResetReferrerBeforeRescrape(boolean resetReferrerBeforeRescrape)
Sets whether or not this policy requests session variables to be reset before a rescrape is attempted
|
void |
setResetSessionVariablesBeforeRescrape(boolean resetSessionVariablesBeforeRescrape)
Sets whether or not this policy requests session variables to be reset before a rescrape is attempted
|
void |
setScrapeableFile(ScrapeableFile scrapeableFile)
Sets the scrapeable file this policy will use to get the downloaded content
|
void |
setScrapingSession(ScrapingSession session)
Sets the scraping session for this policy to access, if need be
|
void |
setScriptContext(ScriptContext scriptContext)
Sets the current script context
|
boolean |
shouldLogErrors()
Returns true if errors should be logged to the log/web interface when they occur
|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
runOnError
protected int maxNumberOfRetries
@Nullable protected ScrapeableFile theScrapeableFile
@Nullable protected ScrapingSession scrapingSession
@Nullable protected java.util.Map<java.lang.String,java.lang.Object> errorOutputMap
protected boolean resetSessionVariablesBeforeRescrape
protected boolean resetReferrerBeforeRescrape
protected ScriptContext scriptContext
protected AbstractRetryPolicy()
@Nullable public AbstractRetryPolicy duplicate()
Note that the copy can share internal references if necessary for the functionality of the policy. For example, if the policy is tracking total number of failures
it may have a shared AtomicInteger
reference for counting, which is passed in to the duplicate policy by reference. Therefore, duplicate isn't necessarily
an independent duplicate. It should be noted though that a "duplicate" copy of the policy is used for each scrapeable file when called
public boolean isError() throws java.lang.Exception
RetryPolicy
isError
in interface RetryPolicy
java.lang.Exception
- If something goes wrong while executing this methodpublic void setScrapeableFile(ScrapeableFile scrapeableFile)
scrapeableFile
- The scrapeable file to usepublic void setScrapingSession(ScrapingSession session)
session
- The current scraping session@Nullable public java.util.Map<java.lang.String,?> getErrorChecksMap()
RetryPolicy
getErrorChecksMap
in interface RetryPolicy
public boolean resetSessionVariablesBeforeRescrape()
RetryPolicy
RetryPolicy.runOnError()
resetSessionVariablesBeforeRescrape
in interface RetryPolicy
public boolean resetReferrerBeforeRescrape()
RetryPolicy
RetryPolicy.runOnError()
resetReferrerBeforeRescrape
in interface RetryPolicy
public boolean shouldLogErrors()
RetryPolicy
shouldLogErrors
in interface RetryPolicy
public void setMaxNumberOfRetries(int retries)
retries
- How many times this policy will retry before failingpublic int getMaxRetryAttempts()
RetryPolicy
getMaxRetryAttempts
in interface RetryPolicy
public void setResetSessionVariablesBeforeRescrape(boolean resetSessionVariablesBeforeRescrape)
resetSessionVariablesBeforeRescrape
- True to reset session variables before a rescrapepublic void setResetReferrerBeforeRescrape(boolean resetReferrerBeforeRescrape)
resetReferrerBeforeRescrape
- True to reset the referrer before a rescrapepublic ScriptContext getScriptContext()
public void setScriptContext(ScriptContext scriptContext)
scriptContext
- The current script contextpublic void runOnAllAttemptsFailed() throws java.lang.Exception
RetryPolicy
RetryPolicy.runOnError()
will be called just before this, as it is called after each time the scrapeable file fails to load
correctly, including the last time it fails to load.
This should only contain code that handles the final error. Any proxy rotating, cookie clearing, etc... should generally be done in the RetryPolicy.runOnError()
method, especially since it will still be called after the final error.runOnAllAttemptsFailed
in interface RetryPolicy
java.lang.Exception
- If the contents of this method throw an exception in the child class