ScrapeableFile

java.lang.Object
- com.screenscraper.scraper.ScrapeableFile

All Implemented Interfaces:: Deleteable, Uniqueable, RunningScrapeableFile, ScriptInstancesContainer, java.lang.Comparable

public class ScrapeableFile
extends java.lang.Object
implements ScriptInstancesContainer, Deleteable, RunningScrapeableFile, java.lang.Comparable, Uniqueable

Nested Class Summary

Nested Classes
Modifier and Type Class and Description

static class ScrapeableFile.RequestType
Types of requests a scrapeable file can issue

Nested Classes
Modifier and Type	Class and Description
`static class`	`ScrapeableFile.RequestType` Types of requests a scrapeable file can issue

Field Summary

Fields
Modifier and Type Field and Description

(package private) boolean extractorPatternsMatched

static org.apache.log4j.Logger log
Used for logging.

Fields
Modifier and Type	Field and Description
`(package private) boolean`	`extractorPatternsMatched`
`static org.apache.log4j.Logger`	`log` Used for logging.

Constructor Summary

Constructors
Constructor and Description
`ScrapeableFile()` Needed by castor.
`ScrapeableFile(int scrapeableFileID)` Generates and loads a `ScrapeableFile` based on the ID.
`ScrapeableFile(int scrapeableFileID, ScrapingSession scrapingSession)` Generates a new ScrapeableFile.
`ScrapeableFile(ScrapeableFile scrapeableFile)` Generates a new scrapeable file based on the given scrapeable file.
`ScrapeableFile(java.lang.String name)` Generates a new ScrapeableFile.
`ScrapeableFile(java.lang.String name, HTTPTransaction httpTransaction)` Generates a new ScrapeableFile with the given name, based on the given `HTTPTransaction`.
`ScrapeableFile(java.lang.String name, java.lang.String url, int sequence)` Generates a new ScrapeableFile.
`ScrapeableFile(java.lang.String name, java.net.URL url)` Generates a new ScrapeableFile.
`ScrapeableFile(java.lang.String name, java.net.URL url, java.lang.String postData)` Generates a new ScrapeableFile.

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`void`	`addBasicHeaders()` Adds request headers to the custom headers, so they can be removed
`void`	`addExtractorPattern(ExtractorPattern extractorPattern)` Adds a pattern to the scrapeable file.
`void`	`addGETHTTPParameter(java.lang.String key, java.lang.String value)` Adds a GET HTTP parameter with the given parameters to the end of the existing parameters.
`void`	`addGETHTTPParameter(java.lang.String key, java.lang.String value, int sequence)` Adds a GET HTTP parameter with the given parameters.
`void`	`addHTTPHeader(java.lang.String key, java.lang.String value)` Adds a custom header.
`void`	`addHTTPParameter(HTTPParameter httpParameter)` Adds a parameter to the scrapeable file.
`void`	`addNavigationAction(NavigationAction navigationAction)` Adds a `NavigationCriteria` to the set.
`void`	`addPOSTHTTPParameter(java.lang.String key, java.lang.String value)` Adds a POST HTTP parameter with the given parameters to the end of the existing parameters.
`void`	`addPOSTHTTPParameter(java.lang.String key, java.lang.String value, int sequence)` Adds a POST HTTP parameter with the given parameters.
`protected void`	`addRedirectURL(java.lang.String url)` Adds a redirect URL to the list.
`XmlNode`	`applyXPathExpression(java.lang.String expression)` Applies an XPath expression to the current response.
`Form`	`buildForm(java.lang.String formText)` Builds a form using the text given, which should contain all the form values (including the form open and close tags)
`void`	`bumpHTTPParameterSequences(int bottomSequence, int topSequence, int bumpModifier)` Bumps sequences up or down between the bottom and top sequences.
`int`	`compareTo(java.lang.Object scrapeableFile)` Required by the `Comparable` interface.
`boolean`	`containsFileUpload()` Indicates whether or not the file contains POST parameters.
`void`	`delete()` Deletes the scrapeable file from the database.
`void`	`dumpHTTPParametersToLog()` Dumps all of the HTTP parameters to the log.
`void`	`dumpToLog()` Dumps the scrapeable file to the log.
`protected void`	`ensureNavigationActionsHaveScrapeableFileReference()` Ensures that all of the navigation actions held by this object have a reference to the object.
`boolean`	`equals(java.lang.Object object)` Overriding equals...
`DataSet`	`extractData(ScriptContext scriptContext, ExtractorPattern extractorPattern, DataSet dataSet, boolean scripts)` Extracts data by applying the given `ExtractorPattern` to the supplied text.
`DataSet`	`extractData(ScriptContext scriptContext, ExtractorPattern extractorPattern, DataSet dataSet, boolean scripts, boolean isApplyPattern)` Extract Data from an Extracter Pattern Handle any mapping, resolving of url's or saving of session variables that need to be done
`DataSet`	`extractData(java.lang.String text, java.lang.String name)` Manually extracts data using an existing `ExtractorPattern`.
`java.lang.String`	`extractOneValue(java.lang.String text, java.lang.String name)` Manually extracts one value from data using an existing `ExtractorPattern`.
`java.lang.String`	`extractOneValue(java.lang.String text, java.lang.String name, java.lang.String token)` Manually extracts one value from data using an existing `ExtractorPattern`.
`void`	`flagAsLoaded()` Flags it as being fully loaded.
`DataRecord`	`getASPXValues(boolean onlyStandard)` Gets the ASPX .NET values from the string.
`boolean`	`getAuthenticationPreemptive()` Indicates whether or not authentication should be done preemptively.
`java.lang.String`	`getBASICAuthenticationPassword()` Gets the BASIC authentication password of the file to be scraped.
`java.lang.String`	`getBASICAuthenticationUsername()` Gets the BASIC authentication username of the file to be scraped.
`java.lang.String`	`getCharacterSet()` Gets the character encoding of the content given by the server.
`java.lang.String`	`getContentAsString()` Gets the scraped content as a string.
`java.lang.String`	`getContentBodyOnly()` Gets only the body of the response (no headers or status line) as a string.
`java.lang.String`	`getContentType()` Gets the custom "Content-Type" header to be sent.
`java.lang.String`	`getCurrentPOSTData()` Gets the current value of the POST data.
`java.lang.String`	`getCurrentURL()` Depending on when this value is requested it will return either the unresolved or resolved URL of the file to be scraped.
`java.util.Set<ScrapingHttpHeader>`	`getCustomHeaders()` Gets the custom headers hash set (not a copy of it)
`boolean`	`getEnableJavaScript()` Indicates whether or not JavaScript embedded in the HTML page should be processed.
`ExtractorPattern`	`getExtractorPattern(int sequence)` Gets an extractor pattern from the `ScrapeableFile`.
`ExtractorPattern`	`getExtractorPattern(java.lang.String name)` Gets an extractor pattern from the `ScrapeableFile`.
`ExtractorPattern`	`getExtractorPatternByID(int extractorPatternID)` Gets the extractor pattern by its ID
`java.util.HashSet`	`getExtractorPatterns()` Returns all extractor patterns held by this scrapeable file.
`java.util.Iterator`	`getExtractorPatternsIterator()` Gets an `Iterator` that can be used to cycle through the `ExtractorPattern` objects held by this object.
`boolean`	`getExtractorPatternTimedOut()` Indicates whether or not the extractor pattern timed out on the most recent try.
`boolean`	`getForceMultiPart()` Indicates whether or not a multi-part request should be forced.
`boolean`	`getForceNonBinary()` Indicates whether or not the contents of this response should be forced to be treated as non-binary.
`boolean`	`getForcePOST()` Indicates whether or not a POST request should be forced.
`com.gargoylesoftware.htmlunit.html.HtmlPage`	`getHtmlPage()` Gets an `HtmlPage` first by attempting to see if one was already given from the last request.
`HTTPParameter`	`getHTTPParameter(int sequence)` Gets an HTTP parameter from the session.
`HTTPParameter`	`getHTTPParameter(java.lang.String key)` Gets an HTTP parameter from the session.
`HTTPParameter`	`getHTTPParameterByName(java.lang.String key)` Gets an HTTP parameter by its name.
`java.util.HashSet<HTTPParameter>`	`getHTTPParameters()` Gets all of the HTTP parameters held by this scrapeable file.
`java.lang.String`	`getHTTPResponseHeader(java.lang.String header)` Gets the value of the header in the response of the scrapeable file, or returns null if it couldn't be found
`java.util.Map<java.lang.String,java.lang.String>`	`getHTTPResponseHeaders()` Gets the headers of the HTTP Response as a map, and returns them.
`java.lang.String`	`getHTTPResponseHeaderSection()` Gets the header section of the HTTP Response
`int`	`getHTTPTransactionID()` Gets the ID of the HTTP transaction associated with this scrapeable file.
`int`	`getID()` Gets the ID of the scrapeable file.
`java.lang.String`	`getIdentifier()` Gets the identifier for this object.
`ScrapingResponse`	`getLastHttpResponse()` Gets the last http response received
`java.lang.String`	`getLastRequest()` Gets the request from the last time it was scraped.
`java.lang.String`	`getLastRequestShort()` Returns the last request, truncating it if it's too long.
`java.net.URL`	`getLastRequestURL()` Gets the last request URL used for this scrapeable file
`java.lang.String`	`getLastScrapedData()` Gets the data from the file from the last time it was scraped.
`java.lang.String`	`getLastScrapedDataShort()` Gets the data from the file from the last time it was scraped, truncating it, if necessary.
`boolean`	`getLastTidyAttemptFailed()` Indicates whether or not the last attempt to tidy the response failed.
`boolean`	`getMaxRequestAttemptsReached()` Indicates whether or not the maximum requests attempts were reached on the most recent try.
`int`	`getMaxResponseLength()` Gets the maximum response length, in kilobytes.
`java.lang.String`	`getName()` Gets the name.
`NavigationAction`	`getNavigationAction(int sequence)` Gets a navigation action from the `ScrapeableFile`.
`protected NavigationActionDoer`	`getNavigationActionDoer()` Gets the current navigation action doer to be processed.
`java.util.TreeSet`	`getNavigationActions()` Gets a reference to the `TreeSet` containing the `NavigationAction` objects.
`java.util.Iterator`	`getNavigationActionsIterator()` Gets an `Iterator` that can be used to cycle through the `NavigationAction` objects held by this object.
`javax.swing.tree.DefaultMutableTreeNode`	`getNode()` Gets the node representing this scraping file.
`java.lang.String`	`getNonTidiedHTML()` Gets the non-tidied HTML.
`java.lang.String`	`getNTLMAuthenticationDomain()` get the domain for NTML Authentication
`java.lang.String`	`getNTLMAuthenticationHost(java.net.URL url)`
`int`	`getNumExtractorPatterns()` Returns the number of extractor patterns currently held by the file.
`int`	`getNumHTTPParameters()` Returns the number of parameters currently held by the file.
`int`	`getNumNavigationActions()` Gets the number of `NavigationAction` objects in the set.
`java.lang.String[]`	`getRedirectURLs()` Gets an array of strings containing the redirect URL's for the current scrapeable file request attempt.
`java.net.URL`	`getReferer()` Gets the URL that should be used for the referer HTTP header when scraping this file.
`java.lang.String`	`getRequestEntity()` Gets the request entity to be used in lieu of POST data.
`ScrapeableFile.RequestType`	`getRequestType()` Returns the type of the request that has been set
`java.lang.String`	`getResolvedURL()` Gets the URL after it's been resolved but before any redirects.
`boolean`	`getRetainNonTidiedHTML()` Gets whether or not the tidied HTML should be retained.
`RetryPolicy`	`getRetryPolicy()` Returns the retry policy.
`RunningScrapingSession`	`getRunningScrapingSession()` Gets the current running scraping session.
`ScrapeableFileNotifiable`	`getScrapeableFileNotifiable()` Gets the `ScrapeableFileNotifiable` object associated with this scrapeable file.
`ScrapingSession`	`getScrapingSession()` Gets the scraping session associated with this scrapeable file.
`ScriptInstances`	`getScriptInstances()` Returns a `ScriptInstances` object containing all script instances associated with this scrapeable file.
`int`	`getSequence()` Gets the sequence with which the parameter should be sent.
`boolean`	`getShouldHighlightSyntax()`
`int`	`getStatusCode()` Gets the status code that was returned from the server when the request was made.
`java.lang.String`	`getStrippedContentAsString()` Gets the scraped content as a string, with all new line characters removed
`java.lang.String`	`getTidyHTML()`
`int`	`getTidyHTMLAsInt()` Indicates how the HTML should be tidied.
`java.lang.String`	`getURL()` Gets the URL of the file to be scraped.
`boolean`	`getUsePlusForEncodedSpaces()` Indicates whether or not + instead of %20 should be used to encode space characters.
`java.lang.String`	`getUserAgent()` Gets the user-agent to be used in HTTP requests.
`boolean`	`getWillBeInvokedManually()` Gets whether or not this file will be invoked manually.
`boolean`	`inputOutputErrorOccurred()` Indicates whether or not an input/output error occured while requesting the file.
`void`	`invokeAction(java.lang.String name)` Invokes a navigation action identified by the given name.
`protected void`	`load()` Loads the scrapeable file with all of its data.
`protected void`	`loadNavigationActions()` Loads navigation actions related to this scrapeable file.
`boolean`	`noExtractorPatternsMatched()` Indicates whether or not none of the extractor patterns for this scrapeable file found a match.
`void`	`removeAllHTTPHeaders()` Removes all custom HTTP headers that were previously added.
`void`	`removeAllHTTPParameters()` Removes all HTTP parameters from the scrapeable file.
`void`	`removeExtractorPattern(ExtractorPattern extractorPattern)` Removes an extractor pattern.
`void`	`removeHTTPHeader(java.lang.String key)` removes all Http Headers with the key
`void`	`removeHTTPHeader(java.lang.String key, java.lang.String value)` removes all http headers with the corresponding key/value
`void`	`removeHTTPParameter(int sequence)` Removes an HTTP parameter, automatically re-sequencing.
`void`	`removeHTTPParameter(int sequence, boolean doResequence)` Removes an HTTP parameter.
`void`	`removeHTTPParameter(java.lang.String key)` Removes an HTTP parameter by its name, automatically re-sequencing.
`void`	`removeNavigationAction(NavigationAction navigationAction)` Removes a navigation action.
`void`	`resequenceHTTPParameter(java.lang.String key, int sequence)` Gives an HTTP parameter a new sequence.
`void`	`reset()` Resets the object after a scrape to ensure that nothing is being retained in memory that shouldn't be.
`void`	`resetContainsNullableTokens()` Resets the containsNullableTokens value for each of the extractor patterns.
`protected void`	`resetCustomHeaders()` Resets the custom headers hash set.
`void`	`resetNonPersistentValues()` This method should be used to reset values that should not persist across times the scrape is run, but since we store the session in memory they can persist in error
`void`	`resetReferer()` Resets the referer value to null.
`java.lang.String`	`resolveRelativeURL(java.lang.String urlToResolve)` Resolve a relative URL
`void`	`restoreHTTPParametersAfterScraping()` Indicates that HTTP parameters should be restored after the scraping session finishes scraping.
`void`	`restoreOriginalHTTPParameters()` Restores the original HTTP parameters.
`void`	`save()` Saves this scraping session to the database.
`void`	`saveFileBeforeTidying(java.lang.String fileToSaveToBeforeTidying)` Sets the path to save the file to just before it gets tidied.
`void`	`saveFileOnRequest(java.lang.String fileToSaveToOnRequest)` Sets the path to save the file to just after it gets requested.
`protected void`	`scrape(ScrapingHttpClient httpClient)` Causes the scrapeable file to scrape its data, extract the needed information and run any scripts.
`(package private) void`	`scrapeFromString(java.lang.String content)`
`void`	`setAuthenticationPreemptive(boolean authenticationPreemptive)` Sets whether or not authentication should be done preemptively.
`void`	`setBASICAuthenticationPassword(java.lang.String BASICAuthenticationPassword)` Sets the BASIC authentication password of the file to be scraped.
`void`	`setBASICAuthenticationUsername(java.lang.String BASICAuthenticationUsername)` Sets the BASIC authentication username of the file to be scraped.
`void`	`setCharacterSet(java.lang.String characterSet)` Sets the character encoding of the content given by the server.
`void`	`setContentType(java.lang.String contentType)` Sets a custom "Content-Type" header to be sent.
`protected void`	`setCurrentURL(java.lang.String currentURL)` Sets the resolved or unresolved URL.
`void`	`setEnableJavaScript(boolean enableJavaScript)` Determines whether or not JavaScript embedded in the HTML page should be processed.
`void`	`setForcedRequestType(ScrapeableFile.RequestType type)` Sets the request type to use.
`void`	`setForceMultiPart(boolean forceMultiPart)` Determines whether or not a multi-part request should be forced.
`void`	`setForceNonBinary(boolean forceNonBinary)` Determines whether or not the contents of this response should be forced to be treated as non-binary.
`void`	`setForcePOST(boolean forcePOST)` Determines whether or not a POST request should be forced.
`void`	`setHTTPTransactionID(int httpTransactionID)` Sets the ID of the HTTP transaction associated with this scrapeable file.
`void`	`setIdentifier(java.lang.String identifier)` Sets the identifier for this object.
`void`	`setLastHttpResponse(ScrapingResponse lastHttpResponse)` Sets the last http response received
`void`	`setLastRequest(java.lang.String lastRequest)` Sets the request from the last time it was scraped.
`void`	`setLastRequestURL(java.net.URL lastRequestURL)` Sets the last request URL used for this scrapeable file
`void`	`setLastScrapedData(java.lang.String lastScrapedData)` Sets the data from the file from the last time it was scraped.
`void`	`setLastTidyAttemptFailed(boolean lastTidyAttemptFailed)` Sets whether or not the last attempt to tidy the response failed.
`void`	`setMaxResponseLength(int maxKBytes)` Sets the maximum response length, in kilobytes.
`void`	`setName(java.lang.String name)` Sets the name.
`protected void`	`setNavigationActionDoer(NavigationActionDoer navigationActionDoer)` Sets the current navigation action doer to be processed.
`void`	`setNode(javax.swing.tree.DefaultMutableTreeNode node)` Sets the node representing this scraping file.
`void`	`setNTLMAuthenticationDomain(java.lang.String domain)` set the domain for NTML Authentication, default is localhost
`void`	`setNTLMAuthenticationHost(java.lang.String host)` set the host for NTML Authentication, default is
`void`	`setParametersFromForm(Form form)` Sets the parameters for a ScrapeableFile to the values in the form object.
`void`	`setReferer(java.lang.String strReferer)` Sets the URL as a string that should be used for the referer HTTP header when scraping this file.
`void`	`setReferer(java.net.URL referer)` Sets the URL that should be used for the referer HTTP header when scraping this file.
`void`	`setRequestEntity(java.lang.String requestEntity)` Sets the request entity to be used in lieu of POST data.
`void`	`setRetainNonTidiedHTML(boolean retainNonTidiedHTML)` Sets whether or not the tidied HTML should be retained.
`void`	`setRetryPolicy(RetryPolicy policy)` Sets a Retry Policy that will be run to check if a page should be re-downloaded or not.
`void`	`setRunningScrapingSession(RunningScrapingSession runningScrapingSession)` Sets the current running scraping session.
`void`	`setScrapeableFileNotifiable(ScrapeableFileNotifiable scrapeableFileNotifiable)` Sets the file that will be notified as speciifc pieces of data in this object get updated.
`void`	`setScrapingSession(ScrapingSession scrapingSession)` Sets the scraping session to be associated with this scrapeable file.
`protected void`	`setScrapingSessionNotifiable(ScrapingSessionNotifiable scrapingSessionNotifiable)` Sets the object the scrapeable file will use to notify its progress of.
`void`	`setScriptInstances(ScriptInstances scriptInstances)` Sets the script instances to be associated with this scrapeable file.
`void`	`setSequence(int sequence)` Sets the sequence with which the parameter should be sent.
`void`	`setShouldHighlightSyntax(boolean b)` Sets the boolean representing whether or not the HTML on the Last Response pane should be syntax highlighted to the given value.
`void`	`setTidyHTML(java.lang.String tidyHTML)`
`void`	`setTidyHTMLAsInt(int tidyHTML)` Determines how the HTML is to be tidied.
`void`	`setURL(java.lang.String url)` Sets the URL of the file to be scraped.
`void`	`setUsePlusForEncodedSpaces(boolean usePlusForEncodedSpaces)` Sets whether or not + instead of %20 should be used to encode space characters.
`void`	`setUserAgent(java.lang.String userAgent)` Sets the user-agent to be used in HTTP requests.
`void`	`setWillBeInvokedManually(boolean willBeInvokedManually)` Sets whether or not this file will be invoked manually.
`java.lang.String`	`toString()` Overridden so that the name appears as the title in the main tree.
`boolean`	`wasErrorOnRequest()` Determines whether or not an error occurred in requesting the file.

Methods inherited from class java.lang.Object
clone, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

- Field Detail
  - log
```
public static org.apache.log4j.Logger log
```
    Used for logging.
  - extractorPatternsMatched
```
boolean extractorPatternsMatched
```
- Constructor Detail
  - ScrapeableFile
```
public ScrapeableFile()
```
    Needed by castor.
  - ScrapeableFile
```
public ScrapeableFile(int scrapeableFileID)
```
    Generates and loads a ScrapeableFile based on the ID. Note that this constructor should be used only for testing purposes.
    
    Parameters:
    
    scrapeableFileID - The ID of a scrapeable file.
  - ScrapeableFile
```
public ScrapeableFile(@NotNull
                      ScrapeableFile scrapeableFile)
```
    Generates a new scrapeable file based on the given scrapeable file.
    
    Parameters:
    
    scrapeableFile - A ScrapeableFile.
  - ScrapeableFile
```
public ScrapeableFile(java.lang.String name)
```
    Generates a new ScrapeableFile.
    
    Parameters:
    
    name - A name to identify the file.
  - ScrapeableFile
```
public ScrapeableFile(java.lang.String name,
                      java.net.URL url)
```
    Generates a new ScrapeableFile.
    
    Parameters:
    
    name - A name to identify the file.
    
    url - The URL that should be used to populate the file.
  - ScrapeableFile
```
public ScrapeableFile(java.lang.String name,
                      @Nullable
                      java.net.URL url,
                      @Nullable
                      java.lang.String postData)
```
    Generates a new ScrapeableFile.
    
    Parameters:
    
    name - A name to identify the file.
    
    url - The URL that should be used to populate the file.
    
    postData - A string representation of the POST parameters.
  - ScrapeableFile
```
public ScrapeableFile(int scrapeableFileID,
                      ScrapingSession scrapingSession)
```
    Generates a new ScrapeableFile.
    
    Parameters:
    
    scrapeableFileID - The ID of the scrapeable file.
    
    scrapingSession - The ScrapingSession associated with this scrapeable file.
  - ScrapeableFile
```
public ScrapeableFile(java.lang.String name,
                      @Nullable
                      HTTPTransaction httpTransaction)
```
    Generates a new ScrapeableFile with the given name, based on the given HTTPTransaction.
    
    Parameters:
    
    name - A name to identify the file.
    
    httpTransaction - The HTTPTransaction on which it's to be based.
  - ScrapeableFile
```
public ScrapeableFile(java.lang.String name,
                      java.lang.String url,
                      int sequence)
```
    Generates a new ScrapeableFile.
    
    Parameters:
    
    name - A name to identify the file.
    
    url - The URL of the file to be scraped.
    
    sequence - The sequence.
- Method Detail
  - getID
```
@InternalOnly
public int getID()
```
    Gets the ID of the scrapeable file.
    
    Specified by:
    
    getID in interface ScriptInstancesContainer
    
    Returns:
    
    The ID.
  - getName
```
@Nullable
public java.lang.String getName()
```
    Gets the name.
    
    Specified by:
    
    getName in interface RunningScrapeableFile
    
    Specified by:
    
    getName in interface ScriptInstancesContainer
    
    Returns:
    
    The name.
  - setName
```
@InternalOnly
public void setName(java.lang.String name)
```
    Sets the name.
    
    Parameters:
    
    name - The name.
  - getURL
```
@Nullable
public java.lang.String getURL()
```
    Gets the URL of the file to be scraped.
    
    Returns:
    
    The URL.
  - setURL
```
public void setURL(@Nullable
                   java.lang.String url)
```
    Sets the URL of the file to be scraped.
    
    Parameters:
    
    url - The URL.
  - getCurrentURL
```
@Nullable
public java.lang.String getCurrentURL()
```
    Depending on when this value is requested it will return either the unresolved or resolved URL of the file to be scraped.
    
    Specified by:
    
    getCurrentURL in interface RunningScrapeableFile
    
    Returns:
    
    The URL.
  - setCurrentURL
```
protected void setCurrentURL(java.lang.String currentURL)
```
    Sets the resolved or unresolved URL.
    
    Parameters:
    
    currentURL - The URL.
  - getResolvedURL
```
@Nullable
public java.lang.String getResolvedURL()
```
    Gets the URL after it's been resolved but before any redirects.
    
    Returns:
    
    The resolved URL.
  - getSequence
```
public int getSequence()
```
    Gets the sequence with which the parameter should be sent.
    
    Returns:
    
    The sequence.
  - setSequence
```
@InternalOnly
public void setSequence(int sequence)
```
    Sets the sequence with which the parameter should be sent.
    
    Parameters:
    
    sequence - The sequence.
  - getBASICAuthenticationUsername
```
@Nullable
public java.lang.String getBASICAuthenticationUsername()
```
    Gets the BASIC authentication username of the file to be scraped.
    
    Returns:
    
    The BASIC authentication username.
  - setBASICAuthenticationUsername
```
public void setBASICAuthenticationUsername(java.lang.String BASICAuthenticationUsername)
```
    Sets the BASIC authentication username of the file to be scraped.
    
    Parameters:
    
    BASICAuthenticationUsername - The BASIC authentication username.
  - getBASICAuthenticationPassword
```
@Nullable
public java.lang.String getBASICAuthenticationPassword()
```
    Gets the BASIC authentication password of the file to be scraped.
    
    Returns:
    
    The BASIC authentication password.
  - setBASICAuthenticationPassword
```
public void setBASICAuthenticationPassword(java.lang.String BASICAuthenticationPassword)
```
    Sets the BASIC authentication password of the file to be scraped.
    
    Parameters:
    
    BASICAuthenticationPassword - The BASIC authentication password.
  - getNTLMAuthenticationDomain
```
@Nullable
public java.lang.String getNTLMAuthenticationDomain()
```
    get the domain for NTML Authentication
    
    Returns:
    
    The String representing the domain
  - setNTLMAuthenticationDomain
```
public void setNTLMAuthenticationDomain(java.lang.String domain)
```
    set the domain for NTML Authentication, default is localhost
    
    Parameters:
    
    domain - The domain for the NTML Authentication
  - setNTLMAuthenticationHost
```
public void setNTLMAuthenticationHost(java.lang.String host)
```
    set the host for NTML Authentication, default is
    
    Parameters:
    
    host - The host for NTML Authentication
  - getNTLMAuthenticationHost
```
@Nullable
 @InternalOnly
public java.lang.String getNTLMAuthenticationHost(@NotNull
                                                                             java.net.URL url)
```
  - setLastHttpResponse
```
@InternalOnly
public void setLastHttpResponse(@Nullable
                                               ScrapingResponse lastHttpResponse)
```
    Sets the last http response received
    
    Parameters:
    
    lastHttpResponse - The last http response
  - getLastHttpResponse
```
@Nullable
 @InternalOnly
public ScrapingResponse getLastHttpResponse()
```
    Gets the last http response received
    
    Returns:
    
    The last http response
  - getLastRequest
```
@Nullable
public java.lang.String getLastRequest()
```
    Gets the request from the last time it was scraped.
    
    Returns:
    
    The request.
  - getLastRequestShort
```
@Nullable
public java.lang.String getLastRequestShort()
```
    Returns the last request, truncating it if it's too long.
    
    Returns:
    
    The last request.
  - setLastRequest
```
@InternalOnly
public void setLastRequest(@Nullable
                                          java.lang.String lastRequest)
```
    Sets the request from the last time it was scraped.
    
    Parameters:
    
    lastRequest - The data.
  - wasErrorOnRequest
```
public boolean wasErrorOnRequest()
```
    Determines whether or not an error occurred in requesting the file.
    
    Specified by:
    
    wasErrorOnRequest in interface RunningScrapeableFile
    
    Returns:
    
    A boolean.
  - inputOutputErrorOccurred
```
public boolean inputOutputErrorOccurred()
```
    Indicates whether or not an input/output error occured while requesting the file.
    
    Returns:
    
    A boolean.
  - noExtractorPatternsMatched
```
public boolean noExtractorPatternsMatched()
```
    Indicates whether or not none of the extractor patterns for this scrapeable file found a match.
    
    Specified by:
    
    noExtractorPatternsMatched in interface RunningScrapeableFile
    
    Returns:
    
    A boolean.
  - getContentAsString
```
@Nullable
public java.lang.String getContentAsString()
```
    Gets the scraped content as a string. This is used by the RunnableScrapeableFile facade.
    
    Specified by:
    
    getContentAsString in interface RunningScrapeableFile
    
    Returns:
    
    The content.
  - getStrippedContentAsString
```
@Nullable
public java.lang.String getStrippedContentAsString()
```
    Description copied from interface: RunningScrapeableFile
    
    Gets the scraped content as a string, with all new line characters removed
    
    Specified by:
    
    getStrippedContentAsString in interface RunningScrapeableFile
    
    Returns:
    
    The content
  - getLastScrapedData
```
@Nullable
public java.lang.String getLastScrapedData()
```
    Gets the data from the file from the last time it was scraped.
    
    Returns:
    
    The data.
  - getContentBodyOnly
```
@Nullable
public java.lang.String getContentBodyOnly()
```
    Gets only the body of the response (no headers or status line) as a string.
    
    Returns:
    
    The data.
  - getLastScrapedDataShort
```
@Nullable
public java.lang.String getLastScrapedDataShort()
```
    Gets the data from the file from the last time it was scraped, truncating it, if necessary.
    
    Returns:
    
    The data.
  - setLastScrapedData
```
@InternalOnly
public void setLastScrapedData(@Nullable
                                              java.lang.String lastScrapedData)
```
    Sets the data from the file from the last time it was scraped.
    
    Parameters:
    
    lastScrapedData - The data.
  - setLastRequestURL
```
@InternalOnly
public void setLastRequestURL(@Nullable
                                             java.net.URL lastRequestURL)
```
    Sets the last request URL used for this scrapeable file
    
    Parameters:
    
    lastRequestURL - The last request URL used, or null if unknown or wasn't a valid URL
  - getLastRequestURL
```
@Nullable
 @InternalOnly
public java.net.URL getLastRequestURL()
```
    Gets the last request URL used for this scrapeable file
    
    Returns:
    
    The last request URL used, or null if unknown or wasn't a valid URL
  - getScrapingSession
```
@Nullable
 @InternalOnly
public ScrapingSession getScrapingSession()
```
    Gets the scraping session associated with this scrapeable file.
    
    Returns:
    
    The scraping session.
  - setScrapingSession
```
@InternalOnly
public void setScrapingSession(ScrapingSession scrapingSession)
```
    Sets the scraping session to be associated with this scrapeable file.
    
    Parameters:
    
    scrapingSession - The scraping session.
  - setNode
```
@InternalOnly
public void setNode(javax.swing.tree.DefaultMutableTreeNode node)
```
    Sets the node representing this scraping file.
    
    Parameters:
    
    node - A DefaultMutableTreeNode.
  - getNode
```
@Nullable
 @InternalOnly
public javax.swing.tree.DefaultMutableTreeNode getNode()
```
    Gets the node representing this scraping file.
    
    Returns:
    
    A DefaultMutableTreeNode.
  - getRunningScrapingSession
```
@Nullable
 @InternalOnly
public RunningScrapingSession getRunningScrapingSession()
```
    Gets the current running scraping session.
    
    Returns:
    
    The running scraping session.
  - setRunningScrapingSession
```
@InternalOnly
public void setRunningScrapingSession(RunningScrapingSession runningScrapingSession)
```
    Sets the current running scraping session.
    
    Parameters:
    
    runningScrapingSession - The running scraping session.
  - getHTTPParameters
```
@NotNull
public java.util.HashSet<HTTPParameter> getHTTPParameters()
```
    Gets all of the HTTP parameters held by this scrapeable file. Note that this is only public because of Castor.
    
    Returns:
    
    A HashSet of HTTPParameters.
  - getHTTPParameterByName
```
@Nullable
public HTTPParameter getHTTPParameterByName(java.lang.String key)
```
    Gets an HTTP parameter by its name.
    
    Parameters:
    
    key - The name of the parameter.
    
    Returns:
    
    The parameter with the given name, or null if no such parameter is found
  - addGETHTTPParameter
```
@RequiredVersion(value=1)
public void addGETHTTPParameter(java.lang.String key,
                                                           java.lang.String value,
                                                           int sequence)
```
    Adds a GET HTTP parameter with the given parameters.
    
    Parameters:
    
    key - The name of the parameter.
    
    value - The value of the parameter.
    
    sequence - The sequence of the parameter.
  - addGETHTTPParameter
```
@RequiredVersion(value=1)
public void addGETHTTPParameter(java.lang.String key,
                                                           java.lang.String value)
```
    Adds a GET HTTP parameter with the given parameters to the end of the existing parameters.
    
    Parameters:
    
    key - The name of the parameter.
    
    value - The value of the parameter.
  - addPOSTHTTPParameter
```
@RequiredVersion(value=1)
public void addPOSTHTTPParameter(java.lang.String key,
                                                            java.lang.String value,
                                                            int sequence)
```
    Adds a POST HTTP parameter with the given parameters.
    
    Parameters:
    
    key - The name of the parameter.
    
    value - The value of the parameter.
    
    sequence - The sequence of the parameter.
  - addPOSTHTTPParameter
```
@RequiredVersion(value=1)
public void addPOSTHTTPParameter(java.lang.String key,
                                                            java.lang.String value)
```
    Adds a POST HTTP parameter with the given parameters to the end of the existing parameters.
    
    Parameters:
    
    key - The name of the parameter.
    
    value - The value of the parameter.
  - addHTTPParameter
```
public void addHTTPParameter(@Nullable
                             HTTPParameter httpParameter)
```
    Adds a parameter to the scrapeable file.
    
    Specified by:
    
    addHTTPParameter in interface RunningScrapeableFile
    
    Parameters:
    
    httpParameter - An HTTPParameter.
  - resequenceHTTPParameter
```
@RequiredVersion(value=1)
public void resequenceHTTPParameter(java.lang.String key,
                                                               int sequence)
```
    Gives an HTTP parameter a new sequence.
    
    Parameters:
    
    key - The name of the parameter.
    
    sequence - The new sequence.
  - bumpHTTPParameterSequences
```
@InternalOnly
public void bumpHTTPParameterSequences(int bottomSequence,
                                                      int topSequence,
                                                      int bumpModifier)
```
    Bumps sequences up or down between the bottom and top sequences.
    
    Parameters:
    
    bottomSequence - The bottom sequence.
    
    topSequence - The top sequence.
    
    bumpModifier - Modifies each sequence.
  - removeHTTPParameter
```
public void removeHTTPParameter(int sequence)
```
    Removes an HTTP parameter, automatically re-sequencing.
    
    Parameters:
    
    sequence - The sequence.
  - removeHTTPParameter
```
@RequiredVersion(value=1)
public void removeHTTPParameter(java.lang.String key)
```
    Removes an HTTP parameter by its name, automatically re-sequencing.
    
    Parameters:
    
    key - The name of the parameter.
  - removeHTTPParameter
```
@InternalOnly
public void removeHTTPParameter(int sequence,
                                               boolean doResequence)
```
    Removes an HTTP parameter.
    
    Parameters:
    
    sequence - The sequence.
    
    doResequence - Indicates whether or not the parametrs should be re-sequenced afterward.
  - removeAllHTTPParameters
```
@RequiredVersion(value=1)
public void removeAllHTTPParameters()
```
    Removes all HTTP parameters from the scrapeable file.
    
    Specified by:
    
    removeAllHTTPParameters in interface RunningScrapeableFile
  - dumpHTTPParametersToLog
```
@InternalOnly
public void dumpHTTPParametersToLog()
```
    Dumps all of the HTTP parameters to the log.
  - getHTTPParameter
```
@Nullable
public HTTPParameter getHTTPParameter(int sequence)
```
    Gets an HTTP parameter from the session.
    
    Parameters:
    
    sequence - The sequence of the parameter to be retrieved.
    
    Returns:
    
    The HTTP parameter at the specified sequence
  - getHTTPParameter
```
@Nullable
public HTTPParameter getHTTPParameter(java.lang.String key)
```
    Gets an HTTP parameter from the session.
    
    Parameters:
    
    key - The key of the parameter to be retrieved.
  - addExtractorPattern
```
@InternalOnly
public void addExtractorPattern(@Nullable
                                               ExtractorPattern extractorPattern)
```
    Adds a pattern to the scrapeable file.
    
    Parameters:
    
    extractorPattern - An ExtractorPattern.
  - removeExtractorPattern
```
@InternalOnly
public void removeExtractorPattern(@Nullable
                                                  ExtractorPattern extractorPattern)
```
    Removes an extractor pattern.
    
    Parameters:
    
    extractorPattern - The pattern to be removed.
  - getExtractorPattern
```
@InternalOnly
 @Nullable
public ExtractorPattern getExtractorPattern(int sequence)
```
    Gets an extractor pattern from the ScrapeableFile.
    
    Parameters:
    
    sequence - The sequence of the pattern to be retrieved.
    
    Returns:
    
    The Extractor Pattern, or null if there wasn't one with the given sequence
  - getExtractorPattern
```
@Nullable
 @InternalOnly
public ExtractorPattern getExtractorPattern(@Nullable
                                                                       java.lang.String name)
```
    Gets an extractor pattern from the ScrapeableFile.
    
    Parameters:
    
    name - The name of the pattern to be retrieved. If more than one is found the first one will be returned, ordering by sequence.
  - getExtractorPatternByID
```
@Nullable
 @InternalOnly
public ExtractorPattern getExtractorPatternByID(int extractorPatternID)
```
    Gets the extractor pattern by its ID
    
    Parameters:
    
    extractorPatternID - The ID of the pattern to be retrieved.
  - resetContainsNullableTokens
```
@InternalOnly
public void resetContainsNullableTokens()
```
    Resets the containsNullableTokens value for each of the extractor patterns.
  - getExtractorPatterns
```
@InternalOnly
public java.util.HashSet getExtractorPatterns()
```
    Returns all extractor patterns held by this scrapeable file. Note that this is only public because of Castor.
    
    Returns:
    
    A HashSet containing the patterns.
  - getExtractorPatternsIterator
```
@NotNull
 @InternalOnly
public java.util.Iterator getExtractorPatternsIterator()
```
    Gets an Iterator that can be used to cycle through the ExtractorPattern objects held by this object.
    
    Returns:
    
    The Iterator.
  - getNumExtractorPatterns
```
@InternalOnly
public int getNumExtractorPatterns()
```
    Returns the number of extractor patterns currently held by the file.
    
    Returns:
    
    The number of patterns.
  - getNumHTTPParameters
```
public int getNumHTTPParameters()
```
    Returns the number of parameters currently held by the file.
    
    Returns:
    
    The number of parameters.
  - getScriptInstances
```
@InternalOnly
public ScriptInstances getScriptInstances()
```
    Returns a ScriptInstances object containing all script instances associated with this scrapeable file.
    
    Specified by:
    
    getScriptInstances in interface ScriptInstancesContainer
    
    Returns:
    
    The script instances.
  - setScriptInstances
```
@InternalOnly
public void setScriptInstances(ScriptInstances scriptInstances)
```
    Sets the script instances to be associated with this scrapeable file.
    
    Specified by:
    
    setScriptInstances in interface ScriptInstancesContainer
    
    Parameters:
    
    scriptInstances - The script instances.
  - toString
```
@Nullable
 @InternalOnly
public java.lang.String toString()
```
    Overridden so that the name appears as the title in the main tree.
    
    Overrides:
    
    toString in class java.lang.Object
    
    Returns:
    
    The name of the session.
  - compareTo
```
@InternalOnly
public int compareTo(@NotNull
                                    java.lang.Object scrapeableFile)
```
    Required by the Comparable interface.
    
    Specified by:
    
    compareTo in interface java.lang.Comparable
    
    Parameters:
    
    scrapeableFile - The file to compare to.
    
    Returns:
    
    An int indicating how the given object compares to this object.
  - equals
```
@InternalOnly
public boolean equals(@Nullable
                                     java.lang.Object object)
```
    Overriding equals...
    
    Overrides:
    
    equals in class java.lang.Object
    
    Parameters:
    
    object - The object we're comparing.
  - save
```
@InternalOnly
public void save()
```
    Saves this scraping session to the database.
  - delete
```
@InternalOnly
public void delete()
```
    Deletes the scrapeable file from the database.
    
    Specified by:
    
    delete in interface Deleteable
  - containsFileUpload
```
@InternalOnly
public boolean containsFileUpload()
```
    Indicates whether or not the file contains POST parameters.
    
    Returns:
    
    A boolean.
  - getCurrentPOSTData
```
@Nullable
public java.lang.String getCurrentPOSTData()
```
    Gets the current value of the POST data. Note that this is only used by RunningScrapeableFile so that the resolved POST data can be made accessible in a script. If this method is called before this scrapeable file is scraped it will return null.
    
    Specified by:
    
    getCurrentPOSTData in interface RunningScrapeableFile
    
    Returns:
    
    The resolved POST data.
  - scrape
```
protected void scrape(@Nullable
                      ScrapingHttpClient httpClient)
```
    Causes the scrapeable file to scrape its data, extract the needed information and run any scripts.
    
    Parameters:
    
    httpClient - The HttpClient object to be used in scraping the data.
  - setForcedRequestType
```
@RequiredVersion(value=1,
                 behavior=LOG_ERROR_AND_RETURN_NULL_NEG_ONE_OR_FALSE)
public void setForcedRequestType(ScrapeableFile.RequestType type)
```
    Sets the request type to use.
    
    Parameters:
    
    type - The type of request to issue, or null to let screen-scraper decide
  - addRedirectURL
```
protected void addRedirectURL(java.lang.String url)
```
    Adds a redirect URL to the list.
    
    Parameters:
    
    url - The URL as a string.
  - getRedirectURLs
```
@NotNull
 @RequiredVersion(value=1)
public java.lang.String[] getRedirectURLs()
```
    Gets an array of strings containing the redirect URL's for the current scrapeable file request attempt.
    
    Returns:
    
    The array of strings; may be empty.
  - scrapeFromString
```
void scrapeFromString(@Nullable
                      java.lang.String content)
```
  - getCustomHeaders
```
@NotNull
 @InternalOnly
public java.util.Set<ScrapingHttpHeader> getCustomHeaders()
```
    Gets the custom headers hash set (not a copy of it)
    
    Returns:
    
    The hash set.
  - removeAllHTTPHeaders
```
public void removeAllHTTPHeaders()
```
    Removes all custom HTTP headers that were previously added.
  - addHTTPHeader
```
@RequiredVersion(value=1)
public void addHTTPHeader(@NotNull
                                                     java.lang.String key,
                                                     @Nullable
                                                     java.lang.String value)
```
    Adds a custom header.
    
    Parameters:
    
    key - The key.
    
    value - The value.
  - removeHTTPHeader
```
@RequiredVersion(value=1)
public void removeHTTPHeader(@Nullable
                                                        java.lang.String key)
```
    removes all Http Headers with the key
    
    Parameters:
    
    key -
    
    Throws:
    
    java.lang.Exception
  - removeHTTPHeader
```
@RequiredVersion(value=2)
public void removeHTTPHeader(@Nullable
                                                        java.lang.String key,
                                                        @Nullable
                                                        java.lang.String value)
```
    removes all http headers with the corresponding key/value
    
    Parameters:
    
    key - The name of the header
    
    value - The value of the header
    
    Throws:
    
    java.lang.Exception - In an invalid version of screen-scraper
  - resetCustomHeaders
```
protected void resetCustomHeaders()
```
    Resets the custom headers hash set.
  - addBasicHeaders
```
@InternalOnly
public void addBasicHeaders()
```
    Adds request headers to the custom headers, so they can be removed
  - setScrapingSessionNotifiable
```
protected void setScrapingSessionNotifiable(ScrapingSessionNotifiable scrapingSessionNotifiable)
```
    Sets the object the scrapeable file will use to notify its progress of.
    
    Parameters:
    
    scrapingSessionNotifiable - The object.
  - getWillBeInvokedManually
```
@InternalOnly
public boolean getWillBeInvokedManually()
```
    Gets whether or not this file will be invoked manually.
    
    Returns:
    
    A boolean.
  - setWillBeInvokedManually
```
@InternalOnly
public void setWillBeInvokedManually(boolean willBeInvokedManually)
```
    Sets whether or not this file will be invoked manually.
    
    Parameters:
    
    willBeInvokedManually - A boolean.
  - getScrapeableFileNotifiable
```
@Nullable
 @InternalOnly
public ScrapeableFileNotifiable getScrapeableFileNotifiable()
```
    Gets the ScrapeableFileNotifiable object associated with this scrapeable file.
    
    Returns:
    
    The ScrapeableFileNotifiable.
  - setScrapeableFileNotifiable
```
@InternalOnly
public void setScrapeableFileNotifiable(ScrapeableFileNotifiable scrapeableFileNotifiable)
```
    Sets the file that will be notified as speciifc pieces of data in this object get updated.
    
    Parameters:
    
    scrapeableFileNotifiable - The notifiable object.
  - getReferer
```
@Nullable
public java.net.URL getReferer()
```
    Gets the URL that should be used for the referer HTTP header when scraping this file.
    
    Returns:
    
    The referer URL.
  - setReferer
```
public void setReferer(java.net.URL referer)
```
    Sets the URL that should be used for the referer HTTP header when scraping this file.
    
    Parameters:
    
    referer - The referer URL.
  - setReferer
```
@RequiredVersion(value=1,
                 behavior=LOG_ERROR_AND_RETURN_NULL_NEG_ONE_OR_FALSE)
public void setReferer(@Nullable
                                                                                                                         java.lang.String strReferer)
```
    Sets the URL as a string that should be used for the referer HTTP header when scraping this file.
    
    Parameters:
    
    strReferer - The referer URL.
  - resetReferer
```
public void resetReferer()
```
    Resets the referer value to null.
  - setContentType
```
@RequiredVersion(value=1,
                 behavior=LOG_ERROR_AND_RETURN_NULL_NEG_ONE_OR_FALSE)
public void setContentType(java.lang.String contentType)
```
    Sets a custom "Content-Type" header to be sent.
    
    Parameters:
    
    contentType - The "Content-Type" value to be sent.
  - getContentType
```
@Nullable
public java.lang.String getContentType()
```
    Gets the custom "Content-Type" header to be sent.
    
    Returns:
    
    The "Content-Type" value to be sent.
  - extractData
```
@Nullable
 @InternalOnly
public DataSet extractData(@NotNull
                                                      ScriptContext scriptContext,
                                                      @NotNull
                                                      ExtractorPattern extractorPattern,
                                                      DataSet dataSet,
                                                      boolean scripts)
```
    Extracts data by applying the given ExtractorPattern to the supplied text. Calls extractData( scriptContext, extractorPattern, dataSet, scripts, true ).
    
    Parameters:
    
    scriptContext - The ScriptContext to use when extracting data
    
    extractorPattern - The ExtractorPattern to extract from
    
    dataSet - The DataSet with the extracted data
    
    scripts - Whether to execute the scripts associated with this extractor pattern
    
    Returns:
    
    A DataSet containing the extracted data.
  - extractData
```
@Nullable
 @InternalOnly
public DataSet extractData(@NotNull
                                                      ScriptContext scriptContext,
                                                      @NotNull
                                                      ExtractorPattern extractorPattern,
                                                      DataSet dataSet,
                                                      boolean scripts,
                                                      boolean isApplyPattern)
```
    Extract Data from an Extracter Pattern Handle any mapping, resolving of url's or saving of session variables that need to be done
    
    Parameters:
    
    scriptContext - The ScriptContext to use when extracting data
    
    extractorPattern - The ExtractorPattern to extract from
    
    dataSet - The DataSet with the extracted data
    
    scripts - Whether to execute the scripts associated with this extractor pattern
    
    isApplyPattern - Whether to execute extra commands when 'Apply Pattern to Last Scraped Data' pushed.
    
    Returns:
    
    The DataSet with the extracted data
  - applyXPathExpression
```
@Nullable
 @RequiredVersion(value=1)
public XmlNode applyXPathExpression(java.lang.String expression)
```
    Applies an XPath expression to the current response.
    
    Parameters:
    
    expression - An XPath expression.
    
    Returns:
    
    An XmlNode.
  - resolveRelativeURL
```
@Nullable
 @RequiredVersion(value=1,
                 behavior=LOG_ERROR_ONLY)
 @Contract(value="!null -> !null; null -> null")
public java.lang.String resolveRelativeURL(@Nullable
                                                                                                                                                                               java.lang.String urlToResolve)
```
    Resolve a relative URL
    
    Specified by:
    
    resolveRelativeURL in interface RunningScrapeableFile
    
    Parameters:
    
    urlToResolve - The relative URL to resolve.
    
    Returns:
    
    A string with the resolved URL or the original URL if it couldn't resolve
  - extractData
```
@Nullable
 @RequiredVersion(value=1,
                 behavior=LOG_ERROR_AND_RETURN_NULL_NEG_ONE_OR_FALSE)
public DataSet extractData(@Nullable
                                                                                                                                         java.lang.String text,
                                                                                                                                         @Nullable
                                                                                                                                         java.lang.String name)
```
    Manually extracts data using an existing ExtractorPattern.
    
    Specified by:
    
    extractData in interface RunningScrapeableFile
    
    Parameters:
    
    text - The text to extract data from.
    
    name - The name of the ExtractorPattern to be used. The ExtractorPattern must be associated with this ScrapeableFile. If more than one ExtractorPattern is found with the given name the first (by sequence) will be used.
    
    Returns:
    
    A DataSet containing the extracted data.
  - extractOneValue
```
@Nullable
 @RequiredVersion(value=1,
                 behavior=LOG_ERROR_AND_RETURN_NULL_NEG_ONE_OR_FALSE)
public java.lang.String extractOneValue(java.lang.String text,
                                                                                                                                                      java.lang.String name)
```
    Manually extracts one value from data using an existing ExtractorPattern.
    
    Specified by:
    
    extractOneValue in interface RunningScrapeableFile
    
    Parameters:
    
    text - The text to extract data from.
    
    name - The name of the ExtractorPattern to be used. The ExtractorPattern must be associated with this ScrapeableFile. If more than one ExtractorPattern is found with the given name the first (by sequence) will be used.
    
    Returns:
    
    A String containing the extracted data. If nothing mathces null is returned.
  - extractOneValue
```
@Nullable
 @RequiredVersion(value=1,
                 behavior=LOG_ERROR_AND_RETURN_NULL_NEG_ONE_OR_FALSE)
public java.lang.String extractOneValue(@Nullable
                                                                                                                                                      java.lang.String text,
                                                                                                                                                      @Nullable
                                                                                                                                                      java.lang.String name,
                                                                                                                                                      @Nullable
                                                                                                                                                      java.lang.String token)
```
    Manually extracts one value from data using an existing ExtractorPattern.
    
    Specified by:
    
    extractOneValue in interface RunningScrapeableFile
    
    Parameters:
    
    text - The text to extract data from.
    
    name - The name of the ExtractorPattern to be used. The ExtractorPattern must be associated with this ScrapeableFile. If more than one ExtractorPattern is found with the given name the first (by sequence) will be used.
    
    token - The name of the Token to return
    
    Returns:
    
    A String containing the extracted data. If nothing mathces null is returned.
  - load
```
protected void load()
```
    Loads the scrapeable file with all of its data.
  - flagAsLoaded
```
@InternalOnly
public void flagAsLoaded()
```
    Flags it as being fully loaded.
  - getTidyHTMLAsInt
```
@InternalOnly
public int getTidyHTMLAsInt()
```
    Indicates how the HTML should be tidied.
    
    Returns:
    
    An int corresponding to one of the TIDY_HTML constants.
  - getTidyHTML
```
@NotNull
 @InternalOnly
public java.lang.String getTidyHTML()
```
  - setTidyHTML
```
@InternalOnly
public void setTidyHTML(java.lang.String tidyHTML)
```
  - setTidyHTMLAsInt
```
@InternalOnly
public void setTidyHTMLAsInt(int tidyHTML)
```
    Determines how the HTML is to be tidied.
    
    Parameters:
    
    tidyHTML - An int corresponding to one of the TIDY_HTML constants.
  - getCharacterSet
```
@Nullable
public java.lang.String getCharacterSet()
```
    Gets the character encoding of the content given by the server.
    
    Returns:
    
    The character encoding.
  - setCharacterSet
```
public void setCharacterSet(java.lang.String characterSet)
```
    Sets the character encoding of the content given by the server.
    
    Parameters:
    
    characterSet - The character encoding.
  - getUserAgent
```
@Nullable
public java.lang.String getUserAgent()
```
    Gets the user-agent to be used in HTTP requests.
    
    Returns:
    
    The user-agent.
  - setUserAgent
```
@RequiredVersion(value=1,
                 behavior=LOG_ERROR_AND_RETURN_NULL_NEG_ONE_OR_FALSE)
public void setUserAgent(@Nullable
                                                                                                                           java.lang.String userAgent)
```
    Sets the user-agent to be used in HTTP requests.
    
    Parameters:
    
    userAgent - The user-agent.
  - getRetainNonTidiedHTML
```
public boolean getRetainNonTidiedHTML()
```
    Gets whether or not the tidied HTML should be retained.
    
    Returns:
    
    A boolean.
  - setRetainNonTidiedHTML
```
@RequiredVersion(value=2,
                 behavior=LOG_ERROR_AND_RETURN_NULL_NEG_ONE_OR_FALSE)
public void setRetainNonTidiedHTML(boolean retainNonTidiedHTML)
```
    Sets whether or not the tidied HTML should be retained.
    
    Parameters:
    
    retainNonTidiedHTML - A boolean.
  - getNonTidiedHTML
```
@Nullable
 @RequiredVersion(value=2,
                 behavior=LOG_ERROR_AND_RETURN_NULL_NEG_ONE_OR_FALSE)
public java.lang.String getNonTidiedHTML()
```
    Gets the non-tidied HTML.
    
    Returns:
    
    The non-tidied HTML.
  - getRequestEntity
```
@Nullable
public java.lang.String getRequestEntity()
```
    Gets the request entity to be used in lieu of POST data.
    
    Returns:
    
    The entity, as a string.
  - setRequestEntity
```
public void setRequestEntity(java.lang.String requestEntity)
```
    Sets the request entity to be used in lieu of POST data.
    
    Parameters:
    
    requestEntity - The entity, as a string.
  - getStatusCode
```
@RequiredVersion(value=1,
                 behavior=LOG_ERROR_AND_RETURN_NULL_NEG_ONE_OR_FALSE)
public int getStatusCode()
```
    Gets the status code that was returned from the server when the request was made.
    
    Returns:
    
    The status code.
  - restoreOriginalHTTPParameters
```
public void restoreOriginalHTTPParameters()
```
    Restores the original HTTP parameters.
  - restoreHTTPParametersAfterScraping
```
public void restoreHTTPParametersAfterScraping()
```
    Indicates that HTTP parameters should be restored after the scraping session finishes scraping.
  - saveFileOnRequest
```
@RequiredVersion(value=2,
                 behavior=LOG_ERROR_AND_RETURN_NULL_NEG_ONE_OR_FALSE)
public void saveFileOnRequest(java.lang.String fileToSaveToOnRequest)
```
    Sets the path to save the file to just after it gets requested.
    
    Specified by:
    
    saveFileOnRequest in interface RunningScrapeableFile
    
    Parameters:
    
    fileToSaveToOnRequest - The path of the file to save to.
  - saveFileBeforeTidying
```
@RequiredVersion(value=1,
                 behavior=LOG_ERROR_AND_RETURN_NULL_NEG_ONE_OR_FALSE)
public void saveFileBeforeTidying(java.lang.String fileToSaveToBeforeTidying)
```
    Sets the path to save the file to just before it gets tidied.
    
    Parameters:
    
    fileToSaveToBeforeTidying - The path of the file to save to.
  - getForceMultiPart
```
public boolean getForceMultiPart()
```
    Indicates whether or not a multi-part request should be forced.
    
    Returns:
    
    A boolean.
  - setForceMultiPart
```
@RequiredVersion(value=1,
                 behavior=LOG_ERROR_AND_RETURN_NULL_NEG_ONE_OR_FALSE)
public void setForceMultiPart(boolean forceMultiPart)
```
    Determines whether or not a multi-part request should be forced.
    
    Parameters:
    
    forceMultiPart - A boolean.
  - getForcePOST
```
public boolean getForcePOST()
```
    Indicates whether or not a POST request should be forced.
  - setForcePOST
```
@RequiredVersion(value=1,
                 behavior=LOG_ERROR_AND_RETURN_NULL_NEG_ONE_OR_FALSE)
public void setForcePOST(boolean forcePOST)
```
    Determines whether or not a POST request should be forced.
    
    Parameters:
    
    forcePOST - Whether a POST should be forced or not
  - getRequestType
```
@NotNull
 @InternalOnly
public ScrapeableFile.RequestType getRequestType()
```
    Returns the type of the request that has been set
    
    Returns:
    
    The type of the request that has been set
  - invokeAction
```
@InternalOnly
public void invokeAction(@Nullable
                                        java.lang.String name)
```
    Invokes a navigation action identified by the given name.
    
    Parameters:
    
    name - The name of the NavigationAction.
  - resetNonPersistentValues
```
@InternalOnly
public void resetNonPersistentValues()
```
    This method should be used to reset values that should not persist across times the scrape is run, but since we store the session in memory they can persist in error
  - getHtmlPage
```
@Nullable
 @InternalOnly
public com.gargoylesoftware.htmlunit.html.HtmlPage getHtmlPage()
```
    Gets an HtmlPage first by attempting to see if one was already given from the last request. Failing that, it generates one based on the last scraped data and current URL.
    
    Returns:
    
    An HtmlPage.
  - loadNavigationActions
```
protected void loadNavigationActions()
```
    Loads navigation actions related to this scrapeable file.
  - getNavigationActions
```
@Nullable
 @InternalOnly
public java.util.TreeSet getNavigationActions()
```
    Gets a reference to the TreeSet containing the NavigationAction objects. Note that this is public only for Castor.
    
    Returns:
    
    The TreeSet.
  - ensureNavigationActionsHaveScrapeableFileReference
```
protected void ensureNavigationActionsHaveScrapeableFileReference()
```
    Ensures that all of the navigation actions held by this object have a reference to the object.
  - addNavigationAction
```
@InternalOnly
public void addNavigationAction(@NotNull
                                               NavigationAction navigationAction)
```
    Adds a NavigationCriteria to the set.
    
    Parameters:
    
    navigationAction - The NavigationCriteria.
  - getNavigationActionsIterator
```
@NotNull
 @InternalOnly
public java.util.Iterator getNavigationActionsIterator()
```
    Gets an Iterator that can be used to cycle through the NavigationAction objects held by this object.
    
    Returns:
    
    The Iterator.
  - getNumNavigationActions
```
@InternalOnly
public int getNumNavigationActions()
```
    Gets the number of NavigationAction objects in the set.
    
    Returns:
    
    The number of objects.
  - removeNavigationAction
```
@InternalOnly
public void removeNavigationAction(@Nullable
                                                  NavigationAction navigationAction)
```
    Removes a navigation action.
    
    Parameters:
    
    navigationAction - The navigation action to be removed.
  - getNavigationAction
```
@Nullable
 @InternalOnly
public NavigationAction getNavigationAction(int sequence)
```
    Gets a navigation action from the ScrapeableFile.
    
    Parameters:
    
    sequence - The sequence of the action to be retrieved.
  - reset
```
@InternalOnly
public void reset()
```
    Resets the object after a scrape to ensure that nothing is being retained in memory that shouldn't be.
  - getNavigationActionDoer
```
@Nullable
protected NavigationActionDoer getNavigationActionDoer()
```
    Gets the current navigation action doer to be processed.
    
    Returns:
    
    The NavigationActionDoer to be processed.
  - setNavigationActionDoer
```
protected void setNavigationActionDoer(NavigationActionDoer navigationActionDoer)
```
    Sets the current navigation action doer to be processed.
    
    Parameters:
    
    navigationActionDoer - The NavigationActionDoer to be processed.
  - getEnableJavaScript
```
@InternalOnly
public boolean getEnableJavaScript()
```
    Indicates whether or not JavaScript embedded in the HTML page should be processed.
    
    Returns:
    
    A boolean.
  - setEnableJavaScript
```
@InternalOnly
public void setEnableJavaScript(boolean enableJavaScript)
```
    Determines whether or not JavaScript embedded in the HTML page should be processed.
    
    Parameters:
    
    enableJavaScript - A boolean.
  - getAuthenticationPreemptive
```
@InternalOnly
public boolean getAuthenticationPreemptive()
```
    Indicates whether or not authentication should be done preemptively.
    
    Returns:
    
    A boolean
  - setAuthenticationPreemptive
```
@InternalOnly
public void setAuthenticationPreemptive(boolean authenticationPreemptive)
```
    Sets whether or not authentication should be done preemptively.
    
    Parameters:
    
    authenticationPreemptive - A boolean
  - getHTTPTransactionID
```
@InternalOnly
public int getHTTPTransactionID()
```
    Gets the ID of the HTTP transaction associated with this scrapeable file.
    
    Returns:
    
    The ID.
  - setHTTPTransactionID
```
@InternalOnly
public void setHTTPTransactionID(int httpTransactionID)
```
    Sets the ID of the HTTP transaction associated with this scrapeable file.
    
    Parameters:
    
    httpTransactionID - The ID.
  - getMaxResponseLength
```
public int getMaxResponseLength()
```
    Gets the maximum response length, in kilobytes.
    
    Returns:
    
    The maximum length.
  - setMaxResponseLength
```
@RequiredVersion(value=1,
                 behavior=LOG_ERROR_AND_RETURN_NULL_NEG_ONE_OR_FALSE)
public void setMaxResponseLength(int maxKBytes)
```
    Sets the maximum response length, in kilobytes.
    
    Parameters:
    
    maxKBytes - The maximum length.
  - getLastTidyAttemptFailed
```
public boolean getLastTidyAttemptFailed()
```
    Indicates whether or not the last attempt to tidy the response failed.
    
    Returns:
    
    A boolean.
  - setLastTidyAttemptFailed
```
@InternalOnly
public void setLastTidyAttemptFailed(boolean lastTidyAttemptFailed)
```
    Sets whether or not the last attempt to tidy the response failed.
    
    Parameters:
    
    lastTidyAttemptFailed - A boolean.
  - getMaxRequestAttemptsReached
```
@RequiredVersion(value=1)
public boolean getMaxRequestAttemptsReached()
```
    Indicates whether or not the maximum requests attempts were reached on the most recent try.
  - getExtractorPatternTimedOut
```
@RequiredVersion(value=1)
public boolean getExtractorPatternTimedOut()
```
    Indicates whether or not the extractor pattern timed out on the most recent try.
    
    Returns:
    
    True iff the extractors timed out
  - getUsePlusForEncodedSpaces
```
public boolean getUsePlusForEncodedSpaces()
```
    Indicates whether or not + instead of %20 should be used to encode space characters.
    
    Returns:
    
    A boolean.
  - setUsePlusForEncodedSpaces
```
public void setUsePlusForEncodedSpaces(boolean usePlusForEncodedSpaces)
```
    Sets whether or not + instead of %20 should be used to encode space characters.
    
    Parameters:
    
    usePlusForEncodedSpaces - A boolean.
  - getForceNonBinary
```
public boolean getForceNonBinary()
```
    Indicates whether or not the contents of this response should be forced to be treated as non-binary.
    
    Returns:
    
    Whether or not it should be forced as binary.
  - setForceNonBinary
```
public void setForceNonBinary(boolean forceNonBinary)
```
    Determines whether or not the contents of this response should be forced to be treated as non-binary.
    
    Parameters:
    
    forceNonBinary - Whether or not it should be forced as binary.
  - dumpToLog
```
@InternalOnly
public void dumpToLog()
```
    Dumps the scrapeable file to the log.
  - getASPXValues
```
@Nullable
 @RequiredVersion(value=1)
public DataRecord getASPXValues(boolean onlyStandard)
```
    Gets the ASPX .NET values from the string. The standard values are __VIEWSTATE, __EVENTTARGET, __EVENTVALIDATION, and __EVENTARGUMENT. They will be stored in the returned DataRecord as ASPX_VIEWSTATE, ASPX_EVENTTARGET, etc...
    
    Parameters:
    
    onlyStandard - Sets whether or not to only get the four standard tags, or look for any tags that begin with __
    
    Returns:
    
    A DataRecord object with each ASPX name as ASPX_[NAME] mapped to it's value
  - getHTTPResponseHeaders
```
@NotNull
 @RequiredVersion(value=1)
public java.util.Map<java.lang.String,java.lang.String> getHTTPResponseHeaders()
```
    Gets the headers of the HTTP Response as a map, and returns them.
    
    Returns:
    
    A Map from header name to header value for the response headers
    
    Throws:
    
    java.lang.Exception - When the version of Screen-scraper doesn't allow use of this feature
  - getHTTPResponseHeaderSection
```
@NotNull
 @RequiredVersion(value=1)
public java.lang.String getHTTPResponseHeaderSection()
```
    Gets the header section of the HTTP Response
    
    Returns:
    
    A String containing the HTTP Response Headers
    
    Throws:
    
    java.lang.Exception - When the version of Screen-scraper doesn't allow use of this feature
  - getHTTPResponseHeader
```
@Nullable
 @RequiredVersion(value=1)
public java.lang.String getHTTPResponseHeader(java.lang.String header)
```
    Gets the value of the header in the response of the scrapeable file, or returns null if it couldn't be found
    
    Parameters:
    
    header - The header name (case-insensitive) to get
    
    Returns:
    
    The value of the header, or null if not found
    
    Throws:
    
    java.lang.Exception - When the version of Screen-scraper doesn't allow use of this feature
  - setRetryPolicy
```
@RequiredVersion(value=1,
                 behavior=LOG_ERROR_AND_RETURN_NULL_NEG_ONE_OR_FALSE)
public void setRetryPolicy(RetryPolicy policy)
```
    Sets a Retry Policy that will be run to check if a page should be re-downloaded or not. The policy will be checked after all the extractors have run, and will check for an error on the page based on a set of conditions. If the policy shows an error on the page, it can run scripts or other code to attempt to remedy the situation, and then it will rescrape the file.
    
    The file will be re-downloaded without rerunning any of the scripts that run before the file is downloaded, and before any of the scripts marked to run after the file is scraped. If there is any change that needs to be made to session variables/headers, etc... they should be made in the script or runnable that will be executed. Also, the policy can specify that session variables should be restored to their previous values before the file is rescraped. If it does, they will be reset after the error checking portion of the policy but before the policy runs the code to make changes before a retry.
    
    The retry policy should be set in a script run 'Before file is scraped', but can also be set by a script on an extractor pattern. It it is set on an extractor pattern, session variables will not be restored if the retry is required
    
    Parameters:
    
    policy - The policy that should be run. See the RetryPolicyFactory for standard policies, or one can be created by implementing the RetryPolicy interface
  - getRetryPolicy
```
@Nullable
 @RequiredVersion(value=1,
                 behavior=LOG_ERROR_AND_RETURN_NULL_NEG_ONE_OR_FALSE)
public RetryPolicy getRetryPolicy()
```
    Returns the retry policy. Note that in any 'After file is scraped' scripts this is null
    
    Returns:
    
    The Retry Policy that will be used by this scrapeable file
  - getIdentifier
```
@Nullable
 @InternalOnly
public java.lang.String getIdentifier()
```
    Description copied from interface: Uniqueable
    
    Gets the identifier for this object.
    
    Specified by:
    
    getIdentifier in interface Uniqueable
    
    Returns:
    
    The identifier.
  - setIdentifier
```
@InternalOnly
public void setIdentifier(java.lang.String identifier)
```
    Description copied from interface: Uniqueable
    
    Sets the identifier for this object.
    
    Specified by:
    
    setIdentifier in interface Uniqueable
    
    Parameters:
    
    identifier - The identifier.
  - buildForm
```
@Nullable
 @RequiredVersion(value=1,
                 behavior=LOG_ERROR_AND_RETURN_NULL_NEG_ONE_OR_FALSE)
public Form buildForm(@NotNull
                                                                                                                                    java.lang.String formText)
```
    Builds a form using the text given, which should contain all the form values (including the form open and close tags)
    
    Parameters:
    
    formText - The HTML containing the form tag and all inputs in it
    
    Returns:
    
    The Form object holding all the values in this form
  - setParametersFromForm
```
@RequiredVersion(value=1,
                 behavior=LOG_ERROR_AND_RETURN_NULL_NEG_ONE_OR_FALSE)
public void setParametersFromForm(@Nullable
                                                                                                                                    Form form)
```
    Sets the parameters for a ScrapeableFile to the values in the form object. This includes updating the URL if it is set. It will also clear any previously set parameters in the scrapeable file. To manually set certain values, use methods available in the Form object.
    
    Parameters:
    
    form - The form with all the values to use for parameters as well as the URL
  - getShouldHighlightSyntax
```
@InternalOnly
public boolean getShouldHighlightSyntax()
```
    Returns:
    
    A boolean representing whether or not the HTML on the Last Response pane should be syntax highlighted.
  - setShouldHighlightSyntax
```
@InternalOnly
public void setShouldHighlightSyntax(boolean b)
```
    Sets the boolean representing whether or not the HTML on the Last Response pane should be syntax highlighted to the given value.
    
    Parameters:
    
    b - the boolean that will determine if HTML on the Last Response pane is syntax highlighted.

Class ScrapeableFile

Nested Class Summary

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

log

extractorPatternsMatched

Constructor Detail

ScrapeableFile

ScrapeableFile

ScrapeableFile

ScrapeableFile

ScrapeableFile

ScrapeableFile

ScrapeableFile

ScrapeableFile

ScrapeableFile

Method Detail

getID

getName

setName

getURL

setURL

getCurrentURL

setCurrentURL

getResolvedURL

getSequence

setSequence

getBASICAuthenticationUsername

setBASICAuthenticationUsername

getBASICAuthenticationPassword

setBASICAuthenticationPassword

getNTLMAuthenticationDomain

setNTLMAuthenticationDomain

setNTLMAuthenticationHost

getNTLMAuthenticationHost

setLastHttpResponse

getLastHttpResponse

getLastRequest

getLastRequestShort

setLastRequest

wasErrorOnRequest

inputOutputErrorOccurred

noExtractorPatternsMatched

getContentAsString

getStrippedContentAsString

getLastScrapedData

getContentBodyOnly

getLastScrapedDataShort

setLastScrapedData

setLastRequestURL

getLastRequestURL

getScrapingSession

setScrapingSession

setNode

getNode

getRunningScrapingSession

setRunningScrapingSession

getHTTPParameters

getHTTPParameterByName

addGETHTTPParameter

addGETHTTPParameter

addPOSTHTTPParameter

addPOSTHTTPParameter

addHTTPParameter

resequenceHTTPParameter

bumpHTTPParameterSequences

removeHTTPParameter

removeHTTPParameter

removeHTTPParameter

removeAllHTTPParameters

dumpHTTPParametersToLog

getHTTPParameter

getHTTPParameter

addExtractorPattern

removeExtractorPattern

getExtractorPattern

getExtractorPattern