ScrapingUtil

java.lang.Object
- com.screenscraper.util.ScrapingUtil

public class ScrapingUtil
extends java.lang.Object

Constructor Summary

Constructors
Constructor and Description

ScrapingUtil(ScrapingSession scrapingSession)
Creates a new scraping util

Constructors
Constructor and Description
`ScrapingUtil(ScrapingSession scrapingSession)` Creates a new scraping util

Method Summary

All Methods Instance Methods Concrete Methods Deprecated Methods
Modifier and Type	Method and Description
`boolean`	`alphabetLoopWithSubdivision(java.lang.String letters, java.util.function.Function<java.lang.String,java.lang.Boolean> function)` Runs a subdividing search by alphabet letters, a-z (lowercase).
`boolean`	`alphabetLoopWithSubdivision(java.lang.String letters, java.util.function.Function<java.lang.String,java.lang.Boolean> function, boolean includeAlpha, boolean includeDigits, java.lang.String skipToLetters)` Runs a subdividing search by alphabet letters, a-z (lowercase).
`boolean`	`alphabetLoopWithSubdivision(java.lang.String firstLetters, java.lang.String lastLetters, java.util.function.BiFunction<java.lang.String,java.lang.String,java.lang.Boolean> function)` Runs a subdividing search by alphabet letters, a-z (lowercase).
`boolean`	`alphabetLoopWithSubdivision(java.lang.String firstLetters, java.lang.String lastLetters, java.util.function.BiFunction<java.lang.String,java.lang.String,java.lang.Boolean> function, boolean includeAlpha, boolean includeDigits, java.lang.String skipToFirstLetters, java.lang.String skipToLastLetters)` Runs a subdividing search by alphabet letters, a-z (lowercase).
`XmlNode`	`applyXPathExpression(java.lang.String content, java.lang.String expression)` Applies an XPath expression to a string.
`java.lang.String`	`convertDateToString(java.util.Date date)` Converts the Date given to a string in the DEFAULT_DATE_FORMAT
`java.lang.String`	`convertDateToString(java.util.Date date, java.lang.String format)` Converts the Date given to a string in a specified format, or in the DEFAULT_DATE_FORMAT if the format is null
`java.lang.String`	`convertFromUpperCase(java.lang.String text)` Converts things such as FIELD_NAME to Field Name
`java.lang.String`	`convertHTMLEntities(java.lang.String val)` Replaces HTML entities in a string with their non-entity equivalents.
`java.util.Date`	`convertStringToDate(java.lang.String dateString, java.lang.String format)` Converts a date string to a Date object using the given format.
`java.lang.String`	`convertUTFWhitespace(java.lang.String input)` Replaces the UTF variants on whitespace with a regular space character
`boolean`	`dateIsWithinDays(java.util.Date date1, java.util.Date date2, int days)` Checks to see if one date is within a certain number of days of another
`boolean`	`equalsIgnoreCase(java.lang.String string1, java.lang.String string2)` Compare the values of two strings independent of case.
`java.lang.String`	`extractXmlBlock(java.lang.String content, int startingDepth)` Extracts a section of xml (roughly) by counting start and end tags.
`java.lang.String`	`formatNumber(java.lang.String number)` Returns a number formatted in such a way that it could be parsed as a Float, such as xxxxxxxxx.xxxx It attempts to figure out if the number is formatted as European or American style, but if it cannot determine which it is, it defaults to American.
`java.lang.String`	`formatNumber(java.lang.String number, boolean powerOfThousandConversion)` Returns a number formatted in such a way that it could be parsed as a Float, such as xxxxxxxxx.xxxx It attempts to figure out if the number is formatted as European or American style, but if it cannot determine which it is, it defaults to American.
`java.lang.String`	`formatNumber(java.lang.String number, int decimals, boolean padDecimals)` Returns a number formatted in such a way that it could be parsed as a Float, such as xxxxxxxxx.xxxx It attempts to figure out if the number is formatted as European or American style, but if it cannot determine which it is, it defaults to American.
`java.lang.String`	`formatNumber(java.lang.String number, int decimals, boolean padDecimals, boolean powerOfThousandConversion)` Returns a number formatted in such a way that it could be parsed as a Float, such as xxxxxxxxx.xxxx It attempts to figure out if the number is formatted as European or American style, but if it cannot determine which it is, it defaults to American.
`java.lang.String`	`formatUSPhoneNumber(java.lang.String number)` Converts a String to a US formatted phone number, as +1 (123) 456-7890x2.
`java.lang.String`	`formatUSZipCode(java.lang.String zip)` Formats and returns a US style zip code as 12345-6789.
`int`	`getCpuUsage()` Gets the current cpu usage.
`java.lang.String`	`getCurrentDate(java.lang.String format)` Returns the current date in a specified format, or uses the DEFAULT_DATE_FORMAT if null is given.
`java.util.Map<java.lang.String,java.lang.Boolean>`	`getDefaultTidySettings()` Get the default tidy settings, as a map
`int`	`getImageHeight(java.lang.String imagePath)` Gets the height of an image.
`int`	`getImageWidth(java.lang.String imagePath)` Gets the width of an image.
`java.lang.String`	`getInstallDir()` Gets the path to the directory where screen-scraper is installed.
`org.json.JSONArray`	`getJSONArray(org.json.JSONObject json, java.lang.String key)` Gets a json array from a json object with a given key, or returns null if there is no such value, the input is null.
`org.json.JSONObject`	`getJSONObject(org.json.JSONObject json, java.lang.String key)` Gets a json object from a json object with a given key, or returns null if there is no such value, the input is null, or the value is null or not a json object
`java.lang.String`	`getJSONString(org.json.JSONObject json, java.lang.String key)` Gets a string from a json object.
`java.lang.String`	`getJSONString(org.json.JSONObject json, java.lang.String key, java.lang.String defaultValue)` Gets a string from a json object.
`boolean`	`getLogInternalMessages()`
`int`	`getMaximumMemoryAllocation()` Gets the value of the MaximumMemoryAllocation property.
`int`	`getMemoryUsage()` Gets the average memory usage.
`java.lang.String`	`getMimeType(java.lang.String path)` Gets the MIME-type for a file.
`int`	`getNumRunnableScrapingSessions()` Gets the number of scraping sessions that are queued up to be run.
`int`	`getNumRunningScrapingSessions()` Gets the number of scraping sessions that are currently being run.
`int`	`getNumRunningScrapingSessions(java.lang.String scrapingSessionName)` Gets the number of named running scraping sessions.
`DataSet`	`getOptionSet(java.lang.String options)` Gets a DataSet containing each of the elements of a <select> tag.
`DataSet`	`getOptionSet(java.lang.String options, java.util.Collection<java.lang.String> ignoreLabels, boolean tidyRecords)` Gets a DataSet containing each of the elements of a <select> tag.
`DataSet`	`getOptionSet(java.lang.String options, java.lang.String ignoreLabel)` Gets a DataSet containing each of the elements of a <select> tag.
`DataSet`	`getOptionSet(java.lang.String options, java.lang.String[] ignoreLabels, boolean tidyRecords)` Gets a DataSet containing each of the elements of a <select> tag.
`DataSet`	`getOptionSet(java.lang.String options, java.lang.String ignoreLabel, boolean tidyRecords)` Gets a DataSet containing each of the elements of a <select> tag.
`DataSet`	`getRadioButtonSet(java.lang.String buttons, java.lang.String buttonName)` Gets all the options from a radio button group.
`DataSet`	`getRadioButtonSet(java.lang.String buttons, java.lang.String buttonName, java.util.Collection<java.lang.String> ignoreLabels)` Gets all the options from a radio button group.
`DataSet`	`getRadioButtonSet(java.lang.String buttons, java.lang.String buttonName, java.util.Collection<java.lang.String> ignoreLabels, boolean tidyRecords)` Gets all the options from a radio button group.
`DataSet`	`getRadioButtonSet(java.lang.String buttons, java.lang.String buttonName, java.lang.String ignoreLabel)` Gets all the options from a radio button group.
`DataSet`	`getRadioButtonSet(java.lang.String buttons, java.lang.String buttonName, java.lang.String[] ignoreLabels)` Gets all the options from a radio button group.
`java.lang.String`	`getRandomReferrer()` Gets a random referrer page from a list of many different search engine web sites and a few other pages
`java.lang.String`	`getRandomUserAgent()` Returns a random User Agent.
`java.lang.String`	`getRandomUserAgent(BrowserType browserType, java.time.LocalDate newerThan, java.time.LocalDate olderThan)` Returns a random User Agent.
`java.lang.String`	`getScreenScraperEdition()` Gets the edition of this instance.
`java.lang.String`	`getScreenScraperVersion()` Gets the version of this instance.
`java.lang.Object`	`getTimestampObject()` Returns an object whose toString method always resolves to the current timestamp.
`boolean`	`isInt(java.lang.Object string)` Indicates whether or not the given `String` can be parsed as an int.
`boolean`	`isNullOrEmptyString(java.lang.Object object)` Indicates whether or not an object is either null or an empty string.
`boolean`	`isPlatformLinux()` Indicates whether or not we're running on linux.
`boolean`	`isPlatformMac()` Indicates whether or not we're running on a Mac.
`boolean`	`isPlatformWindows()` Indicates whether or not we're running on Windows.
`double`	`linear(double number, double minIn, double maxIn, double minOut, double maxOut)` Performs a linear interpolation between two values and returns the result.
`void`	`makeDELETERequest(java.lang.String urlString)` Makes a DELETE request.
`void`	`makeDELETERequest(java.lang.String urlString, java.util.Map headers)` Makes a DELETE request.
`java.lang.String`	`makeGETRequest(java.lang.String urlString)` Makes a GET request and returns the result as a string.
`java.lang.String`	`makeGETRequestNoSessionProxy(java.lang.String urlString)` Makes a GET request and returns the result as a string.
`java.lang.String`	`makeGETRequestUseSessionProxy(java.lang.String urlString)` Makes a GET request and returns the result as a string.
`java.lang.String[][]`	`makeHEADRequest(java.lang.String urlString)` Makes a HEAD request and returns the headers in a two-deminsional array.
`void`	`makePUTRequest(java.lang.String urlString)` Makes a PUT request.
`void`	`makePUTRequest(java.lang.String urlString, java.util.Map headers)` Makes a PUT request.
`DataRecord`	`mergeDataRecords(DataRecord first, DataRecord second, boolean saveNonEmptyString)` Merges two data records by copying all values from the second record over values of the first record, and returning a new DataRecord with these values.
`java.lang.String`	`nullToEmptyString(java.lang.Object obj)` Turns a potentially null value into an empty string.
`Name`	`parseName(java.lang.String name)` Attempts to parse a string to a name.
`Address`	`parseUSAddress(java.lang.String address)` Attempts to parse a string to an address.
`void`	`pause(long time)` Indicates that the scraping session should pause for the given number of milliseconds.
`void`	`randomPause(long min, long max)` Pauses for a random amount of time.
`java.lang.String`	`reformatDate(java.lang.String date, java.lang.String dateFormatTo)` Reformat a date.
`java.lang.String`	`reformatDate(java.lang.String date, java.lang.String dateFormatFrom, java.lang.String dateFormatTo)` Reformats a date.
`Address`	`reorderUSAddressBestGuessFromLines(java.lang.String addressee, java.lang.String address, java.lang.String address2)` Attempts to reorder the addressee, address, and address2 lines using the address parser to attempt to validate each part.
`void`	`resizeImageFixHeight(java.lang.String originalFile, java.lang.String thumbnailFile, int newHeightSize, boolean deleteOriginalFile)` Creates a thumbnail for an existing image.
`void`	`resizeImageFixWidth(java.lang.String originalFile, java.lang.String thumbnailFile, int newWidthSize, boolean deleteOriginalFile)` Creates a thumbnail for an existing image.
`void`	`resizeImageFixWidthAndHeight(java.lang.String originalFile, java.lang.String thumbnailFile, int newWidthSize, int newHeightSize, boolean deleteOriginalFile)` Creates a thumbnail for an existing image.
`void`	`runMultipleScrapes(java.lang.String scrapeName, java.lang.String key, java.util.List<?> values, int maxConcurrentScrapes)` Deprecated.
`void`	`runMultipleScrapes(java.lang.String scrapeName, java.lang.String key, java.util.List<?> values, int maxConcurrentScrapes, boolean shouldInheritFromParent)` Deprecated.
`void`	`runMultipleScrapes(java.lang.String scrapeName, java.lang.String key, java.lang.Object[] values, int maxConcurrentScrapes)` Deprecated.
`void`	`runMultipleScrapes(java.lang.String scrapeName, java.lang.String key, java.lang.String file, int maxConcurrentScrapes)` Deprecated.
`<T> void`	`runMultipleSessionsConcurrentlyForInput(int numToRun, java.util.function.Function<T,java.util.List<T>> individualRunFunction, java.util.concurrent.Callable<java.util.function.Supplier<T>> searchInputSupplierInitializer)` Runs multiple scraping sessions in parallel over the same input data.
`<T> void`	`runMultipleSessionsConcurrentlyForInput(int numToRun, int queueSize, java.util.function.Function<T,java.util.List<T>> processInputFunction, java.util.concurrent.Callable<java.util.function.Supplier<T>> supplierInitializer)` Runs multiple scraping sessions in parallel over the same input data.
`void`	`sendMail(java.lang.String subject, java.lang.String body, java.lang.String recipients)` Sends mail to a pre-specified SMTP server.
`void`	`sendMail(java.lang.String subject, java.lang.String body, java.lang.String recipients, java.lang.String attachments, java.lang.String headers)` Sends mail to a pre-specified SMTP server.
`void`	`sendMail(java.lang.String subject, java.lang.String body, java.lang.String recipients, java.lang.String contentType, java.lang.String attachments, java.lang.String headers)` Sends mail to a pre-specified SMTP server.
`void`	`setLogInternalMessages(boolean shouldLog)`
`void`	`setupAutoIncrementParentScrapeRecordsFromChild(ScrapingSession parent)` Sets up events to automatically increment the records for a parent scrape when the current scrape changes records count.
`<T> java.util.List<T>`	`sortSet(java.util.Set<T> set)` Sorts the elements in a set into an ordered list
`<T> java.util.List<T>`	`sortSet(java.util.Set<T> set, boolean ignoreCase)` Sorts the elements in a set into an ordered list
`<T> java.util.List<T>`	`sortSet(java.util.Set<T> set, java.util.Comparator<? super T> comparator)` Sorts the elements in a set into an ordered list
`boolean`	`startsWithUpperCase(java.lang.String string1, java.lang.String string2)` Compare the uppercase startsWith values of two strings
`float`	`stringToFloat(java.lang.String str)` Takes a string and attempts to turn it into a float.
`java.lang.String`	`stripHTML(java.lang.String content)` Strips HTML from a string, replacing some tags with corresonding text-only equivalents.
`DataRecord`	`tidyDataRecord(DataRecord record)` Tidies the DataRecord by performing actions based on the values of the settings map given (or getDefaultTidySettings if none is given).
`DataRecord`	`tidyDataRecord(DataRecord record, boolean ignoreLowerCaseKeys)` Tidies the DataRecord by performing actions based on the values of the settings map given (or getDefaultTidySettings if none is given).
`DataRecord`	`tidyDataRecord(DataRecord record, java.util.Map<java.lang.String,java.lang.Boolean> settings)` Tidies the DataRecord by performing actions based on the values of the settings map given (or getDefaultTidySettings if none is given).
`DataRecord`	`tidyDataRecord(DataRecord record, java.util.Map<java.lang.String,java.lang.Boolean> settings, boolean ignoreLowerCaseKeys)` Tidies the DataRecord by performing actions based on the values of the settings map given (or getDefaultTidySettings if none is given).
`DataRecord`	`tidyDataRecord(ScrapeableFile scrapeableFile, DataRecord record)` Tidies the DataRecord by performing actions based on the values of the settings map given (or getDefaultTidySettings if none is given).
`DataRecord`	`tidyDataRecord(ScrapeableFile scrapeableFile, DataRecord record, boolean ignoreLowerCaseKeys)` Tidies the DataRecord by performing actions based on the values of the settings map given (or getDefaultTidySettings if none is given).
`DataRecord`	`tidyDataRecord(ScrapeableFile scrapeableFile, DataRecord record, java.util.Map<java.lang.String,java.lang.Boolean> settings)` Tidies the DataRecord by performing actions based on the values of the settings map given (or getDefaultTidySettings if none is given).
`DataRecord`	`tidyDataRecord(ScrapeableFile scrapeableFile, DataRecord record, java.util.Map<java.lang.String,java.lang.Boolean> settings, boolean ignoreLowerCaseKeys)` Tidies the DataRecord by performing actions based on the values of the settings map given (or getDefaultTidySettings if none is given).
`java.lang.String`	`tidyString(ScrapeableFile scrapeableFile, java.lang.String value)` Tidies the string by performing actions based on the values of the settings map
`java.lang.String`	`tidyString(ScrapeableFile scrapeableFile, java.lang.String value, java.util.Map<java.lang.String,java.lang.Boolean> settings)` Tidies the string by performing actions based on the values of the settings map
`java.lang.String`	`tidyString(java.lang.String value)` Tidies the string by performing actions based on the value of flags, using the flags as the operation list
`java.lang.String`	`tidyString(java.lang.String value, java.util.Map<java.lang.String,java.lang.Boolean> settings)` Tidies the string by performing actions based on the values of the settings map
`java.lang.String`	`toCamelCase(java.lang.String text)` Converts an input string to have each word start with an uppercase character.
`void`	`unzipFile(java.lang.String zippedFile)` Unzips a file.
`void`	`writeValueToFile(java.lang.Object value, java.lang.String file, java.lang.String charSet)` Writes a value to a file using the given character set.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Constructor Detail
  - ScrapingUtil
```
public ScrapingUtil(@Nullable
                    ScrapingSession scrapingSession)
```
    Creates a new scraping util
    
    Parameters:
    
    scrapingSession - The session to use
- Method Detail
  - reformatDate
```
@Nullable
 @RequiredVersion(value=1)
public java.lang.String reformatDate(@Nullable
                                                                            java.lang.String date,
                                                                            @Nullable
                                                                            java.lang.String dateFormatFrom,
                                                                            @Nullable
                                                                            java.lang.String dateFormatTo)
                                                                     throws java.lang.Exception
```
    Reformats a date.
    
    Parameters:
    
    date - The date to be formatted, as a string.
    
    dateFormatFrom - The pattern of the existing date.
    
    dateFormatTo - The pattern the date should be converted to.
    
    Returns:
    
    The re-formatted date, if it can be parsed; otherwise, returns the original date.
    
    Throws:
    
    java.lang.Exception - If either of the date formats are invalid.
  - reformatDate
```
@Nullable
 @RequiredVersion(value=2)
public java.lang.String reformatDate(@Nullable
                                                                            java.lang.String date,
                                                                            @Nullable
                                                                            java.lang.String dateFormatTo)
                                                                     throws java.lang.Exception
```
    Reformat a date.
    
    Parameters:
    
    date - The date to be formatted, as a string.
    
    dateFormatTo - The pattern the date should be converted to.
    
    Returns:
    
    The re-formatted date, if it can be parsed; otherwise, returns the original date.
    
    Throws:
    
    java.lang.Exception - If either of the date formats are invalid.
  - getImageHeight
```
@RequiredVersion(value=2)
public int getImageHeight(@NotNull
                                                     java.lang.String imagePath)
                                              throws java.lang.Exception
```
    Gets the height of an image.
    
    Parameters:
    
    imagePath - The path to the image.
    
    Returns:
    
    An int, representing the height of the image.
    
    Throws:
    
    java.lang.Exception - If an error occurs while reading the image
  - getImageWidth
```
@RequiredVersion(value=2)
public int getImageWidth(@NotNull
                                                    java.lang.String imagePath)
                                             throws java.lang.Exception
```
    Gets the width of an image.
    
    Parameters:
    
    imagePath - The path to the image.
    
    Returns:
    
    An int, representing the width of the image.
    
    Throws:
    
    java.lang.Exception - If an error occurs while reading the image
  - getMaximumMemoryAllocation
```
@RequiredVersion(value=2)
public int getMaximumMemoryAllocation()
```
    Gets the value of the MaximumMemoryAllocation property.
    
    Returns:
    
    The value as an int.
  - getMemoryUsage
```
@RequiredVersion(value=2)
public int getMemoryUsage()
```
    Gets the average memory usage.
    
    Returns:
    
    The memory usage as a whole number percentage.
  - getCpuUsage
```
@RequiredVersion(value=2)
public int getCpuUsage()
```
    Gets the current cpu usage.
    
    Returns:
    
    The cpu usage as a whole number percentage.
  - pause
```
@RequiredVersion(value=1)
public void pause(long time)
```
    Indicates that the scraping session should pause for the given number of milliseconds.
    
    Parameters:
    
    time - The amount of time to pause, in milliseconds.
  - resizeImageFixWidth
```
@RequiredVersion(value=2)
public void resizeImageFixWidth(@NotNull
                                                           java.lang.String originalFile,
                                                           @NotNull
                                                           java.lang.String thumbnailFile,
                                                           int newWidthSize,
                                                           boolean deleteOriginalFile)
                                                    throws java.lang.Exception
```
    Creates a thumbnail for an existing image. If an exception occurs in the process the original file will not be deleted.
    
    Parameters:
    
    originalFile - The path to the original file.
    
    thumbnailFile - The path where the thumbnail should be created.
    
    newWidthSize - The new width of the image.
    
    deleteOriginalFile - Whether or not the original file should be deleted after the thumbnail is created.
    
    Throws:
    
    java.lang.Exception - If an error occurrs while resizing the image
  - resizeImageFixHeight
```
@RequiredVersion(value=2)
public void resizeImageFixHeight(@NotNull
                                                            java.lang.String originalFile,
                                                            @NotNull
                                                            java.lang.String thumbnailFile,
                                                            int newHeightSize,
                                                            boolean deleteOriginalFile)
                                                     throws java.lang.Exception
```
    Creates a thumbnail for an existing image. If an exception occurs in the process the original file will not be deleted.
    
    Parameters:
    
    originalFile - The path to the original file.
    
    thumbnailFile - The path where the thumbnail should be created.
    
    newHeightSize - The new height of the image.
    
    deleteOriginalFile - Whether or not the original file should be deleted after the thumbnail is created.
    
    Throws:
    
    java.lang.Exception - If an error occurrs while resizing the image
  - resizeImageFixWidthAndHeight
```
@RequiredVersion(value=2)
public void resizeImageFixWidthAndHeight(@NotNull
                                                                    java.lang.String originalFile,
                                                                    @NotNull
                                                                    java.lang.String thumbnailFile,
                                                                    int newWidthSize,
                                                                    int newHeightSize,
                                                                    boolean deleteOriginalFile)
                                                             throws java.lang.Exception
```
    Creates a thumbnail for an existing image. If an exception occurs in the process the original file will not be deleted.
    
    Parameters:
    
    originalFile - The path to the original file.
    
    thumbnailFile - The path where the thumbnail should be created.
    
    newWidthSize - The new width of the image.
    
    newHeightSize - The new height of the image.
    
    deleteOriginalFile - Whether or not the original file should be deleted after the thumbnail is created.
    
    Throws:
    
    java.lang.Exception - If an error occurrs while resizing the image
  - sendMail
```
@RequiredVersion(value=2)
public void sendMail(java.lang.String subject,
                                                java.lang.String body,
                                                java.lang.String recipients)
                                         throws java.lang.Exception
```
    Sends mail to a pre-specified SMTP server. All exceptions will be caught by the user of this method.
    
    Parameters:
    
    subject - The subject line of the e-mail.
    
    body - The text of the body of the e-mail.
    
    recipients - A comma separated list of recipients.
    
    Throws:
    
    java.lang.Exception - On error sending the mail
  - sendMail
```
@RequiredVersion(value=2)
public void sendMail(java.lang.String subject,
                                                java.lang.String body,
                                                java.lang.String recipients,
                                                java.lang.String attachments,
                                                java.lang.String headers)
                                         throws java.lang.Exception
```
    Sends mail to a pre-specified SMTP server. All exceptions will be caught by the user of this method.
    
    Parameters:
    
    subject - The subject line of the e-mail.
    
    body - The text of the body of the e-mail.
    
    recipients - A comma separated list of recipients.
    
    attachments - A comma separated list of local files to be attached.
    
    headers - A tab separated list of additional headers one would like to add to the e-mail.
    
    Throws:
    
    java.lang.Exception - On error sending the mail
  - sendMail
```
@RequiredVersion(value=2)
public void sendMail(@Nullable
                                                java.lang.String subject,
                                                @Nullable
                                                java.lang.String body,
                                                @Nullable
                                                java.lang.String recipients,
                                                @Nullable
                                                java.lang.String contentType,
                                                @Nullable
                                                java.lang.String attachments,
                                                @Nullable
                                                java.lang.String headers)
                                         throws java.lang.Exception
```
    Sends mail to a pre-specified SMTP server. All exceptions will be caught by the user of this method.
    
    Parameters:
    
    subject - The subject line of the e-mail.
    
    body - The text of the body of the e-mail.
    
    recipients - A comma separated list of recipients.
    
    contentType - The content type as a valid MIME type.
    
    attachments - A comma separated list of local files to be attached.
    
    headers - A tab separated list of additional headers one would like to add to the e-mail.
    
    Throws:
    
    java.lang.Exception - If there is an error preparing or sending the email
  - isNullOrEmptyString
```
public boolean isNullOrEmptyString(java.lang.Object object)
```
    Indicates whether or not an object is either null or an empty string.
    
    Parameters:
    
    object - The Object to test.
    
    Returns:
    
    A boolean.
  - equalsIgnoreCase
```
public boolean equalsIgnoreCase(java.lang.String string1,
                                java.lang.String string2)
```
    Compare the values of two strings independent of case.
    
    Parameters:
    
    string1 - The string1 to compare.
    
    string2 - The string2 to compare.
    
    Returns:
    
    The boolean if the uppercase of the strings equals
  - startsWithUpperCase
```
public boolean startsWithUpperCase(java.lang.String string1,
                                   java.lang.String string2)
```
    Compare the uppercase startsWith values of two strings
    
    Parameters:
    
    string1 - The string1 to compare.
    
    string2 - The string2 to compare.
    
    Returns:
    
    The boolean if the startsWith uppercase of the strings equals
  - isInt
```
public boolean isInt(java.lang.Object string)
              throws java.lang.Exception
```
    Indicates whether or not the given String can be parsed as an int.
    
    Parameters:
    
    string - The String to check.
    
    Returns:
    
    A boolean.
    
    Throws:
    
    java.lang.Exception - If the input value isn't a string
  - nullToEmptyString
```
public java.lang.String nullToEmptyString(java.lang.Object obj)
```
    Turns a potentially null value into an empty string. This function is to be used when submitting information to a database so that an empty string gets submitted instead of the word "null". Also handles escaping out any quotes.
    
    Parameters:
    
    obj - A String that may be null.
    
    Returns:
    
    An empty string if the value is null, the original string, otherwise.
  - writeValueToFile
```
public void writeValueToFile(@NotNull
                             java.lang.Object value,
                             @NotNull
                             java.lang.String file,
                             java.lang.String charSet)
                      throws java.lang.Exception
```
    Writes a value to a file using the given character set. If no character set is given the default character set defined in the "Settings" dialog box will be used.
    
    Parameters:
    
    value - The string to be written.
    
    file - The name of the file.
    
    charSet - The character set to use.
    
    Throws:
    
    java.lang.Exception - On error writing to the file
  - convertHTMLEntities
```
@Nullable
 @Contract(value="!null -> !null; null -> null")
public java.lang.String convertHTMLEntities(@Nullable
                                                                                                         java.lang.String val)
```
    Replaces HTML entities in a string with their non-entity equivalents.
    
    Parameters:
    
    val - The string containing the entities.
    
    Returns:
    
    The string with the entities replaced out.
  - unzipFile
```
public void unzipFile(@NotNull
                      java.lang.String zippedFile)
               throws java.io.IOException
```
    Unzips a file.
    
    Parameters:
    
    zippedFile - The path to the zipped file.
    
    Throws:
    
    java.io.IOException - On error unzipping the file
  - makeGETRequestNoSessionProxy
```
@NotNull
public java.lang.String makeGETRequestNoSessionProxy(java.lang.String urlString)
                                                        throws java.lang.Exception
```
    Makes a GET request and returns the result as a string. This method will use the proxy settings indicated in the "Settings" dialog box, if any.
    
    Parameters:
    
    urlString - The URL to request, as a string.
    
    Returns:
    
    The result of the request.
    
    Throws:
    
    java.lang.Exception - If anything naughty happens.
  - makeGETRequestUseSessionProxy
```
@NotNull
public java.lang.String makeGETRequestUseSessionProxy(java.lang.String urlString)
                                                         throws java.lang.Exception
```
    Makes a GET request and returns the result as a string. This method will use the proxy settings attached to the current scraping session.
    
    Parameters:
    
    urlString - The URL to request, as a string.
    
    Returns:
    
    The result of the request.
    
    Throws:
    
    java.lang.Exception - If anything naughty happens.
  - makeGETRequest
```
@NotNull
public java.lang.String makeGETRequest(java.lang.String urlString)
                                          throws java.lang.Exception
```
    Makes a GET request and returns the result as a string.
    
    Parameters:
    
    urlString - The URL to request, as a string.
    
    Returns:
    
    The result of the request.
    
    Throws:
    
    java.lang.Exception - If anything naughty happens.
  - makeHEADRequest
```
@NotNull
public java.lang.String[][] makeHEADRequest(java.lang.String urlString)
                                               throws java.lang.Exception
```
    Makes a HEAD request and returns the headers in a two-deminsional array. This method will use the proxy settings indicated in the "Settings" dialog box, if any.
    
    Parameters:
    
    urlString - The URL to request, as a string.
    
    Returns:
    
    The resulting headers.
    
    Throws:
    
    java.lang.Exception - If anything naughty happens.
  - makeDELETERequest
```
@RequiredVersion(value=2)
public void makeDELETERequest(java.lang.String urlString)
                                                  throws java.lang.Exception
```
    Makes a DELETE request.
    
    Parameters:
    
    urlString - The URL to request, as a string.
    
    Throws:
    
    java.lang.Exception - If anything naughty happens.
  - makeDELETERequest
```
@RequiredVersion(value=2)
public void makeDELETERequest(java.lang.String urlString,
                                                         java.util.Map headers)
                                                  throws java.lang.Exception
```
    Makes a DELETE request.
    
    Parameters:
    
    urlString - The URL to request, as a string.
    
    headers - An optional Map of request headers.
    
    Throws:
    
    java.lang.Exception - If anything naughty happens.
  - makePUTRequest
```
@RequiredVersion(value=2)
public void makePUTRequest(java.lang.String urlString)
                                               throws java.lang.Exception
```
    Makes a PUT request.
    
    Parameters:
    
    urlString - The URL to request, as a string.
    
    Throws:
    
    java.lang.Exception - If anything naughty happens.
  - makePUTRequest
```
@RequiredVersion(value=2)
public void makePUTRequest(java.lang.String urlString,
                                                      java.util.Map headers)
                                               throws java.lang.Exception
```
    Makes a PUT request.
    
    Parameters:
    
    urlString - The URL to request, as a string.
    
    headers - An optional Map of request headers.
    
    Throws:
    
    java.lang.Exception - If anything naughty happens.
  - getScreenScraperVersion
```
public java.lang.String getScreenScraperVersion()
```
    Gets the version of this instance.
    
    Returns:
    
    The version.
  - getScreenScraperEdition
```
@NotNull
public java.lang.String getScreenScraperEdition()
```
    Gets the edition of this instance.
    
    Returns:
    
    The edition.
  - isPlatformLinux
```
public boolean isPlatformLinux()
```
    Indicates whether or not we're running on linux.
    
    Returns:
    
    A boolean.
  - isPlatformMac
```
public boolean isPlatformMac()
```
    Indicates whether or not we're running on a Mac.
    
    Returns:
    
    A boolean.
  - isPlatformWindows
```
public boolean isPlatformWindows()
```
    Indicates whether or not we're running on Windows.
    
    Returns:
    
    A boolean.
  - getMimeType
```
public java.lang.String getMimeType(@NotNull
                                    java.lang.String path)
                             throws java.lang.Exception
```
    Gets the MIME-type for a file.
    
    Parameters:
    
    path - The path to the file.
    
    Returns:
    
    A MIME-type. If something goes wrong "application/octet-stream" is returned.
    
    Throws:
    
    java.lang.Exception - On error determining the Mime type
  - getInstallDir
```
@Nullable
public java.lang.String getInstallDir()
```
    Gets the path to the directory where screen-scraper is installed.
    
    Returns:
    
    The path to the directory where screen-scraper is installed
  - stripHTML
```
@Nullable
 @RequiredVersion(value=2)
public java.lang.String stripHTML(java.lang.String content)
```
    Strips HTML from a string, replacing some tags with corresonding text-only equivalents.
    
    Parameters:
    
    content - The content to be stripped.
    
    Returns:
    
    The stripped content.
  - getNumRunningScrapingSessions
```
@RequiredVersion(value=2)
public int getNumRunningScrapingSessions(java.lang.String scrapingSessionName)
```
    Gets the number of named running scraping sessions.
    
    Parameters:
    
    scrapingSessionName - The name of the scraping session to check.
    
    Returns:
    
    A boolean.
  - getNumRunningScrapingSessions
```
@RequiredVersion(value=2)
public int getNumRunningScrapingSessions()
```
    Gets the number of scraping sessions that are currently being run.
    
    Returns:
    
    An int.
  - getNumRunnableScrapingSessions
```
@RequiredVersion(value=2)
public int getNumRunnableScrapingSessions()
```
    Gets the number of scraping sessions that are queued up to be run.
    
    Returns:
    
    An int.
  - stringToFloat
```
@RequiredVersion(value=1)
public float stringToFloat(@NotNull
                                                      java.lang.String str)
```
    Takes a string and attempts to turn it into a float.
    
    Parameters:
    
    str - The string to convert.
    
    Returns:
    
    A float, if successful.
  - setLogInternalMessages
```
public void setLogInternalMessages(boolean shouldLog)
```
  - getLogInternalMessages
```
public boolean getLogInternalMessages()
```
  - formatUSPhoneNumber
```
@Nullable
 @RequiredVersion(value=1)
public java.lang.String formatUSPhoneNumber(@Nullable
                                                                                   java.lang.String number)
```
    Converts a String to a US formatted phone number, as +1 (123) 456-7890x2. Expects a 7 digit or 10+ digit phone number. The extension is optional, and will be any digits found after an x. This allows for extensions listed as ext, x, or extension
    
    Parameters:
    
    number - String containing the phone number. The only digits in this String should be the digits of the phone number.
    
    Returns:
    
    String formatted as a phone number, such as +1 (123) 456-7890x2
  - formatNumber
```
@Nullable
 @RequiredVersion(value=1)
public java.lang.String formatNumber(java.lang.String number,
                                                                            int decimals,
                                                                            boolean padDecimals)
```
    Returns a number formatted in such a way that it could be parsed as a Float, such as xxxxxxxxx.xxxx It attempts to figure out if the number is formatted as European or American style, but if it cannot determine which it is, it defaults to American. If the number is something with a k on the end, it will convert the k to thousand (as 000). It will also try to convert m for million and b for billion. It also assumes that you won't have a number like 3.123k or 3.765m, however 3.54m is fine. It figures if you wanted all three of those digits you would have specified it as 3765k or 3,765k
    
    Parameters:
    
    number - The string representation of the number
    
    decimals - How many decimals to display
    
    padDecimals - Sets whether or not to force the number of decimals (ie convert 5.1 to 5.10 if 2 decimals are specified)
    
    Returns:
    
    String representation of the formatted number
  - formatNumber
```
@Nullable
 @RequiredVersion(value=1)
public java.lang.String formatNumber(java.lang.String number,
                                                                            int decimals,
                                                                            boolean padDecimals,
                                                                            boolean powerOfThousandConversion)
```
    Returns a number formatted in such a way that it could be parsed as a Float, such as xxxxxxxxx.xxxx It attempts to figure out if the number is formatted as European or American style, but if it cannot determine which it is, it defaults to American. If the number is something with a k on the end, it will convert the k to thousand (as 000). It will also try to convert m for million and b for billion. It also assumes that you won't have a number like 3.123k or 3.765m, however 3.54m is fine. It figures if you wanted all three of those digits you would have specified it as 3765k or 3,765k
    
    Parameters:
    
    number - The string representation of the number
    
    decimals - How many decimals to display
    
    padDecimals - Sets whether or not to force the number of decimals (ie convert 5.1 to 5.10 if 2 decimals are specified)
    
    powerOfThousandConversion - True if the formatter should attempt to deal with things like 5k meaning 5000. This can cause unexpected results if text is given with the number that isn't actually a power of a thousand
    
    Returns:
    
    String representation of the formatted number
  - formatNumber
```
@Nullable
 @RequiredVersion(value=1)
public java.lang.String formatNumber(java.lang.String number)
```
    Returns a number formatted in such a way that it could be parsed as a Float, such as xxxxxxxxx.xxxx It attempts to figure out if the number is formatted as European or American style, but if it cannot determine which it is, it defaults to American. If the number is something with a k on the end, it will convert the k to thousand (as 000). It will also try to convert m for million and b for billion. It also assumes that you won't have a number like 3.123k or 3.765m, however 3.54m is fine. It figures if you wanted all three of those digits you would have specified it as 3765k or 3,765k
    
    Parameters:
    
    number - The string representation of the number
    
    Returns:
    
    String representation of the formatted number
  - formatNumber
```
@Nullable
 @RequiredVersion(value=1)
public java.lang.String formatNumber(@Nullable
                                                                            java.lang.String number,
                                                                            boolean powerOfThousandConversion)
```
    Returns a number formatted in such a way that it could be parsed as a Float, such as xxxxxxxxx.xxxx It attempts to figure out if the number is formatted as European or American style, but if it cannot determine which it is, it defaults to American. If the number is something with a k on the end, it will convert the k to thousand (as 000). It can also try to convert m for million and b for billion. It also assumes that you won't have a number like 3.123k or 3.765m, however 3.54m is fine. It figures if you wanted all three of those digits you would have specified it as 3765k or 3,765k
    
    Parameters:
    
    number - The string representation of the number
    
    powerOfThousandConversion - True if the formatter should attempt to deal with things like 5k meaning 5000. This can cause unexpected results if text is given with the number that isn't actually a power of a thousand
    
    Returns:
    
    String representation of the formatted number
  - formatUSZipCode
```
@Nullable
 @RequiredVersion(value=1)
public java.lang.String formatUSZipCode(@Nullable
                                                                               java.lang.String zip)
```
    Formats and returns a US style zip code as 12345-6789. If the given zip code isn't 5 or 9 digits, will log a warning, but it will put 5 digits before the - and anything else (if any) after the -
    
    Parameters:
    
    zip - String to format as a zip code, either 5 or 9 digits
    
    Returns:
    
    Zip code formatted String, as 12345-6789 or 12345
  - convertStringToDate
```
@Nullable
 @RequiredVersion(value=1)
public java.util.Date convertStringToDate(@Nullable
                                                                                 java.lang.String dateString,
                                                                                 @Nullable
                                                                                 java.lang.String format)
```
    Converts a date string to a Date object using the given format. If null is given as a format, the DEFAULT_DATE_FORMAT is used
    
    Parameters:
    
    dateString - The date string
    
    format - The format of the date, following SimpleDateFormat formatting.
    
    Returns:
    
    The Date object matching the date given in the String
  - convertUTFWhitespace
```
@Contract(value="null -> null; !null -> !null")
 @Nullable
public java.lang.String convertUTFWhitespace(@Nullable
                                                                                                          java.lang.String input)
```
    Replaces the UTF variants on whitespace with a regular space character
    
    Parameters:
    
    input - The input string
    
    Returns:
    
    The converted string
  - convertDateToString
```
@Nullable
 @RequiredVersion(value=1)
public java.lang.String convertDateToString(java.util.Date date)
```
    Converts the Date given to a string in the DEFAULT_DATE_FORMAT
    
    Parameters:
    
    date - The date to convert
    
    Returns:
    
    A String representing the date given
  - convertDateToString
```
@Nullable
public java.lang.String convertDateToString(@Nullable
                                                       java.util.Date date,
                                                       @Nullable
                                                       java.lang.String format)
```
    Converts the Date given to a string in a specified format, or in the DEFAULT_DATE_FORMAT if the format is null
    
    Parameters:
    
    date - The date to convert
    
    format - A String representation (as a SimpleDateFormat) for the output
    
    Returns:
    
    A String representing the date given
  - getCurrentDate
```
@NotNull
public java.lang.String getCurrentDate(@Nullable
                                                 java.lang.String format)
```
    Returns the current date in a specified format, or uses the DEFAULT_DATE_FORMAT if null is given. Uses the session's timezone.
    
    Parameters:
    
    format - The format for the output string, input in the same format as a SimpleDateFormat
    
    Returns:
    
    A String representing the date and time this method was invoked
  - sortSet
```
@NotNull
public <T> java.util.List<T> sortSet(@NotNull
                                               java.util.Set<T> set)
```
    Sorts the elements in a set into an ordered list
    
    Type Parameters:
    
    T - The type of object
    
    Parameters:
    
    set - The set whose elements should be sorted
    
    Returns:
    
    A List of sorted elements
  - sortSet
```
@NotNull
public <T> java.util.List<T> sortSet(@NotNull
                                               java.util.Set<T> set,
                                               boolean ignoreCase)
```
    Sorts the elements in a set into an ordered list
    
    Type Parameters:
    
    T - The type of object
    
    Parameters:
    
    set - The set whose elements should be sorted
    
    ignoreCase - True if case is irrelevant when sorting strings
    
    Returns:
    
    A List of sorted elements
  - sortSet
```
@NotNull
public <T> java.util.List<T> sortSet(@NotNull
                                               java.util.Set<T> set,
                                               @Nullable
                                               java.util.Comparator<? super T> comparator)
```
    Sorts the elements in a set into an ordered list
    
    Type Parameters:
    
    T - The type of object
    
    Parameters:
    
    set - The set whose elements should be sorted
    
    comparator - The Comparator used to compare objects in the set to determine order
    
    Returns:
    
    A List of sorted elements
  - getDefaultTidySettings
```
@NotNull
 @InternalOnly
public java.util.Map<java.lang.String,java.lang.Boolean> getDefaultTidySettings()
```
    Get the default tidy settings, as a map
    
    Returns:
    
    The default tidy settings
  - tidyString
```
@Nullable
 @RequiredVersion(value=1)
public java.lang.String tidyString(java.lang.String value)
```
    Tidies the string by performing actions based on the value of flags, using the flags as the operation list
    
    Parameters:
    
    value - The String to tidy
    
    Returns:
    
    The tidied string
  - tidyString
```
@Nullable
 @RequiredVersion(value=1)
public java.lang.String tidyString(java.lang.String value,
                                                                          java.util.Map<java.lang.String,java.lang.Boolean> settings)
```
    Tidies the string by performing actions based on the values of the settings map
    
    Parameters:
    
    value - The String to tidy
    
    settings - The operations to perform when tidying, using a Map<String, Boolean>
    
    Returns:
    
    The tidied string
  - tidyString
```
@Nullable
 @RequiredVersion(value=1)
public java.lang.String tidyString(ScrapeableFile scrapeableFile,
                                                                          java.lang.String value)
```
    Tidies the string by performing actions based on the values of the settings map
    
    Parameters:
    
    scrapeableFile - The current ScrapeableFile, used for resolving relative URLs when tidying links
    
    value - The String to tidy
    
    Returns:
    
    The tidied string
  - tidyString
```
@Nullable
 @RequiredVersion(value=1)
public java.lang.String tidyString(@Nullable
                                                                          ScrapeableFile scrapeableFile,
                                                                          @Nullable
                                                                          java.lang.String value,
                                                                          @Nullable
                                                                          java.util.Map<java.lang.String,java.lang.Boolean> settings)
```
    Tidies the string by performing actions based on the values of the settings map
    
    Parameters:
    
    scrapeableFile - The current ScrapeableFile, used for resolving relative URLs when tidying links
    
    value - The String to tidy
    
    settings - The operations to perform when tidying, using a Map<String, Boolean>
    
    Returns:
    
    The tidied string
  - tidyDataRecord
```
@Nullable
 @RequiredVersion(value=1)
public DataRecord tidyDataRecord(DataRecord record)
```
    Tidies the DataRecord by performing actions based on the values of the settings map given (or getDefaultTidySettings if none is given). Each value in the record that is a string will be tidied. Keys are ignored. The record given will not be modified, but a new record with the tidied values will be returned.
    
    Parameters:
    
    record - The DataRecord to tidy (values in the record will be overwritten with the tidied values)
    
    Returns:
    
    A new DataRecord containing the tidied values
  - tidyDataRecord
```
@Nullable
 @RequiredVersion(value=1)
public DataRecord tidyDataRecord(DataRecord record,
                                                                        boolean ignoreLowerCaseKeys)
```
    Tidies the DataRecord by performing actions based on the values of the settings map given (or getDefaultTidySettings if none is given). Each value in the record that is a string will be tidied. Keys are ignored. The record given will not be modified, but a new record with the tidied values will be returned.
    
    Parameters:
    
    record - The DataRecord to tidy (values in the record will be overwritten with the tidied values)
    
    ignoreLowerCaseKeys - True if values with keys containing lowercase characters should be ignored
    
    Returns:
    
    A new DataRecord containing the tidied values
  - tidyDataRecord
```
@Nullable
 @RequiredVersion(value=1)
public DataRecord tidyDataRecord(DataRecord record,
                                                                        java.util.Map<java.lang.String,java.lang.Boolean> settings)
```
    Tidies the DataRecord by performing actions based on the values of the settings map given (or getDefaultTidySettings if none is given). Each value in the record that is a string will be tidied. Keys are ignored. The record given will not be modified, but a new record with the tidied values will be returned.
    
    Parameters:
    
    record - The DataRecord to tidy (values in the record will be overwritten with the tidied values)
    
    settings - The operations to perform when tidying, using a Map<String, Boolean>
    
    Returns:
    
    A new DataRecord containing the tidied values
  - tidyDataRecord
```
@Nullable
 @RequiredVersion(value=1)
public DataRecord tidyDataRecord(DataRecord record,
                                                                        java.util.Map<java.lang.String,java.lang.Boolean> settings,
                                                                        boolean ignoreLowerCaseKeys)
```
    Tidies the DataRecord by performing actions based on the values of the settings map given (or getDefaultTidySettings if none is given). Each value in the record that is a string will be tidied. Keys are ignored. The record given will not be modified, but a new record with the tidied values will be returned.
    
    Parameters:
    
    record - The DataRecord to tidy (values in the record will be overwritten with the tidied values)
    
    settings - The operations to perform when tidying, using a Map<String, Boolean>
    
    ignoreLowerCaseKeys - True if values with keys containing lowercase characters should be ignored
    
    Returns:
    
    A new DataRecord containing the tidied values
  - tidyDataRecord
```
@Nullable
 @RequiredVersion(value=1)
public DataRecord tidyDataRecord(ScrapeableFile scrapeableFile,
                                                                        DataRecord record)
```
    Tidies the DataRecord by performing actions based on the values of the settings map given (or getDefaultTidySettings if none is given). Each value in the record that is a string will be tidied. Keys are ignored. The record given will not be modified, but a new record with the tidied values will be returned.
    
    Parameters:
    
    scrapeableFile - The current ScrapeableFile, used for resolving relative URLs when tidying links
    
    record - The DataRecord to tidy (values in the record will be overwritten with the tidied values)
    
    Returns:
    
    A new DataRecord containing the tidied values
  - tidyDataRecord
```
@Nullable
 @RequiredVersion(value=1)
public DataRecord tidyDataRecord(ScrapeableFile scrapeableFile,
                                                                        DataRecord record,
                                                                        boolean ignoreLowerCaseKeys)
```
    Tidies the DataRecord by performing actions based on the values of the settings map given (or getDefaultTidySettings if none is given). Each value in the record that is a string will be tidied. Keys are ignored. The record given will not be modified, but a new record with the tidied values will be returned.
    
    Parameters:
    
    scrapeableFile - The current ScrapeableFile, used for resolving relative URLs when tidying links
    
    record - The DataRecord to tidy (values in the record will be overwritten with the tidied values)
    
    ignoreLowerCaseKeys - True if values with keys containing lowercase characters should be ignored
    
    Returns:
    
    A new DataRecord containing the tidied values
  - tidyDataRecord
```
@Nullable
 @RequiredVersion(value=1)
public DataRecord tidyDataRecord(ScrapeableFile scrapeableFile,
                                                                        DataRecord record,
                                                                        java.util.Map<java.lang.String,java.lang.Boolean> settings)
```
    Tidies the DataRecord by performing actions based on the values of the settings map given (or getDefaultTidySettings if none is given). Each value in the record that is a string will be tidied. Keys are ignored. The record given will not be modified, but a new record with the tidied values will be returned.
    
    Parameters:
    
    scrapeableFile - The current ScrapeableFile, used for resolving relative URLs when tidying links
    
    record - The DataRecord to tidy (values in the record will be overwritten with the tidied values)
    
    settings - The operations to perform when tidying, using a Map<String, Boolean>
    
    Returns:
    
    A new DataRecord containing the tidied values
  - tidyDataRecord
```
@Nullable
 @RequiredVersion(value=1)
 @Contract(value="_, null, _, _ -> null; _, !null, _, _ -> !null")
public DataRecord tidyDataRecord(ScrapeableFile scrapeableFile,
                                                                                                                                            @Nullable
                                                                                                                                            DataRecord record,
                                                                                                                                            java.util.Map<java.lang.String,java.lang.Boolean> settings,
                                                                                                                                            boolean ignoreLowerCaseKeys)
```
    Tidies the DataRecord by performing actions based on the values of the settings map given (or getDefaultTidySettings if none is given). Each value in the record that is a string will be tidied. Keys are not modified. The record given will not be modified, but a new record with the tidied values will be returned.
    
    Parameters:
    
    scrapeableFile - The current ScrapeableFile, used for resolving relative URLs when tidying links
    
    record - The DataRecord to tidy (values in the record will be overwritten with the tidied values)
    
    settings - The operations to perform when tidying, using a Map<String, Boolean>
    
    ignoreLowerCaseKeys - True if values with keys containing lowercase characters should be ignored
    
    Returns:
    
    A new DataRecord containing the tidied values
  - getOptionSet
```
@NotNull
 @RequiredVersion(value=1)
public DataSet getOptionSet(@Nullable
                                                                  java.lang.String options)
```
    Gets a DataSet containing each of the elements of a <select> tag. The returned DataRecords will contain a key for the text found between the tags (possibly with html tags removed), a value indicating if it was the selected option, and the value to submit for the specific option. Note that this only looks for option tags, and as such passing in text containing more than a single select tag will produce false output. Returns all options, and tidies the text between the option tags.
    
    Parameters:
    
    options - The text containing the select tag
    
    Returns:
    
    A DataSet with one record per option. Values will be stored in
    VALUE : The value the browser would submit for this option
    TEXT : The text that was between the tags
    SELECTED : A boolean that is true if this option was selected by default
  - getOptionSet
```
@NotNull
 @RequiredVersion(value=1)
public DataSet getOptionSet(@Nullable
                                                                  java.lang.String options,
                                                                  @Nullable
                                                                  java.lang.String ignoreLabel)
```
    Gets a DataSet containing each of the elements of a <select> tag. The returned DataRecords will contain a key for the text found between the tags (possibly with html tags removed), a value indicating if it was the selected option, and the value to submit for the specific option. Note that this only looks for option tags, and as such passing in text containing more than a single select tag will produce false output.
    
    Parameters:
    
    options - The text containing the select tag
    
    ignoreLabel - Text value to ignore in the output set. Usually this would be a string like "Please select a category"
    
    Returns:
    
    A DataSet with one record per option. Values will be stored in
    VALUE : The value the browser would submit for this option
    TEXT : The text that was between the tags
    SELECTED : A boolean that is true if this option was selected by default
  - getOptionSet
```
@NotNull
 @RequiredVersion(value=1)
public DataSet getOptionSet(@Nullable
                                                                  java.lang.String options,
                                                                  @Nullable
                                                                  java.lang.String ignoreLabel,
                                                                  boolean tidyRecords)
```
    Gets a DataSet containing each of the elements of a <select> tag. The returned DataRecords will contain a key for the text found between the tags (possibly with html tags removed), a value indicating if it was the selected option, and the value to submit for the specific option. Note that this only looks for option tags, and as such passing in text containing more than a single select tag will produce false output.
    
    Parameters:
    
    options - The text containing the select tag
    
    ignoreLabel - Text value to ignore in the output set. Usually this would be a string like "Please select a category"
    
    tidyRecords - Should the TEXT be tidied before being stored.
    
    Returns:
    
    A DataSet with one record per option. Values will be stored in
    VALUE : The value the browser would submit for this option
    TEXT : The text that was between the tags
    SELECTED : A boolean that is true if this option was selected by default
  - getOptionSet
```
@NotNull
 @RequiredVersion(value=1)
public DataSet getOptionSet(@Nullable
                                                                  java.lang.String options,
                                                                  @Nullable
                                                                  java.lang.String[] ignoreLabels,
                                                                  boolean tidyRecords)
```
    Gets a DataSet containing each of the elements of a <select> tag. The returned DataRecords will contain a key for the text found between the tags (possibly with html tags removed), a value indicating if it was the selected option, and the value to submit for the specific option. Note that this only looks for option tags, and as such passing in text containing more than a single select tag will produce false output.
    
    Parameters:
    
    options - The text containing the select tag
    
    ignoreLabels - Text values to ignore in the output set. Usually this would include the strings like "Please select a category"
    
    tidyRecords - Should the TEXT be tidied before being stored.
    
    Returns:
    
    A DataSet with one record per option. Values will be stored in
    VALUE : The value the browser would submit for this option
    TEXT : The text that was between the tags
    SELECTED : A boolean that is true if this option was selected by default
  - getOptionSet
```
@NotNull
 @RequiredVersion(value=1)
public DataSet getOptionSet(@Nullable
                                                                  java.lang.String options,
                                                                  @Nullable
                                                                  java.util.Collection<java.lang.String> ignoreLabels,
                                                                  boolean tidyRecords)
```
    Gets a DataSet containing each of the elements of a <select> tag. The returned DataRecords will contain a key for the text found between the tags (possibly with html tags removed), a value indicating if it was the selected option, and the value to submit for the specific option. Note that this only looks for option tags, and as such passing in text containing more than a single select tag will produce false output.
    
    Parameters:
    
    options - The text containing the select tag
    
    ignoreLabels - Text values to ignore in the output set. Usually this would include the strings like "Please select a category"
    
    tidyRecords - Should the TEXT be tidied before being stored.
    
    Returns:
    
    A DataSet with one record per option. Values will be stored in
    VALUE : The value the browser would submit for this option
    TEXT : The text that was between the tags
    SELECTED : A boolean that is true if this option was selected by default
  - getRadioButtonSet
```
@NotNull
 @RequiredVersion(value=1)
public DataSet getRadioButtonSet(java.lang.String buttons,
                                                                       java.lang.String buttonName)
```
    Gets all the options from a radio button group. The values are returned in a data record. Any labels that are to be ignored will not be included in the returned set. Not all buttons have a label, as radio buttons do not require a label, and it would be difficult to know in a regular expression exactly what to extract as the label unless there is a label tag.
    
    Parameters:
    
    buttons - The text containing the buttons
    
    buttonName - The name of the buttons that should be grabbed, as a Regex pattern
    
    Returns:
    
    DataSet containing one record for each of the extracted radio buttons. Values will be stored in
    VALUE : The value the browser would submit for this radio button
    TEXT : The text that represents this button, or null if no label could be found for it
    SELECTED : A boolean that is true if this button was selected by default
    ID : The ID of the radio button, or null if no ID was found
  - getRadioButtonSet
```
@NotNull
 @RequiredVersion(value=1)
public DataSet getRadioButtonSet(java.lang.String buttons,
                                                                       java.lang.String buttonName,
                                                                       @Nullable
                                                                       java.lang.String ignoreLabel)
```
    Gets all the options from a radio button group. The values are returned in a data record. Any labels that are to be ignored will not be included in the returned set. Not all buttons have a label, as radio buttons do not require a label, and it would be difficult to know in a regular expression exactly what to extract as the label unless there is a label tag.
    
    Parameters:
    
    buttons - The text containing the buttons
    
    buttonName - The name of the buttons that should be grabbed, as a Regex pattern
    
    ignoreLabel - A label that should be excluded from the resulting set
    
    Returns:
    
    DataSet containing one record for each of the extracted radio buttons. Values will be stored in
    VALUE : The value the browser would submit for this radio button
    TEXT : The text that represents this button, or null if no label could be found for it
    SELECTED : A boolean that is true if this button was selected by default
    ID : The ID of the radio button, or null if no ID was found
  - getRadioButtonSet
```
@NotNull
 @RequiredVersion(value=1)
public DataSet getRadioButtonSet(java.lang.String buttons,
                                                                       java.lang.String buttonName,
                                                                       @Nullable
                                                                       java.lang.String[] ignoreLabels)
```
    Gets all the options from a radio button group. The values are returned in a data record. Any labels that are to be ignored will not be included in the returned set. Not all buttons have a label, as radio buttons do not require a label, and it would be difficult to know in a regular expression exactly what to extract as the label unless there is a label tag.
    
    Parameters:
    
    buttons - The text containing the buttons
    
    buttonName - The name of the buttons that should be grabbed, as a Regex pattern
    
    ignoreLabels - Any labels that should be excluded from the resulting set
    
    Returns:
    
    DataSet containing one record for each of the extracted radio buttons. Values will be stored in
    VALUE : The value the browser would submit for this radio button
    TEXT : The text that represents this button, or null if no label could be found for it
    SELECTED : A boolean that is true if this button was selected by default
    ID : The ID of the radio button, or null if no ID was found
  - getRadioButtonSet
```
@NotNull
 @RequiredVersion(value=1)
public DataSet getRadioButtonSet(java.lang.String buttons,
                                                                       java.lang.String buttonName,
                                                                       java.util.Collection<java.lang.String> ignoreLabels)
```
    Gets all the options from a radio button group. The values are returned in a data record. Any labels that are to be ignored will not be included in the returned set. Not all buttons have a label, as radio buttons do not require a label, and it would be difficult to know in a regular expression exactly what to extract as the label unless there is a label tag.
    
    Parameters:
    
    buttons - The text containing the buttons
    
    buttonName - The name of the buttons that should be grabbed, as a Regex pattern
    
    ignoreLabels - Any labels that should be excluded from the resulting set
    
    Returns:
    
    DataSet containing one record for each of the extracted radio buttons. Values will be stored in
    VALUE : The value the browser would submit for this radio button
    TEXT : The text that represents this button, or null if no label could be found for it
    SELECTED : A boolean that is true if this button was selected by default
    ID : The ID of the radio button, or null if no ID was found
  - getRadioButtonSet
```
@NotNull
 @RequiredVersion(value=1)
public DataSet getRadioButtonSet(@Nullable
                                                                       java.lang.String buttons,
                                                                       java.lang.String buttonName,
                                                                       @Nullable
                                                                       java.util.Collection<java.lang.String> ignoreLabels,
                                                                       boolean tidyRecords)
```
    Gets all the options from a radio button group. The values are returned in a data record. Any labels that are to be ignored will not be included in the returned set. Not all buttons have a label, as radio buttons do not require a label, and it would be difficult to know in a regular expression exactly what to extract as the label unless there is a label tag.
    
    Parameters:
    
    buttons - The text containing the buttons
    
    buttonName - The name of the buttons that should be grabbed, as a Regex pattern
    
    ignoreLabels - Any labels that should be excluded from the resulting set
    
    tidyRecords - True if the text label should be tidied, false to leave it exactly as it was
    
    Returns:
    
    DataSet containing one record for each of the extracted radio buttons. Values will be stored in
    VALUE : The value the browser would submit for this radio button
    TEXT : The text that represents this button, or null if no label could be found for it
    SELECTED : A boolean that is true if this button was selected by default
    ID : The ID of the radio button, or null if no ID was found
  - mergeDataRecords
```
@NotNull
 @RequiredVersion(value=1)
public DataRecord mergeDataRecords(@NotNull
                                                                         DataRecord first,
                                                                         @NotNull
                                                                         DataRecord second,
                                                                         boolean saveNonEmptyString)
```
    Merges two data records by copying all values from the second record over values of the first record, and returning a new DataRecord with these values. Doesn't modify either original record
    
    Parameters:
    
    first - The first DataRecord, into which the values from the second record will be copied
    
    second - The second DataRecord, whose values will be copied into the first
    
    saveNonEmptyString - True if blank values should not overwrite non-blank values, whether the non-blank value is in the first or second record. If false, all values in the second record will overwrite any values in the first record.
    
    Returns:
    
    A new DataRecord with the merged values
  - linear
```
public double linear(double number,
                     double minIn,
                     double maxIn,
                     double minOut,
                     double maxOut)
```
    Performs a linear interpolation between two values and returns the result. For example, 5 input from 0 to 10 with output 0 to 100 will return 50. This doesn't ensure the output or input is within the specified range, rather it outputs a value that is the same percent between the output as the input value was between input. 5 in with range of -10 to 0 and output range -100 to 0 will produce the output 50
    
    Parameters:
    
    number - The input value
    
    minIn - The minimum value the input should be
    
    maxIn - The maximum value the input should be
    
    minOut - The minimum value the output should be
    
    maxOut - The maximum value the output should be
    
    Returns:
    
    The linearly interpolated number.
  - toCamelCase
```
@Nullable
public java.lang.String toCamelCase(@Nullable
                                               java.lang.String text)
```
    Converts an input string to have each word start with an uppercase character. Underscores count as a break between words. Hyphens do not separate words unless there is a white space character before the next character.
    
    Parameters:
    
    text - The text to convert
    
    Returns:
    
    The converted text
  - convertFromUpperCase
```
@Nullable
public java.lang.String convertFromUpperCase(@Nullable
                                                        java.lang.String text)
```
    Converts things such as FIELD_NAME to Field Name
    
    Parameters:
    
    text - The text to convert
    
    Returns:
    
    The converted text
  - randomPause
```
@RequiredVersion(value=1)
public void randomPause(long min,
                                                   long max)
```
    Pauses for a random amount of time. This is also setup to stop immediately if the stop scrape button is clicked, and to allow breakpoints to be triggered while it is pausing.
    
    Parameters:
    
    min - The minimum duration of the pause, in milliseconds
    
    max - The maximum duration of the pause, in milliseconds
  - runMultipleScrapes
```
@Deprecated
 @InternalOnly
 @RequiredVersion(value=1,
                 behavior=LOG_ERROR_AND_RETURN_NULL_NEG_ONE_OR_FALSE)
public void runMultipleScrapes(@Nullable
                                                                                                                                                               java.lang.String scrapeName,
                                                                                                                                                               @Nullable
                                                                                                                                                               java.lang.String key,
                                                                                                                                                               @Nullable
                                                                                                                                                               java.lang.String file,
                                                                                                                                                               int maxConcurrentScrapes)
                                                                                                                                                        throws java.lang.Exception
```
    Deprecated.
    
    Runs multiple scrapes concurrently.
    
    Parameters:
    
    scrapeName - The scrape to run
    
    key - The session variable to set values for
    
    file - The file containing the values, 1 per line, with ### indicating a comment. Values will be trimmed
    
    maxConcurrentScrapes - The maximum number of scrapes that can be run at once
    
    Throws:
    
    java.lang.Exception - If no valid scraping session can be found with the given name
  - runMultipleScrapes
```
@Deprecated
 @InternalOnly
public void runMultipleScrapes(java.lang.String scrapeName,
                                                            java.lang.String key,
                                                            @NotNull
                                                            java.lang.Object[] values,
                                                            int maxConcurrentScrapes)
                                                     throws java.lang.Exception
```
    Deprecated.
    
    Runs multiple scrapes concurrently.
    
    Parameters:
    
    scrapeName - The scrape to run
    
    key - The session variable to set values for
    
    values - The values to set under the given key for each scrape
    
    maxConcurrentScrapes - The maximum number of scrapes that can be run at once
    
    Throws:
    
    java.lang.Exception - If no valid scraping session could be found with the given name
  - runMultipleScrapes
```
@Deprecated
 @InternalOnly
public void runMultipleScrapes(java.lang.String scrapeName,
                                                            java.lang.String key,
                                                            java.util.List<?> values,
                                                            int maxConcurrentScrapes)
                                                     throws java.lang.Exception
```
    Deprecated.
    
    Runs multiple scrapes concurrently.
    
    Parameters:
    
    scrapeName - The scrape to run
    
    key - The session variable to set values for
    
    values - The values to set under the given key for each scrape
    
    maxConcurrentScrapes - The maximum number of scrapes that can be run at once
    
    Throws:
    
    java.lang.Exception - If no valid scraping session could be found with the given name
  - runMultipleScrapes
```
@Deprecated
 @InternalOnly
 @RequiredVersion(value=1,
                 behavior=LOG_ERROR_AND_RETURN_NULL_NEG_ONE_OR_FALSE)
public void runMultipleScrapes(@Nullable
                                                                                                                                                               java.lang.String scrapeName,
                                                                                                                                                               @Nullable
                                                                                                                                                               java.lang.String key,
                                                                                                                                                               @Nullable
                                                                                                                                                               java.util.List<?> values,
                                                                                                                                                               int maxConcurrentScrapes,
                                                                                                                                                               boolean shouldInheritFromParent)
                                                                                                                                                        throws java.lang.Exception
```
    Deprecated.
    
    Runs multiple scrapes concurrently.
    
    Parameters:
    
    scrapeName - The scrape to run
    
    key - The session variable to set values for
    
    values - The values to set under the given key for each scrape
    
    maxConcurrentScrapes - The maximum number of scrapes that can be run at once
    
    shouldInheritFromParent - True is run scrapes should inherit everything from the parent scrape, false to have each start without inheriting
    
    Throws:
    
    java.lang.Exception - If no valid scraping session could be found with the given name
  - dateIsWithinDays
```
@RequiredVersion(value=1)
public boolean dateIsWithinDays(@NotNull
                                                           java.util.Date date1,
                                                           @NotNull
                                                           java.util.Date date2,
                                                           int days)
```
    Checks to see if one date is within a certain number of days of another
    
    Parameters:
    
    date1 - The first date
    
    date2 - The second date
    
    days - The maximum number of days that can be between the two dates
    
    Returns:
    
    True if the dates are close than or the number of days apart, false otherwise
  - applyXPathExpression
```
@Nullable
 @RequiredVersion(value=1)
public XmlNode applyXPathExpression(@NotNull
                                                                           java.lang.String content,
                                                                           java.lang.String expression)
```
    Applies an XPath expression to a string.
    
    Parameters:
    
    expression - An XPath expression.
    
    content - The string the expression should be applied to.
    
    Returns:
    
    An XmlNode.
  - extractXmlBlock
```
@Nullable
 @RequiredVersion(value=1)
public java.lang.String extractXmlBlock(@NotNull
                                                                               java.lang.String content,
                                                                               int startingDepth)
```
    Extracts a section of xml (roughly) by counting start and end tags. It doesn't look for well-formatted values, simply counts of opening and closing tags. This method is a convenience method and should only be used when XPath and Regex don't work, as it isn't optimized. Perhaps a future version of it will be optimized more.
    
    Parameters:
    
    content - The content to extract from (the start of the string should already be at the desired location to start extraction). If the startingDepth is 0, the content should start with an opening tag
    
    startingDepth - The current depth of open elements, in general this will be 0
    
    Returns:
    
    The matched string
  - parseName
```
@Nullable
 @RequiredVersion(value=1)
public Name parseName(@NotNull
                                                             java.lang.String name)
```
    Attempts to parse a string to a name. The parser is not perfect and works best on english formatted names (for example, "John Smith Jr." or "Guerrero, Antonio K". This uses standard settings for the parser. To get more control over how the name is parsed, use the EnglishNameParser class
    
    Parameters:
    
    name - The name to be parsed
    
    Returns:
    
    The parsed name, as a Name object
    
    Throws:
    
    java.lang.IllegalArgumentException - If the parse fails. The cause of the exception will contain additional information about the parse error
  - parseUSAddress
```
@Nullable
 @RequiredVersion(value=1)
public Address parseUSAddress(@NotNull
                                                                     java.lang.String address)
```
    Attempts to parse a string to an address. The parser is not perfect and works best on US addresses. Most likely other address formats can be parsed with the USAddressParser class by providing different constraints in the builder. This method is here for convenience in working with US addresses
    
    Parameters:
    
    address - The address to be parsed
    
    Returns:
    
    The parsed address, as an Address object
    
    Throws:
    
    java.lang.IllegalArgumentException - If the parse fails. The cause of the exception will contain additional information about the parse error
  - reorderUSAddressBestGuessFromLines
```
@NotNull
 @RequiredVersion(value=1)
public Address reorderUSAddressBestGuessFromLines(@Nullable
                                                                                        java.lang.String addressee,
                                                                                        @Nullable
                                                                                        java.lang.String address,
                                                                                        @Nullable
                                                                                        java.lang.String address2)
```
    Attempts to reorder the addressee, address, and address2 lines using the address parser to attempt to validate each part. This method WILL NOT BE completely accurate. It is a best effort method but will fail sometimes (especially if the input values are not normal-looking address parts). This will return an address with the given data filled in. If the parsing strategy fails to determine a different order than what is given (or if the determined order matches that which was given) then an address object will be returned with the same order for values as was specified in the parameters, hence the parameter order matters.
    
    Parameters:
    
    addressee - The assumed addressee line
    
    address - The assumed address (street) line
    
    address2 - The assumed secondary address (suite or apartment) line
    
    Returns:
    
    An Address object with the addressee, address, and suiteOrApartment values filled in based on the given input here
  - getRandomUserAgent
```
@NotNull
public java.lang.String getRandomUserAgent()
```
    Returns a random User Agent. The list isn't closely monitored, so it may not include newer user agents, and may include extremely old ones as well. This will returns a desktop user agent
    
    Returns:
    
    A random user agent.
  - getRandomUserAgent
```
@Nullable
public java.lang.String getRandomUserAgent(@Nullable
                                                      BrowserType browserType,
                                                      @Nullable
                                                      java.time.LocalDate newerThan,
                                                      @Nullable
                                                      java.time.LocalDate olderThan)
```
    Returns a random User Agent. The list isn't closely monitored, so it may not include newer user agents, and may include extremely old ones as well.
    
    Parameters:
    
    browserType - The type of browser for which to get a user agent
    
    newerThan - User agents that are newer than (or the same as) this date. Null means no lower release date bound
    
    olderThan - User agents that are older than (or the same as) this date. Null means no upper release date bound
    
    Returns:
    
    A random user agent.
    
    Since:
    
    7.0.1a
  - getTimestampObject
```
@NotNull
public java.lang.Object getTimestampObject()
```
    Returns an object whose toString method always resolves to the current timestamp. That way, in a scrapeable file parameter you can simply do: ~#TIMESTAMP#~ to get the current timestamp. Just set the session variable first, like session.setVariable("TIMESTAMP", sutil.getTimestampObject());
    
    Returns:
    
    An object whose toString method always returns the current timestamp (as a long value)
    
    Since:
    
    7.0.4a
  - getRandomReferrer
```
public java.lang.String getRandomReferrer()
```
    Gets a random referrer page from a list of many different search engine web sites and a few other pages
    
    Returns:
    
    A random referrer URL
  - getJSONObject
```
@Nullable
public org.json.JSONObject getJSONObject(@Nullable
                                                    org.json.JSONObject json,
                                                    @NotNull
                                                    java.lang.String key)
                                             throws org.json.JSONException
```
    Gets a json object from a json object with a given key, or returns null if there is no such value, the input is null, or the value is null or not a json object
    
    Parameters:
    
    json - The input object
    
    key - The key. The key can be a path (such as foo.bar[0].name)
    
    Returns:
    
    The json object for that key, or null if there wasn't a value
    
    Throws:
    
    org.json.JSONException - If there is a json error (shouldn't happen unless a key is invalid)
    
    Since:
    
    7.0.11a
  - getJSONArray
```
@Nullable
public org.json.JSONArray getJSONArray(@Nullable
                                                  org.json.JSONObject json,
                                                  @NotNull
                                                  java.lang.String key)
                                           throws org.json.JSONException
```
    Gets a json array from a json object with a given key, or returns null if there is no such value, the input is null. If there is a non-null value that isn't an array, this returns a new array containing that single element
    
    Parameters:
    
    json - The input object
    
    key - The key. The key can be a path (such as foo.bar[0].name)
    
    Returns:
    
    The json array for that key, a new array containing the value if the value wasn't an array and wasn't null, or null if there wasn't a value or the value was null
    
    Throws:
    
    org.json.JSONException - If there is a json error (shouldn't happen unless a key is invalid)
    
    Since:
    
    7.0.11a
  - getJSONString
```
@Nullable
public java.lang.String getJSONString(@Nullable
                                                 org.json.JSONObject json,
                                                 @NotNull
                                                 java.lang.String key)
                                          throws org.json.JSONException
```
    Gets a string from a json object. If the value is null, the key doesn't exist, or the input json is null then this returns null. Otherwise this returns a string representation of the value
    
    Parameters:
    
    json - The input object
    
    key - The key. The key can be a path (such as foo.bar[0].name)
    
    Returns:
    
    If the value is null, the key doesn't exist, or the input json is null then this returns null. Otherwise this returns a string representation of the value
    
    Throws:
    
    org.json.JSONException - If there is a json error (shouldn't happen unless a key is invalid)
    
    Since:
    
    7.0.11a
  - getJSONString
```
@Contract(value="_, _, null -> _; _, _, !null -> !null")
 @Nullable
public java.lang.String getJSONString(@Nullable
                                                                                                            org.json.JSONObject json,
                                                                                                            @NotNull
                                                                                                            java.lang.String key,
                                                                                                            @Nullable
                                                                                                            java.lang.String defaultValue)
                                                                                                     throws org.json.JSONException
```
    Gets a string from a json object. If the value is null, the key doesn't exist, or the input json is null then this returns null. Otherwise this returns a string representation of the value
    
    Parameters:
    
    json - The input object
    
    key - The key. The key can be a path (such as foo.bar[0].name)
    
    defaultValue - The value to return if the resulting value is null (either a non-present key or a null value for the key)
    
    Returns:
    
    If the value is null, the key doesn't exist, or the input json is null then this returns the defaultValue. Otherwise this returns a string representation of the value
    
    Throws:
    
    org.json.JSONException - If there is a json error (shouldn't happen unless a key is invalid)
    
    Since:
    
    7.0.11a
  - alphabetLoopWithSubdivision
```
@RequiredVersion(value=1)
public boolean alphabetLoopWithSubdivision(@Nullable
                                                                      java.lang.String letters,
                                                                      @NotNull
                                                                      java.util.function.Function<java.lang.String,java.lang.Boolean> function)
```
    Runs a subdividing search by alphabet letters, a-z (lowercase). This will start by calling the function with the empty string if the input letters are null, otherwise starts with the input letters. If the function returns true then it will add a subdivision loop by adding a new letter. For example, if the input letter is 'a' and the function returns true, then this will call the function again with 'aa' then 'ab' through 'az' and then fallback to the previous level of subdivision and search with 'b'. Note if the scraping session is stopped this will stop subdividing and return
    
    Parameters:
    
    letters - The starting letters for subdivision, or null to start with the empty string. Note passing in 'c' doesn't mean it will start at 'c' and then search 'd'. It will simply search 'c' and if false is returned by the function it will end, or if true is returned it'll then search 'ca', 'cb', etc...
    
    function - The function to run for the actual search. The input is the string for the current alphabet search (never null but can be empty) and the output is a boolean for whether subdivision is required or not
    
    Returns:
    
    Returns true (here for internal use)
    
    Since:
    
    7.0.11a
  - alphabetLoopWithSubdivision
```
@RequiredVersion(value=1)
public boolean alphabetLoopWithSubdivision(@Nullable
                                                                      java.lang.String letters,
                                                                      @NotNull
                                                                      java.util.function.Function<java.lang.String,java.lang.Boolean> function,
                                                                      boolean includeAlpha,
                                                                      boolean includeDigits,
                                                                      @Nullable
                                                                      java.lang.String skipToLetters)
```
    Runs a subdividing search by alphabet letters, a-z (lowercase). This will start by calling the function with the empty string if the input letters are null, otherwise starts with the input letters. If the function returns true then it will add a subdivision loop by adding a new letter. For example, if the input letter is 'a' and the function returns true, then this will call the function again with 'aa' then 'ab' through 'az' and then fallback to the previous level of subdivision and search with 'b'. Note if the scraping session is stopped this will stop subdividing and return
    
    Parameters:
    
    letters - The starting letters for subdivision, or null to start with the empty string. Note passing in 'c' doesn't mean it will start at 'c' and then search 'd'. It will simply search 'c' and if false is returned by the function it will end, or if true is returned it'll then search 'ca', 'cb', etc...
    
    function - The function to run for the actual search. The input is the string for the current alphabet search (never null but can be empty) and the output is a boolean for whether subdivision is required or not
    
    includeAlpha - True if it should include letters in the subdivision
    
    includeDigits - True if it should include digits in the subdivision
    
    skipToLetters - The letters to skip to before calling the passed in function. This will run regular subdivision until it reaches these letters and then call the function (null/empty means not to skip any)
    
    Returns:
    
    Returns true if it found the skip to letters. This is used internally for recursion and tracking the skip letters
    
    Since:
    
    7.0.11a
  - alphabetLoopWithSubdivision
```
@RequiredVersion(value=1)
public boolean alphabetLoopWithSubdivision(@Nullable
                                                                      java.lang.String firstLetters,
                                                                      @Nullable
                                                                      java.lang.String lastLetters,
                                                                      @NotNull
                                                                      java.util.function.BiFunction<java.lang.String,java.lang.String,java.lang.Boolean> function)
```
    Runs a subdividing search by alphabet letters, a-z (lowercase). This will start by calling the function with the empty string for both values. If the function returns true then it will add a subdivision loop by adding a new letter (balancing the two inputs to keep a similar length). For example, if the input letters are 'a,a' and the function returns true, then this will call the function again with 'a,aa' then 'a,ab' through 'a,az' and then fallback to the previous level of subdivision and search with 'a,b'. When both letters are the same length it will add the subdivision to the second letter set, otherwise it will add it to the first letter set. Note if the scraping session is stopped this will stop subdividing and return. Generally this would be useful to search by first/last name
    
    Parameters:
    
    firstLetters - The starting letters for subdivision on the first letter set, or null to start with the empty string. Note passing in 'c' doesn't mean it will start at 'c' and then search 'd'. It will simply search 'c' and if false is returned by the function it will end, or if true is returned it'll then search 'ca', 'cb', etc...
    
    lastLetters - The starting letters for subdivision on the second letter set, or null to start with the empty string. Note passing in 'c' doesn't mean it will start at 'c' and then search 'd'. It will simply search 'c' and if false is returned by the function it will end, or if true is returned it'll then search 'ca', 'cb', etc...
    
    function - The function to run for the actual search. The input is the string for the current alphabet search (never null but can be empty) and the output is a boolean for whether subdivision is required or not
    
    Returns:
    
    Returns true (here for internal use)
    
    Since:
    
    7.0.11a
  - alphabetLoopWithSubdivision
```
@RequiredVersion(value=1)
public boolean alphabetLoopWithSubdivision(@Nullable
                                                                      java.lang.String firstLetters,
                                                                      @Nullable
                                                                      java.lang.String lastLetters,
                                                                      @NotNull
                                                                      java.util.function.BiFunction<java.lang.String,java.lang.String,java.lang.Boolean> function,
                                                                      boolean includeAlpha,
                                                                      boolean includeDigits,
                                                                      @Nullable
                                                                      java.lang.String skipToFirstLetters,
                                                                      @Nullable
                                                                      java.lang.String skipToLastLetters)
```
    Runs a subdividing search by alphabet letters, a-z (lowercase). This will start by calling the function with the empty string for both values. If the function returns true then it will add a subdivision loop by adding a new letter (balancing the two inputs to keep a similar length). For example, if the input letters are 'a,a' and the function returns true, then this will call the function again with 'a,aa' then 'a,ab' through 'a,az' and then fallback to the previous level of subdivision and search with 'a,b'. When both letters are the same length it will add the subdivision to the second letter set, otherwise it will add it to the first letter set. Note if the scraping session is stopped this will stop subdividing and return. Generally this would be useful to search by first/last name
    
    Parameters:
    
    firstLetters - The starting letters for subdivision on the first letter set, or null to start with the empty string. Note passing in 'c' doesn't mean it will start at 'c' and then search 'd'. It will simply search 'c' and if false is returned by the function it will end, or if true is returned it'll then search 'ca', 'cb', etc...
    
    lastLetters - The starting letters for subdivision on the second letter set, or null to start with the empty string. Note passing in 'c' doesn't mean it will start at 'c' and then search 'd'. It will simply search 'c' and if false is returned by the function it will end, or if true is returned it'll then search 'ca', 'cb', etc...
    
    function - The function to run for the actual search. The input is the string for the current alphabet search (never null but can be empty) and the output is a boolean for whether subdivision is required or not
    
    includeAlpha - True if it should include letters in the subdivision
    
    includeDigits - True if it should include digits in the subdivision
    
    skipToFirstLetters - The letters to skip to for first letters before calling the passed in function. This will run regular subdivision until it reaches these letters and then call the function once both skip to letter values are reached (null/empty means not to skip any)
    
    skipToLastLetters - The letters to skip to for last letters before calling the passed in function. This will run regular subdivision until it reaches these letters and then call the function once both skip to letter values are reached (null/empty means not to skip any)
    
    Returns:
    
    Returns true if it found the skip to letters. This is used internally for recursion and tracking the skip letters
    
    Since:
    
    7.0.11a
  - runMultipleSessionsConcurrentlyForInput
```
@RequiredVersion(value=2)
public <T> void runMultipleSessionsConcurrentlyForInput(int numToRun,
                                                                                   @NotNull
                                                                                   java.util.function.Function<T,java.util.List<T>> individualRunFunction,
                                                                                   @NotNull
                                                                                   java.util.concurrent.Callable<java.util.function.Supplier<T>> searchInputSupplierInitializer)
```
    Runs multiple scraping sessions in parallel over the same input data. This allows for easily multithreading a scrape. This method should be called from each launched scrape. In other words, setup the scrape so it always calls this method regardless of the starting state. This method will also pass all current session variables (at the time the method is initially called) to the children it launches. Child scrapes will be setup to pass record count increments to the parent scrape as well so the main controlling scrape will have accurate record saved counts
    
    Type Parameters:
    
    T - The type of object returned by the supplier and passed to the run function.
    
    Parameters:
    
    numToRun - How many scrapes to run in parallel. Must be at least 1 which means the current scrape. Note that when running in the workbench this only uses a single thread (the current one) even if a larger number is passed in
    
    individualRunFunction - A function that is called for each input from the supplier. It returns a List of that type (or null) of things to add to the search input, such as any subdivisions to search. Empty or null indicates nothing new needs to be added to the search inputs
    
    searchInputSupplierInitializer - A callable to return a supplier that returns the search elements. This supplier should return null to indicate no more search inputs are found. A callable is used so the initialization only has to be performed on the controlling thread. Note the supplier returned will be called as space is available (elements are needed) to avoid excessive memory use. Also, if the scrape is stopped the get method will no longer be called, so any resource cleanup should be done with callbacks on scrape end to avoid leaking open resources
    
    Since:
    
    7.0.12a
  - runMultipleSessionsConcurrentlyForInput
```
@RequiredVersion(value=2)
public <T> void runMultipleSessionsConcurrentlyForInput(int numToRun,
                                                                                   int queueSize,
                                                                                   @NotNull
                                                                                   java.util.function.Function<T,java.util.List<T>> processInputFunction,
                                                                                   @NotNull
                                                                                   java.util.concurrent.Callable<java.util.function.Supplier<T>> supplierInitializer)
```
    Runs multiple scraping sessions in parallel over the same input data. This allows for easily multithreading a scrape. This method should be called from each launched scrape. In other words, setup the scrape so it always calls this method regardless of the starting state. This method will also pass all current session variables (at the time the method is initially called) to the children it launches. Child scrapes will be setup to pass record count increments to the parent scrape as well so the main controlling scrape will have accurate record saved counts
    
    Type Parameters:
    
    T - The type of object returned by the supplier and passed to the run function.
    
    Parameters:
    
    numToRun - How many scrapes to run in parallel. Must be at least 1 which means the current scrape. Note that when running in the workbench this only uses a single thread (the current one) even if a larger number is passed in
    
    queueSize - How large of a queue to have from the input supplier for the workers. The value must be at least 1 (if it is less than one it will default to 1).
    
    processInputFunction - A function that is called for each input from the supplier. It returns a List of that type (or null) of things to add to the search input, such as any subdivisions to search. Empty or null indicates nothing new needs to be added to the search inputs
    
    supplierInitializer - A callable to return a supplier that returns the search elements. This supplier should return null to indicate no more search inputs are found. A callable is used so the initialization only has to be performed on the controlling thread. Note the supplier returned will be called as space is available (elements are needed) to avoid excessive memory use. Also, if the scrape is stopped the get method will no longer be called, so any resource cleanup should be done with callbacks on scrape end to avoid leaking open resources
    
    Since:
    
    7.0.12a
  - setupAutoIncrementParentScrapeRecordsFromChild
```
public void setupAutoIncrementParentScrapeRecordsFromChild(@NotNull
                                                           ScrapingSession parent)
```
    Sets up events to automatically increment the records for a parent scrape when the current scrape changes records count. This method should only be called when the current scraping session for the scraping util is not null
    
    Parameters:
    
    parent - The parent scrape for which record counts should be adjusted when this scrape updates record counts

Class ScrapingUtil

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Constructor Detail

ScrapingUtil

Method Detail

reformatDate

reformatDate

getImageHeight

getImageWidth

getMaximumMemoryAllocation

getMemoryUsage

getCpuUsage

pause

resizeImageFixWidth

resizeImageFixHeight

resizeImageFixWidthAndHeight

sendMail

sendMail

sendMail

isNullOrEmptyString

equalsIgnoreCase

startsWithUpperCase

isInt

nullToEmptyString

writeValueToFile

convertHTMLEntities

unzipFile

makeGETRequestNoSessionProxy

makeGETRequestUseSessionProxy

makeGETRequest

makeHEADRequest

makeDELETERequest

makeDELETERequest

makePUTRequest

makePUTRequest

getScreenScraperVersion

getScreenScraperEdition

isPlatformLinux

isPlatformMac

isPlatformWindows

getMimeType

getInstallDir

stripHTML

getNumRunningScrapingSessions

getNumRunningScrapingSessions

getNumRunnableScrapingSessions

stringToFloat

setLogInternalMessages

getLogInternalMessages

formatUSPhoneNumber

formatNumber

formatNumber

formatNumber

formatNumber

formatUSZipCode

convertStringToDate

convertUTFWhitespace

convertDateToString

convertDateToString

getCurrentDate

sortSet

sortSet

sortSet

getDefaultTidySettings

tidyString

tidyString

tidyString

tidyString

tidyDataRecord

tidyDataRecord

tidyDataRecord

tidyDataRecord

tidyDataRecord

tidyDataRecord

tidyDataRecord

tidyDataRecord

getOptionSet

getOptionSet