public class General
extends java.lang.Object
Modifier and Type | Class and Description |
---|---|
(package private) static class |
General.CursorChanger |
static class |
General.StringBrowserDisplayer |
Modifier and Type | Field and Description |
---|---|
static long |
BYTES_IN_A_MEGABYTE
The number of bytes in a megabyte.
|
static java.lang.String |
CRLF |
static byte[] |
EMPTY_BYTE_ARRAY |
static java.lang.String[] |
EMPTY_STRING_ARRAY |
static java.lang.String[][] |
EMPTY_STRING_ARRAY_2D |
static java.lang.String |
EOF |
static long |
IL |
static java.lang.Class[] |
parameters
Holds classes for the classpath.
|
static boolean |
RUN_SHUTDOWN_HOOK
Indicates whether or not the shutdown hook should be run on exit.
|
static java.lang.String |
SECOND_INSTANCE_ERROR_MESSAGE
The message to be given when a second instance of screen-scraper attempts to start up and we are going to exit.
|
static java.lang.String |
SECOND_INSTANCE_ERROR_MESSAGE_CONTINUE_ANYWAY
The message to be given when a second instance of screen-scraper attempts to start up and we are going to allow the user to proceed anyway.
|
static java.lang.String |
VARIABLE_BEGIN_DELIMITER |
static java.lang.String |
VARIABLE_END_DELIMITER |
static long |
VL |
static java.lang.String |
WORKING_DIR
The current working directory.
|
Constructor and Description |
---|
General() |
Modifier and Type | Method and Description |
---|---|
static void |
addShutdownHook()
Ensures that things get cleaned up, even if the app is interrupted.
|
static java.lang.String |
adjustNewlines(java.lang.String input)
Normalizes newline symbols in a string to a canonical form.
|
static boolean |
applicationIsOpen(java.lang.String fileLockName)
Indicates whether or not an instance of the server or workbench are open.
|
static boolean |
areMultipleApplicationsOpen()
Indicates whether or not multiple applications are open (e.g., the workbench and the server).
|
static int |
booleanToInt(boolean value)
Turns a boolean value into an int, for storage in the database.
|
static boolean |
characterReaderContainsString(java.io.Reader reader,
java.lang.String val)
Checks the given reader of characters for a specific string.
|
static void |
cleanUpCommandLineLockFiles()
Deletes any lock files that have been queued for deletion.
|
static boolean |
commandLineIsOpen(boolean considerMultiple)
Checks to see if an instance of the command line is already open.
|
static int |
compareToUpperCase(java.lang.String string1,
java.lang.String string2)
Compare the uppercase compareTo values of two strings Note that this is to be used with strings pulled from JTextArea widgets, which don't properly take
into account hard returns when giving selection locations.
|
static boolean |
contains(java.lang.String searchString,
java.lang.String regularExpression)
Indicates whether or not the given
String contains the given regular expression. |
static java.lang.String |
convertHTMLEntities(java.lang.String val)
Replaces HTML entities in a string with their non-entity equivalents.
|
static int |
countMatchesSubString(java.lang.String haystack,
java.lang.String needle,
int limit)
Counts the number of sub-strings within a string up to a certain point in the original string.
|
static java.lang.String |
decode(java.lang.String url)
This is just a wrapper to URLDecoder.decode() that catches any errors while encoding
|
static java.lang.String |
decode(java.lang.String url,
java.lang.String characterSet)
This is just a wrapper to URLDecoder.decode() that catches any errors while encoding
|
static byte[] |
decompressGzipStream(java.io.InputStream inputStream,
int maxKBytes)
Decompresses a GZIP stream, returning the resulting bytes.
|
static byte[] |
decompressZlibContent(byte[] content)
Decompresses an array of bytes in zlib format.
|
static void |
deleteCacheDatabase()
Removes the cache database from the file system.
|
static void |
deleteDeleteMeFile()
Deletes the "delete_me.htm" file.
|
static java.util.HashMap |
deriveDaysHoursMinutesSecondsFromMilliseconds(long millis)
Derives the number of days, hours, minutes, and seconds from a long representing a number of milliseconds.
|
static void |
displayStringInLocalWebBrowser(java.lang.String html,
java.lang.String url)
Displays the HTML in the string in a local web browser.
|
static void |
displayValueInTextArea(java.lang.String value)
Displays a value in a text area.
|
static void |
doFullHotSwapFromDatabase()
Does a hot swap from the database.
|
static void |
doFullHotSwapFromFiles()
Does a hot swap for both scripts and scraping sessions.
|
static void |
dumpFileContentsToLog(java.io.File file)
Dumps the contents of a file to the log.
|
static void |
dumpMemoryUsage()
Writes out to the log information relating to the applications current use of memory.
|
static java.lang.String |
encode(java.lang.String str,
java.lang.String characterSet)
This is just a wrapper to URLEncoder.encode() that catches any errors while encoding
|
static void |
ensureLocksFolderExists()
Ensures that the locks folder exists.
|
static boolean |
equalsUpperCase(java.lang.String string1,
java.lang.String string2)
Compare the uppercase values of two strings
|
static java.lang.String |
escapeSpaces(java.lang.String string)
Replaces space characters with %20.
|
static java.lang.String |
expressionize(java.lang.String contents,
ExtractorPattern extractorPattern)
Converts an extractor pattern so it is ready to run as a regular expression
|
static java.lang.String |
generateDebuggingInformation()
Generates an output string containing data about the threads in the JVM.
|
static int |
getAvailablePort()
Returns a port that is currently open.
|
static int |
getAvailablePort(java.lang.String startPort)
Gets a port available on the system.
|
static java.awt.Point |
getCenteredPoint(java.awt.Component component)
This method returns the point a component, like a dialog, should be placed on the screen based on the components length and width so that it is centerd.
|
static long |
getCheckSum(java.io.File file)
Gets the checksum for a file.
|
static java.lang.String |
getCurrentlySelectedHTML(javax.swing.JEditorPane editorPane,
java.lang.String originalText,
int selectionThreshold)
Gets the currently selected HTML.
|
static java.lang.String |
getCurrentYear()
Gets the current year, as a string.
|
static java.lang.String[] |
getDefaultCharacterSets()
Gets an array containing default character sets.
|
static java.util.List<java.lang.String> |
getDefaultScriptImports()
Returns the list of imports that are used for every script (Java language).
|
static java.lang.String |
getDefaultScriptImportStatement()
Returns the import statement that will be used on all scripts.
|
static java.awt.Frame |
getFrame(java.awt.Component component)
Given a
Component , this method will ascend through the component hierarchy until it hits a Frame . |
static java.util.Map<java.lang.String,java.lang.Object> |
getGlobalInstanceVariables()
Returns a
ConcurrentHashMap of instance-wide variables that can be used in all scrapes. |
static java.lang.String |
getGUID()
Gets the GUID for this installation.
|
static ScrapingHttpClient |
getHttpClient()
Gets a basic HttpClient.
|
static org.apache.http.conn.routing.HttpRoute |
getHttpRouteFromGeneralSettings(org.apache.http.HttpHost target,
org.apache.http.HttpRequest request,
org.apache.http.protocol.HttpContext context,
ScrapingSession scrapingSession)
Sets proxy server settings based on the local proxy configuration.
|
static java.lang.String |
getHumanReadableRepresentationOfMilliseconds(long time)
Gets a human-readable representation of an amount of time in milliseconds.
|
static java.util.Set<java.io.File> |
getImportableFiles(java.lang.Class<?> type)
Looks for importable files in the "import" directory.
|
static int |
getIntForObject(java.lang.Object value)
Returns an int for a
Object . |
static java.io.File |
getLastSelectedDirectory()
Gets the last directory selected from a
JFileChooser . |
static java.lang.String |
getMimeType(java.io.File file)
Gets the MIME-type for a file.
|
static java.lang.String |
getMimeTypeThrowable(java.io.File file)
Gets the MIME-type for a file throwing an exception if a problem occurs.
|
static int |
getNumMatches(java.lang.String stringToSearch,
java.lang.String stringToFind,
boolean isCaseSensitive)
Determines the number of times one string is found in another, allowing for case sensitivity.
|
static int |
getNumStringsWithinString(java.lang.String needle,
java.lang.String haystack)
Gets the number of strings within a string as matched by a regular expression.
|
static java.lang.String |
getPathForURL(java.net.URL url)
Gets the path portion of a URL.
|
static int |
getPortFromSetting(java.lang.String port)
Gets a port based on the setting provided.
|
static java.lang.String |
getQueryString(java.net.URL url)
Gets the query string portion of a URL.
|
static java.lang.String |
getUUID()
Gets a UUID.
|
static java.util.HashSet<java.lang.String> |
getVariableNames(java.lang.String value)
Extracts variable names from a
String . |
static boolean |
hasWriteAccess()
Indicates whether or not we have write access to certain critical files.
|
static boolean |
hasWriteAccess(java.io.File file)
Checks for write access to a specific file.
|
static boolean |
httpResponseIsInGzipFormat(org.apache.http.HttpResponse httpResponse)
Indicates whether or not the method's response is in GZIP format.
|
static boolean |
installIncludesJVM()
Indicates whether or not the installation includes a JVM.
|
static boolean |
intToBoolean(int value)
Turns an int value into a boolean, for retrieval from database.
|
static boolean |
ipAddressAllowedToConnect(java.lang.String ipAddress)
Determines whether or not the given IP address should be allowed to connect.
|
static boolean |
isAnyApplicationOpen()
Indicates if any application is currently open.
|
static boolean |
isInt(java.lang.String string)
Indicates whether or not the given
String can be parsed as an int. |
static boolean |
isNullOrEmptyString(java.lang.Object object)
Indicates whether or not an object is either null or an empty string.
|
static boolean |
isPlatformAIX()
Indicates whether or not we're running on AIX.
|
static boolean |
isPlatformLinux()
Indicates whether or not we're running on linux.
|
static boolean |
isPlatformMac()
Indicates whether or not we're running on a Mac.
|
static boolean |
isPlatformWindows()
Indicates whether or not we're running on Windows.
|
static boolean |
isRunningInWorkbench()
Returns whether or not we are currently running in the workbench.
|
static int |
lessThanZeroToZero(int num)
Converts a number less than 0 to 0.
|
static long |
lessThanZeroToZero(long num)
Converts a number less than 0 to 0.
|
static void |
lockCommandLine()
Obtains a lock for this particular command line instance.
|
static void |
lockFile(java.lang.String fileLockName)
Locks the file with the given name, if possible.
|
static void |
lockServer()
Obtains a lock on the server.
|
static void |
lockWorkbench()
Obtains a lock on the workbench.
|
static void |
logSystemProperties()
Outputs all system properties to the log.
|
static void |
main(java.lang.String[] args)
Just a little convenience method used to get the checksum for a file.
|
static java.lang.String |
makeDELETERequest(java.lang.String urlString,
java.util.Map<java.lang.String,java.lang.String> headers,
ScrapingSession session)
Makes a DELETE request.
|
static java.lang.String |
makeGETRequest(java.lang.String urlString)
Makes a GET request and returns the result as a string.
|
static java.lang.String |
makeGETRequest(java.lang.String urlString,
int attempts)
Makes a GET request and returns the result as a string.
|
static java.lang.String |
makeGETRequest(java.lang.String urlString,
ScrapingSession session)
Makes a GET request and returns the result as a string.
|
static java.lang.String[][] |
makeHEADRequest(java.lang.String urlString,
ScrapingSession session)
Makes a HEAD request and returns the headers in a two-deminsional array.
|
static java.lang.String |
makePUTRequest(java.lang.String urlString,
java.util.Map<java.lang.String,java.lang.String> headers,
ScrapingSession session)
Makes a PUT request.
|
static boolean |
methodResponseIsInGzipFormat(org.apache.commons.httpclient.HttpMethodBase method)
Indicates whether or not the method's response is in GZIP format.
|
static void |
notifyOfCorruptDatabase()
Notifies the user that the database is corrupted.
|
static void |
notifyOfWriteFileError(java.lang.String files)
Notifies the user that they don't have write access to needed files.
|
static void |
notifyUserOfComplexHTML(javax.swing.JComponent component)
If, when trying to get the currently selected HTML, we're unable to for some reason this method will notify the user.
|
static java.lang.String |
nullToEmptyString(java.lang.Object value)
Turns a potentially null value into an empty string.
|
static boolean |
objectNameIsValid(java.lang.String name)
Validates that the given
String represents a valid name for a ScrapingSession . |
static void |
openURLInNativeBrowser(java.lang.String url)
Displays a URL in the local native web browser.
|
static void |
ph()
Phones home.
|
static java.lang.String |
prepareStringForDatabase(java.lang.String value)
If the given string is null this method will simply return null back; otherwise, it will return the string properly escaped and surrounded by
single-quotes to prepare it to be inserted into the database.
|
static void |
queueCleanUpCommandLineLockFiles()
Cleans up any old command line lock files.
|
static byte[] |
readContentFromHttpMethodBase(org.apache.commons.httpclient.HttpMethodBase method,
int maxKBytes)
Reads in content from an
HttpMethodBase and returns a byte array containing the content. |
static byte[] |
readContentFromHttpResponse(org.apache.http.HttpResponse httpResponse,
int maxKBytes,
boolean decompressContent)
Reads in content from an
HttpResponse and returns a byte array containing the content. |
static byte[] |
readStreamIntoByteArray(java.io.InputStream stream)
Reads a stream into a byte array.
|
static java.lang.String |
replace(java.lang.String regularExpression,
java.lang.String substitution,
java.lang.String searchedString)
Finds and replaces a string using a regular expression.
|
static java.lang.String |
replaceInvalidCharactersInURL(java.lang.String url)
HttpClient doesn't like pipes in URL's, but it seems to work to replace the pipes with forward slashes, so this method handles that.
|
static void |
replaceStringWithinFile(java.lang.String find,
java.lang.String replace,
java.io.File inFile)
Replaces all instances of one string for another within a file.
|
static boolean |
serverIsOpen()
Checks to see if an instance of the server is already open.
|
static void |
setFrameLocation(java.awt.Component frame,
int x,
int y)
Sets the location of the frame to the given location, ensuring it is at least partway visible on a screen
|
static void |
setFrameLocation(java.awt.Component frame,
java.awt.Point location)
Sets the location of the frame to the given location, ensuring it is at least partway visible on a screen
|
static void |
setLastSelectedDirectory(java.io.File lastSelectedDirectory)
Sets the last directory selected from a
JFileChooser . |
static void |
setMacSystemProperties()
Sets Mac-specific system properties.
|
static void |
setProxySettingsOnHttpClient(org.apache.commons.httpclient.HttpClient client,
org.apache.commons.httpclient.HostConfiguration hostConfiguration)
Sets proxy settings on the client using the default proxy settings.
|
static void |
setRequiredSystemProperties()
Sets system properties needed globally.
|
static void |
setupMozilla()
Sets up the Mozilla stuff.
|
static void |
showBusyCursor(java.awt.Component component)
Changes the cursor to a busy cursor.
|
static void |
showCursor(java.awt.Component component,
int cursor)
Changes the cursor to the given cursor.
|
static void |
showDefaultCursor(java.awt.Component component)
Changes the cursor to the default cursor.
|
static boolean |
startsWithUpperCase(java.lang.String string1,
java.lang.String string2)
Compare the uppercase startsWith values of two strings
|
static java.lang.String |
stripHTML(java.lang.String content)
Strips HTML from a string, replacing some tags with corresonding text-only equivalents.
|
static java.lang.String |
stripHTMLComments(java.lang.String html)
Strips HTML comments from a string.
|
static java.lang.String |
stripUnnecessaryWhiteSpace(java.lang.String input)
Strips off all \t, \r, and \n characters..
|
static java.lang.String |
swapExtractorPatternTokensForEmbeddedSessionVariables(java.lang.String value)
Swaps any extractor pattern tokens (e.g., ~@FOO@~) for embedded session variables (e.g., ~#FOO#~).
|
static void |
tele()
Phone home in a separate thread.
|
static boolean |
testPorts()
Ensures that all needed ports are available, and notifies the user, if not.
|
static java.lang.String |
tidyHTML(java.lang.String html,
java.lang.String characterSet,
int tidyHTMLType)
Tidies up the HTML string.
|
static java.lang.String |
trimIfNotNull(java.lang.String str)
Trims a string if it's not null; returns null, otherwise.
|
static boolean |
valEA(java.lang.String ea)
Validates that an email address is registered.
|
static void |
valEANoGUI()
Validates a license without using anything graphical.
|
static long |
validate()
Used to validate whether or not the license is valid.
|
static void |
waitForApplicationToClose()
In the case of two applications closing at about the same time this method causes one application to wait until the other finishes first.
|
static boolean |
workbenchIsOpen()
Checks to see if an instance of the workbench is already open.
|
static void |
writeDataToFile(java.lang.String data,
java.io.File file)
Write the
String out to a file. |
static void |
writeValueToFile(java.lang.Object value,
java.lang.String file,
java.lang.String charSet)
Writes a value to a file using the given character set.
|
public static final byte[] EMPTY_BYTE_ARRAY
public static final java.lang.String[] EMPTY_STRING_ARRAY
public static final java.lang.String[][] EMPTY_STRING_ARRAY_2D
public static final java.lang.String VARIABLE_BEGIN_DELIMITER
public static final java.lang.String VARIABLE_END_DELIMITER
public static final java.lang.String CRLF
public static final java.lang.String EOF
@Nullable public static java.lang.String WORKING_DIR
@NotNull public static java.lang.String SECOND_INSTANCE_ERROR_MESSAGE
@NotNull public static java.lang.String SECOND_INSTANCE_ERROR_MESSAGE_CONTINUE_ANYWAY
public static final long VL
public static final long IL
public static final java.lang.Class[] parameters
public static final long BYTES_IN_A_MEGABYTE
public static boolean RUN_SHUTDOWN_HOOK
@NotNull public static java.util.Map<java.lang.String,java.lang.Object> getGlobalInstanceVariables()
ConcurrentHashMap
of instance-wide variables that can be used in all scrapes. Note that this map can cause memory leaks (eg a scrape
adds to it but doesn't clean up what it adds, causing the instance to hold onto that memory until the next restart). Be sure that you clean up anything
you add to this map unless you need it to still be there later.@Nullable public static java.lang.String prepareStringForDatabase(@Nullable java.lang.String value)
value
- The String
to be prepared.public static java.lang.String nullToEmptyString(@Nullable java.lang.Object value)
value
- A String
that may be null.public static int booleanToInt(boolean value)
value
- The boolean value.public static boolean intToBoolean(int value)
value
- The int value.@Nullable public static java.lang.String trimIfNotNull(@Nullable java.lang.String str)
str
- The string to trim.public static boolean isNullOrEmptyString(@Nullable java.lang.Object object)
object
- The Object
to test.public static int getIntForObject(java.lang.Object value)
Object
. Will return -1 if the object can't be parsed.value
- The value to parse.@Nullable public static java.lang.String replace(@Nullable java.lang.String regularExpression, @Nullable java.lang.String substitution, @Nullable java.lang.String searchedString)
regularExpression
- The regular expression to search for.substitution
- The text to substitute in.searchedString
- The string to search.public static boolean contains(@NotNull java.lang.String searchString, java.lang.String regularExpression)
String
contains the given regular expression.searchString
- A String
to search.regularExpression
- A String
representing a regular expression.public static java.awt.Frame getFrame(java.awt.Component component)
Component
, this method will ascend through the component hierarchy until it hits a Frame
. This method is used so that
various
cursors can be displayed and so that dialog boxes can be made modal.component
- A Component
.Frame
.public static void showBusyCursor(java.awt.Component component)
component
- Some Component
within the hierarchy.public static void showDefaultCursor(java.awt.Component component)
component
- Some Component
within the hierarchy.@NotNull public static java.util.List<java.lang.String> getDefaultScriptImports()
@NotNull public static java.lang.String getDefaultScriptImportStatement()
public static void setFrameLocation(@NotNull java.awt.Component frame, @NotNull java.awt.Point location)
frame
- The framelocation
- The new location desiredpublic static void setFrameLocation(@NotNull java.awt.Component frame, int x, int y)
frame
- The framex
- The new x coordinate desiredy
- The new y coordinate desiredpublic static void showCursor(java.awt.Component component, int cursor)
component
- Some Component
within the hierarchy.cursor
- The cursor position@Nullable @Contract(value="null, !null -> null; !null, !null -> !null") public static java.lang.String expressionize(@Nullable java.lang.String contents, @NotNull ExtractorPattern extractorPattern)
contents
- The contents of the expression, including the token namesextractorPattern
- The extractor pattern@Nullable @Contract(value="null -> null; !null -> !null") public static java.lang.String stripUnnecessaryWhiteSpace(@Nullable java.lang.String input)
input
- String
to strip white space from.String
with all unnecessary white space stripped.public static java.lang.String adjustNewlines(@NotNull java.lang.String input)
input
- String
to normalize newlines forString
representing the input string with all newline symbols replaced by the desired newline string.@Nullable @Contract(value="null -> null; !null -> !null") public static java.lang.String stripHTMLComments(@Nullable java.lang.String html)
html
- A string containing HTML.@Nullable @Contract(value="null -> null; !null -> !null") public static java.lang.String stripHTML(@Nullable java.lang.String content)
content
- The content to be stripped.@NotNull public static java.util.HashSet<java.lang.String> getVariableNames(@Nullable java.lang.String value)
String
.value
- The string the variable names are to be extracted from.HashSet
containing the variable names.public static java.awt.Point getCenteredPoint(@NotNull java.awt.Component component)
component
- Component to find the centerd location for.@Nullable public static java.lang.String getQueryString(@Nullable java.net.URL url)
url
- The URL that may contain a query string.public static java.lang.String getPathForURL(@Nullable java.net.URL url)
url
- The URL.@Nullable public static java.lang.String escapeSpaces(java.lang.String string)
string
- The string to fix.public static boolean equalsUpperCase(@Nullable java.lang.String string1, @Nullable java.lang.String string2)
string1
- The string1 to compare.string2
- The string2 to compare.public static boolean startsWithUpperCase(@Nullable java.lang.String string1, @Nullable java.lang.String string2)
string1
- The string1 to compare.string2
- The string2 to compare.public static int compareToUpperCase(@Nullable java.lang.String string1, @Nullable java.lang.String string2)
string1
- The string1 to compare.string2
- The string2 to compare.public static int countMatchesSubString(@Nullable java.lang.String haystack, @Nullable java.lang.String needle, int limit)
haystack
- The string to searchneedle
- The string to findlimit
- the maximum number of times to find the string@Nullable public static java.lang.String getCurrentlySelectedHTML(@Nullable javax.swing.JEditorPane editorPane, @Nullable java.lang.String originalText, int selectionThreshold)
editorPane
- The JEditorPane
from which the selected HTML should be drawn.originalText
- The original text being selected from. This is needed in the event that more than one match is found.selectionThreshold
- Determines the buffer on either side of a selected block of text that will comprise an extractor pattern, when requested.public static void notifyUserOfComplexHTML(javax.swing.JComponent component)
component
- The JComponent
the dialog box needs to be modal to.public static boolean isRunningInWorkbench()
@Nullable public static java.lang.String tidyHTML(@Nullable java.lang.String html, @Nullable java.lang.String characterSet, int tidyHTMLType)
html
- The string of HTML to tidy.characterSet
- The character set to use when tidying.tidyHTMLType
- The type of tidying to be done, as defined by the "TIDY_HTML" constants found in the Settings class.public static boolean isAnyApplicationOpen()
public static boolean areMultipleApplicationsOpen()
public static boolean workbenchIsOpen()
public static void lockWorkbench()
public static boolean serverIsOpen()
public static void lockServer()
public static boolean commandLineIsOpen(boolean considerMultiple)
considerMultiple
- A boolean indicating whether or not we care whether two or more are open, or just one.public static void queueCleanUpCommandLineLockFiles()
public static void cleanUpCommandLineLockFiles()
public static void lockCommandLine()
public static void ensureLocksFolderExists()
public static boolean applicationIsOpen(java.lang.String fileLockName)
fileLockName
- The name of the file lock to check for.public static void waitForApplicationToClose()
public static void lockFile(java.lang.String fileLockName)
fileLockName
- The name of the file lock to check for.public static void deleteDeleteMeFile()
public static void logSystemProperties()
public static void dumpFileContentsToLog(@NotNull java.io.File file)
file
- The file to read.public static long validate()
public static void valEANoGUI()
public static boolean valEA(java.lang.String ea)
ea
- The email address.public static boolean ipAddressAllowedToConnect(@NotNull java.lang.String ipAddress)
ipAddress
- The IP address.public static boolean isInt(@NotNull java.lang.String string)
String
can be parsed as an int.string
- The String
to check.public static void dumpMemoryUsage()
public static boolean installIncludesJVM()
public static long getCheckSum(@NotNull java.io.File file)
file
- The file.@Nullable public static java.io.File getLastSelectedDirectory()
JFileChooser
.File
.public static void setLastSelectedDirectory(java.io.File lastSelectedDirectory)
JFileChooser
.lastSelectedDirectory
- A File
.public static void writeDataToFile(@NotNull java.lang.String data, @NotNull java.io.File file)
String
out to a file.data
- The String
to write.file
- The File
to write to.public static boolean objectNameIsValid(@NotNull java.lang.String name)
String
represents a valid name for a ScrapingSession
.name
- The proposed name.@NotNull public static java.util.Set<java.io.File> getImportableFiles(java.lang.Class<?> type)
type
- The class of objects desired (either ScrapingSession
or Script
.HashSet
containing File
objects representing either ScrapingSession
objects or Script
objects. Note that an empty HashSet
will be returned if no importable objects are found.public static void doFullHotSwapFromFiles()
public static void doFullHotSwapFromDatabase()
public static void setupMozilla()
public static void addShutdownHook()
public static boolean httpResponseIsInGzipFormat(@NotNull org.apache.http.HttpResponse httpResponse)
httpResponse
- The HTTP response to examine.public static byte[] readContentFromHttpResponse(@NotNull org.apache.http.HttpResponse httpResponse, int maxKBytes, boolean decompressContent) throws java.io.IOException
HttpResponse
and returns a byte array containing the content.httpResponse
- The HttpResponse
.maxKBytes
- The maximum number of kilobytes to read.decompressContent
- Decompress the content, if necessary.java.io.IOException
- On error reading the responsepublic static boolean methodResponseIsInGzipFormat(@NotNull org.apache.commons.httpclient.HttpMethodBase method)
method
- The method to examine.public static byte[] readContentFromHttpMethodBase(@NotNull org.apache.commons.httpclient.HttpMethodBase method, int maxKBytes) throws java.io.IOException
HttpMethodBase
and returns a byte array containing the content.method
- The HttpMethodBase
.maxKBytes
- The maximum number of kilobytes to read.java.io.IOException
- On error reading datapublic static byte[] readStreamIntoByteArray(@NotNull java.io.InputStream stream)
stream
- An InputStream
.public static void deleteCacheDatabase()
public static void writeValueToFile(@NotNull java.lang.Object value, @NotNull java.lang.String file, @Nullable java.lang.String charSet) throws java.lang.Exception
value
- The string to be written.file
- The name of the file.charSet
- The character set to use.java.lang.Exception
- On error writing datapublic static void displayValueInTextArea(java.lang.String value)
value
- The bytes to display.public static org.apache.http.conn.routing.HttpRoute getHttpRouteFromGeneralSettings(@NotNull org.apache.http.HttpHost target, org.apache.http.HttpRequest request, @NotNull org.apache.http.protocol.HttpContext context, @Nullable ScrapingSession scrapingSession)
target
- The host targetrequest
- The request issuedcontext
- The context of the requestscrapingSession
- The scraping session for the requestpublic static void setProxySettingsOnHttpClient(@NotNull org.apache.commons.httpclient.HttpClient client, @NotNull org.apache.commons.httpclient.HostConfiguration hostConfiguration)
client
- The client to set the settings on.hostConfiguration
- The host configuration to set the settings on.public static boolean testPorts()
public static int getAvailablePort(@Nullable java.lang.String startPort)
startPort
- Starts at this port, and increments till it finds an available one.public static int getPortFromSetting(java.lang.String port)
port
- The port, which likely comes from the settings file. If the value for this parameter is "dynamic" a port will be found for it.@Nullable @Contract(value="null -> null; !null -> !null") public static java.lang.String replaceInvalidCharactersInURL(@Nullable java.lang.String url)
url
- The URL that may contain pipes.public static void tele()
public static void ph()
public static java.lang.String getGUID()
@NotNull public static java.lang.String getUUID()
@Nullable public static java.lang.String getCurrentYear()
@NotNull public static java.util.HashMap deriveDaysHoursMinutesSecondsFromMilliseconds(long millis)
millis
- The number of millisecondsHashMap
containing the four "REPEAT_" params, identified by the constants in WebConstants.public static void openURLInNativeBrowser(@NotNull java.lang.String url)
url
- The URL to display, as a String
.public static void displayStringInLocalWebBrowser(@Nullable java.lang.String html, java.lang.String url)
html
- The HTML to be displayed.url
- The original URL pointing to the page.@Nullable public static java.lang.String convertHTMLEntities(@Nullable java.lang.String val)
val
- The string containing the entities.public static boolean characterReaderContainsString(@NotNull java.io.Reader reader, @NotNull java.lang.String val)
reader
- A Reader
used to reader the characters.val
- The string to check for.public static boolean isPlatformAIX()
public static boolean isPlatformLinux()
public static boolean isPlatformMac()
public static boolean isPlatformWindows()
public static void setRequiredSystemProperties()
public static void setMacSystemProperties()
public static boolean hasWriteAccess() throws java.lang.RuntimeException
java.lang.RuntimeException
- An exception whose message contains a comma-delimited list of files the current user doesn't have write access to.public static boolean hasWriteAccess(@NotNull java.io.File file)
file
- The File
to check.public static void notifyOfWriteFileError(java.lang.String files)
files
- The list of files to which they don't have write access.public static void notifyOfCorruptDatabase()
@NotNull public static java.lang.String[][] makeHEADRequest(@Nullable java.lang.String urlString, @Nullable ScrapingSession session) throws java.lang.Exception
urlString
- The URL to request, as a string.session
- An optional session object to use for its HttpClient object.java.lang.Exception
- If anything naughty happens.public static java.lang.String makeDELETERequest(@Nullable java.lang.String urlString, @Nullable java.util.Map<java.lang.String,java.lang.String> headers, @Nullable ScrapingSession session) throws java.lang.Exception
urlString
- The URL to request, as a string.session
- An optional session object to use for its HttpClient object.headers
- An optional Map
of request headers.java.lang.Exception
- If anything naughty happens.public static java.lang.String makePUTRequest(@Nullable java.lang.String urlString, @Nullable java.util.Map<java.lang.String,java.lang.String> headers, @Nullable ScrapingSession session) throws java.lang.Exception
urlString
- The URL to request, as a string.session
- An optional session object to use for its HttpClient object.headers
- An optional Map
of request headers.java.lang.Exception
- If anything naughty happens.@NotNull public static ScrapingHttpClient getHttpClient()
CloseableHttpClient
.public static java.lang.String makeGETRequest(java.lang.String urlString) throws java.lang.Exception
urlString
- The URL to request, as a string. HttpClient object.java.lang.Exception
- If anything naughty happens.public static java.lang.String makeGETRequest(@NotNull java.lang.String urlString, int attempts) throws java.io.IOException
urlString
- The URL to request, as a string. HttpClient object.attempts
- The maximum number of times to attempt requesting the URLjava.io.IOException
- If anything naughty happens.public static java.lang.String makeGETRequest(@Nullable java.lang.String urlString, @Nullable ScrapingSession session) throws java.lang.Exception
urlString
- The URL to request, as a string.session
- An optional session object to use for its HttpClient object.java.lang.Exception
- If anything naughty happens.@Nullable public static java.lang.String encode(@Nullable java.lang.String str, @Nullable java.lang.String characterSet)
str
- A string to URL-encodecharacterSet
- The character set to use when encoding.public static java.lang.String decode(@NotNull java.lang.String url)
url
- a String the url to decodepublic static java.lang.String decode(@NotNull java.lang.String url, @Nullable java.lang.String characterSet)
url
- a String the url to decodecharacterSet
- The character set to use when encoding.public static java.lang.String getMimeTypeThrowable(java.io.File file) throws java.lang.Exception
file
- The File
.java.lang.Exception
- On error finding a MIME Typepublic static java.lang.String getMimeType(java.io.File file)
file
- The File
.@NotNull public static byte[] decompressZlibContent(@NotNull byte[] content)
content
- An array of bytes in zlib format.@NotNull public static byte[] decompressGzipStream(java.io.InputStream inputStream, int maxKBytes) throws java.lang.Exception
inputStream
- The InputStream
to decompress.maxKBytes
- The maximum number of kilobytes to read.java.lang.Exception
- On error decompressing the streampublic static int lessThanZeroToZero(int num)
num
- An int.public static long lessThanZeroToZero(long num)
num
- A long.public static java.lang.String[] getDefaultCharacterSets()
public static java.lang.String getHumanReadableRepresentationOfMilliseconds(long time)
time
- The amount of time in milliseconds.@Nullable public static java.lang.String swapExtractorPatternTokensForEmbeddedSessionVariables(@Nullable java.lang.String value)
value
- The valuepublic static void main(java.lang.String[] args)
args
- The command line argumentspublic static int getAvailablePort()
public static void replaceStringWithinFile(@NotNull java.lang.String find, @NotNull java.lang.String replace, @NotNull java.io.File inFile)
find
- The string to find. Can be a regex.replace
- The string to replace the found string with.inFile
- The file to replace within.public static int getNumStringsWithinString(@NotNull java.lang.String needle, @NotNull java.lang.String haystack)
needle
- A regular expression.haystack
- The string to search.public static int getNumMatches(@Nullable java.lang.String stringToSearch, @Nullable java.lang.String stringToFind, boolean isCaseSensitive)
stringToSearch
- The string we're looking in.stringToFind
- The string we're looking for.isCaseSensitive
- Whether or not the search should be case sensitive.public static java.lang.String generateDebuggingInformation()