ProxyServerPool

java.lang.Object
- com.screenscraper.util.ProxyServerPool

public class ProxyServerPool
extends java.lang.Object

Represents a pool of proxies to be used by a scrape

Nested Class Summary

Nested Classes
Modifier and Type Class and Description

(package private) class ProxyServerPool.ProxyServerValidator
Used to validate a ProxyServer in an asynchronous fashion.

Nested Classes
Modifier and Type	Class and Description
`(package private) class`	`ProxyServerPool.ProxyServerValidator` Used to validate a `ProxyServer` in an asynchronous fashion.

Field Summary

Fields
Modifier and Type	Field and Description
`(package private) static java.util.concurrent.atomic.AtomicBoolean`	`HAS_LOADED_AUTOMATIC_ANONYMIZATION_PROXIES`
`static org.apache.log4j.Logger`	`log` Used for logging.

Constructor Summary

Constructors
Constructor and Description

ProxyServerPool()
Generates a new pool.

Constructors
Constructor and Description
`ProxyServerPool()` Generates a new pool.

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`void`	`addProxyServer(ProxyServer proxyServer)` Adds a proxy to the pool.
`void`	`currentProxyServerIsBad()` Indicates that the current proxy server is bad.
`void`	`filter()` Causes the current pool proxy server to filter its list of proxy servers.
`void`	`filter(int connectionTimeout)` Causes the proxy server to filter it's current pool.
`void`	`filter(int connectionTimeout, boolean validateHost)` Causes the proxy server to filter it's current pool.
`boolean`	`getAutomaticProxyCycling()` Returns the current state of automatic proxy cycling.
`int`	`getConnectionTimeout()` Gets the connection timeout to use when validating proxies.
`ProxyServer`	`getCurrentProxyServer()` Gets the current proxy.
`ProxyServer`	`getNextProxyServer()` Gets the next proxy to be cycled through.
`int`	`getNumProxiesToValidateConcurrently()` Gets the number of proxies we can test concurrently.
`int`	`getNumProxyServers()` Gets the number of proxy servers.
`java.io.File`	`getPoolFile()` Gets the file used to populate the pool.
`int`	`getRepopulateThreshold()` Gets the threshold that should be used when determining whether or not to repopulate.
`ScrapingSessionNotifiable`	`getScrapingSessionNotifiable()` Gets the `ScrapingSessionNotifiable` to be used.
`void`	`outputProxyServersToLog()` Outputs the current list of proxies to the log.
`void`	`populateFromFile(java.io.File file)` Gets and loads up the proxy servers, if necessary.
`void`	`populateFromFile(java.lang.String filePath)` Gets and loads up the proxy servers, if necessary.
`boolean`	`populateFromServer()` Populates the pool from the remote server.
`void`	`prepareAnonymousProxies(int numRequiredProxies, java.util.List<ProxyServer> proxyServerList)` Populates the proxy list from the server and if needed spawns more proxy servers.
`void`	`removeAllProxyServers()` Removes all proxy servers from the pool.
`void`	`repopulateIfThresholdMet()` Repopulates the pool if the repopulate threshold has been met
`void`	`setAutomaticProxyCycling(boolean shouldCycle)` Enables or disables automatic proxy cycling.
`void`	`setConnectionTimeout(int connectionTimeout)` Sets the connection timeout to use when validating proxies.
`void`	`setNumProxiesToValidateConcurrently(int numProxiesToValidateConcurrently)` Sets the number of proxies we can test concurrently.
`void`	`setNumRequestsPerProxy(int numRequestsPerProxy)` Sets the number of subsequent requests that should be made for each proxy server.
`void`	`setPoolFile(java.io.File poolFile)` Sets the file used to populate the pool.
`void`	`setPoolFile(java.lang.String filePath)` Sets the path to the file used to populate the pool.
`void`	`setProxyServers(java.util.Collection<ProxyServer> proxies)` Sets the proxies as a `Collection`.
`void`	`setRepopulateThreshold(int repopulateThreshold)` Sets the threshold that should be used when determining whether or not to repopulate.
`void`	`setScrapingSessionNotifiable(ScrapingSessionNotifiable scrapingSessionNotifiable)` Sets the `ScrapingSessionNotifiable` to be used.
`java.lang.String`	`toString()`
`void`	`waitTillCurrentProxyServerIsAlive()` Keeps trying till a proxy server is available.
`void`	`writeProxyPoolToFile(java.lang.String filePath)` Writes the current proxy list to a file.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

- Field Detail
  - log
```
public static org.apache.log4j.Logger log
```
    Used for logging.
  - HAS_LOADED_AUTOMATIC_ANONYMIZATION_PROXIES
```
static final java.util.concurrent.atomic.AtomicBoolean HAS_LOADED_AUTOMATIC_ANONYMIZATION_PROXIES
```
- Constructor Detail
  - ProxyServerPool
```
public ProxyServerPool()
```
    Generates a new pool. Uses proxies.txt.
- Method Detail
  - prepareAnonymousProxies
```
@InternalOnly
public void prepareAnonymousProxies(int numRequiredProxies,
                                                   @NotNull
                                                   java.util.List<ProxyServer> proxyServerList)
```
    Populates the proxy list from the server and if needed spawns more proxy servers.
    
    Parameters:
    
    numRequiredProxies - the number of proxy servers needed
    
    proxyServerList - The central List<ProxyServer> that holds all of the automatic proxy servers.
    
    Throws:
    
    AnonymousProxyException - On error preparing the proxies
  - currentProxyServerIsBad
```
public void currentProxyServerIsBad()
```
    Indicates that the current proxy server is bad.
  - repopulateIfThresholdMet
```
@InternalOnly
public void repopulateIfThresholdMet()
```
    Repopulates the pool if the repopulate threshold has been met
  - removeAllProxyServers
```
public void removeAllProxyServers()
```
    Removes all proxy servers from the pool.
  - getCurrentProxyServer
```
@Nullable
 @InternalOnly
public ProxyServer getCurrentProxyServer()
```
    Gets the current proxy.
    
    Returns:
    
    The ProxyServer.
  - setAutomaticProxyCycling
```
public void setAutomaticProxyCycling(boolean shouldCycle)
```
    Enables or disables automatic proxy cycling. When this is set to false (default is true) the current proxy that was automatically selected from the pool will be used each time the next proxy is requested. When set to true, each call to the getNextProxy method will cycle as normal between all available proxies.
    
    Parameters:
    
    shouldCycle - True if proxies should be allowed to rotate automatically during the scrape.
  - getAutomaticProxyCycling
```
public boolean getAutomaticProxyCycling()
```
    Returns the current state of automatic proxy cycling.
    
    Returns:
    
    True if proxies will automatically cycle throughout the scrape
  - getNextProxyServer
```
@Nullable
 @InternalOnly
public ProxyServer getNextProxyServer()
```
    Gets the next proxy to be cycled through.
    
    Returns:
    
    A ProxyServer.
  - waitTillCurrentProxyServerIsAlive
```
@InternalOnly
public void waitTillCurrentProxyServerIsAlive()
```
    Keeps trying till a proxy server is available.
  - populateFromServer
```
@InternalOnly
public boolean populateFromServer()
```
    Populates the pool from the remote server.
    
    Returns:
    
    Returns true if a proxy was still being spawned, false otherwise
  - populateFromFile
```
public void populateFromFile(@NotNull
                             java.io.File file)
```
    Gets and loads up the proxy servers, if necessary.
    
    Parameters:
    
    file - The File that should be used in populating the list.
  - getPoolFile
```
@Nullable
 @InternalOnly
public java.io.File getPoolFile()
```
    Gets the file used to populate the pool.
    
    Returns:
    
    A File.
  - setPoolFile
```
@InternalOnly
public void setPoolFile(java.io.File poolFile)
```
    Sets the file used to populate the pool.
    
    Parameters:
    
    poolFile - A File.
  - setPoolFile
```
@InternalOnly
public void setPoolFile(@NotNull
                                       java.lang.String filePath)
```
    Sets the path to the file used to populate the pool.
    
    Parameters:
    
    filePath - A String representing the path.
  - populateFromFile
```
public void populateFromFile(@NotNull
                             java.lang.String filePath)
```
    Gets and loads up the proxy servers, if necessary.
    
    Parameters:
    
    filePath - The path to the file that should be used in populating the list.
  - setProxyServers
```
@InternalOnly
public void setProxyServers(@NotNull
                                           java.util.Collection<ProxyServer> proxies)
```
    Sets the proxies as a Collection.
    
    Parameters:
    
    proxies - A Collection containing ProxyServer objects.
  - addProxyServer
```
@InternalOnly
public void addProxyServer(@Nullable
                                          ProxyServer proxyServer)
```
    Adds a proxy to the pool.
    
    Parameters:
    
    proxyServer - The ProxyServer to add.
  - writeProxyPoolToFile
```
public void writeProxyPoolToFile(@NotNull
                                 java.lang.String filePath)
```
    Writes the current proxy list to a file.
    
    Parameters:
    
    filePath - The path to the file that should be used.
  - outputProxyServersToLog
```
public void outputProxyServersToLog()
```
    Outputs the current list of proxies to the log.
  - getNumProxyServers
```
public int getNumProxyServers()
```
    Gets the number of proxy servers.
    
    Returns:
    
    The number.
  - setNumRequestsPerProxy
```
public void setNumRequestsPerProxy(int numRequestsPerProxy)
```
    Sets the number of subsequent requests that should be made for each proxy server.
    
    Parameters:
    
    numRequestsPerProxy - The number of requests.
  - getScrapingSessionNotifiable
```
@Nullable
 @InternalOnly
public ScrapingSessionNotifiable getScrapingSessionNotifiable()
```
    Gets the ScrapingSessionNotifiable to be used.
    
    Returns:
    
    A ScrapingSessionNotifiable.
  - setScrapingSessionNotifiable
```
@InternalOnly
public void setScrapingSessionNotifiable(@Nullable
                                                        ScrapingSessionNotifiable scrapingSessionNotifiable)
```
    Sets the ScrapingSessionNotifiable to be used.
    
    Parameters:
    
    scrapingSessionNotifiable - A ScrapingSessionNotifiable.
  - getConnectionTimeout
```
public int getConnectionTimeout()
```
    Gets the connection timeout to use when validating proxies.
    
    Returns:
    
    The timeout.
  - setConnectionTimeout
```
public void setConnectionTimeout(int connectionTimeout)
```
    Sets the connection timeout to use when validating proxies.
    
    Parameters:
    
    connectionTimeout - The timeout.
  - getNumProxiesToValidateConcurrently
```
public int getNumProxiesToValidateConcurrently()
```
    Gets the number of proxies we can test concurrently.
    
    Returns:
    
    The number of proxies.
  - setNumProxiesToValidateConcurrently
```
public void setNumProxiesToValidateConcurrently(int numProxiesToValidateConcurrently)
```
    Sets the number of proxies we can test concurrently.
    
    Parameters:
    
    numProxiesToValidateConcurrently - The number of proxies.
  - getRepopulateThreshold
```
public int getRepopulateThreshold()
```
    Gets the threshold that should be used when determining whether or not to repopulate.
    
    Returns:
    
    The threshold.
  - setRepopulateThreshold
```
public void setRepopulateThreshold(int repopulateThreshold)
```
    Sets the threshold that should be used when determining whether or not to repopulate.
    
    Parameters:
    
    repopulateThreshold - The threshold.
  - filter
```
public void filter(int connectionTimeout)
```
    Causes the proxy server to filter it's current pool.
    
    Parameters:
    
    connectionTimeout - The maximum number of seconds to wait when testing each proxy.
  - filter
```
public void filter(int connectionTimeout,
                   boolean validateHost)
```
    Causes the proxy server to filter it's current pool.
    
    Parameters:
    
    connectionTimeout - The maximum number of seconds to wait when testing each proxy.
    
    validateHost - Indicates whether or not the host should be validated when filtering proxies.
  - filter
```
public void filter()
```
    Causes the current pool proxy server to filter its list of proxy servers. The current connection timeout will be used in filtering.
  - toString
```
@NotNull
public java.lang.String toString()
```
    Overrides:
    
    toString in class java.lang.Object

Class ProxyServerPool

Nested Class Summary

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

log

HAS_LOADED_AUTOMATIC_ANONYMIZATION_PROXIES

Constructor Detail

ProxyServerPool

Method Detail

prepareAnonymousProxies

currentProxyServerIsBad

repopulateIfThresholdMet

removeAllProxyServers

getCurrentProxyServer

setAutomaticProxyCycling

getAutomaticProxyCycling

getNextProxyServer

waitTillCurrentProxyServerIsAlive

populateFromServer

populateFromFile

getPoolFile

setPoolFile

setPoolFile

populateFromFile

setProxyServers

addProxyServer

writeProxyPoolToFile

outputProxyServersToLog

getNumProxyServers

setNumRequestsPerProxy

getScrapingSessionNotifiable

setScrapingSessionNotifiable

getConnectionTimeout

setConnectionTimeout

getNumProxiesToValidateConcurrently

setNumProxiesToValidateConcurrently

getRepopulateThreshold

setRepopulateThreshold

filter

filter

filter

toString