Release notes for screen-scraper

View Alpha Release Change Log

Public Release 6.0 (03.30.12)

Now allowing circular redirects.
Fixed a bug where the log was being centered incorrectly after a search.
Added "Accept all cookies" cookie policy.
Added preliminary support for using Rackspace as an alternative to Amazon in anonymization service.
Added support for specifying character set in the XmlWriter.
Fixed a backward compatibility issue with extractor pattern tokens.
Fixed an issue that affects editing tokens on some older Macs.
Fixed a bug related to line wrapping in large responses.
Fixed an issue that disallowed creating new scrapeable files.
Now allowing underscores in a URL.
Now allowing back-slashes in a URL.
Fixed an uncommon case of thread deadlock.
Fixed a rare issue with file uploads.
Fixed an issue with extractor patterns getting interrupted.
Added session.setUseServerCharacterSet( boolean useServerCharacterSet ) method.
Fixed a race condition where a scraping session could potentially get started by two different threads.
DataManager: a few logging changes
DataManager: a modification of the order of database writes when foreign keys are manually set
DataManager: transactional support for rolling back writes
DataManager: a framework for making data assertions
Fixed an issue exporting large scripts that call session.executeScript.
Added session.scrapeString(String scrapeableFileName, String content)
Fixed an issue with ss running on 64-bit Windows.
Fixed an issue where a blank file HTTP parameter was being sent incorrectly.
Fixed an issue with the REST interface where the wrong scrapeable_session_id was being returned.
Fixed an issue where the HTTP connection manager was getting closed prematurely.
Fixes a bug introduced in the previous alpha related to closing the HTTP connection manager.
The < > symbols are now being handled properly in URL's.
The host used for screen-scraper's database can now be set via the DatabaseHost property in the screen-scraper.properties file.
Fixed a bug where a null parameter was causing rendering problems.
Added ability to turn on and off automatic proxy cycling via setAutomaticProxyCycling.
Auto-saving can now be enabled by adding an AutoSaveTime=[Time in seconds] in the screen-scraper.properties file.
Filtered data sets now show up as filtered when using the "Test Pattern" button.
Added SetCharacterSet to .NET driver.
Fixed a bug related to terminating anonymous proxies via the "Settings" dialog box.
Fixed an issue related to long key names in parameters.
Fixed an issue related to designating the character set when exporting scripts.
Fixed an issue related to populating a proxy pool from a file.
Fixed an issue related to Rackspace anonymization.
Now outputting message as a warning when extractor pattern times out.
Script pane no longer scrolls to the top when finding text fails.
The last error message will now always be retained in the Web UI.
Now notifying the user if a scrapeable file is generated from an HTTP transaction that contains a multi-part request, but no file parameters.
Changed icon to something friendlier on database backup pop-up.
Added session.setUserAgent.
Fixed an issue related to resolving relative URL's from extracted data.
Fixed an issue related to reordering columns in the workbench.
Fixed an issue related to truncated server responses.
Fixed the PHP driver to allow carriage returns and line feeds to be passed in the setVariable method.
Now initializing the last response view to the top of the page.
Now displaying recently accessed scripts first in the script instances drop-down list.
Enlarged the scraping session notes field a bit.
Added back and forward buttons to the workbench.
Multiple objects can now be selected in the main tree, allowing them to be deleted, exported, and moved.
Deprecated caching and filtering data sets (can be re-enabled with EnableCachingAndFilteringDataSets property).
Now automatically swapping extractor pattern tokens for embedded variables in certain fields in the workbench (e.g., in the URL field ~@FOO@~ is changed to ~#FOO#~).
Added a "Find" button to the "Last Request" tab.
Fixed a bug that was causing the user-agent header to be duplicated.
Fixed a bug where a deleted recent script still shows in the script drop-down list.
Fixed a bug related to multi-exports.
Fixed a bug related to exporting single objects.
Several DataManager fixes.
Fixed a bug related to multi-selection in the tree.
Enabled gzip encoding in the proxy server.
Added saveStateToString and loadStateFromString methods.
Added new methods to script auto-complete feature.
Added executeScriptWithContext method.
Made a fix in the proxy server related to cookies.
Fixed a threading issue related to the REST interface.
Added classes and methods related to decoding images.
Fixed a bug related to use of the "Breakpoint" button with RunnableScrapingSessions.
Added getStatusMessage, setStatusMessage, and appendStatusMessage to the session object, all of which are synonymous with their corresponding "error" methods (e.g., getStatusMessage = getErrorMessage).
In the web UI changed the column "Error Message" to "Status Message".
Added the following methods to the scrapeableFile object: resequenceHTTPParameter( String key, int sequence ), removeHTTPParameter( String key ), addGETHTTPParameter( String key, String value, int sequence ), addGETHTTPParameter( String key, String value ), addPOSTHTTPParameter( String key, String value, int sequence ), addPOSTHTTPParameter( String key, String value )
Made a DataManager fix where child rows weren't getting inserted for duplicate parent rows.
Changed default user agent for newly-created scraping sessions to Internet Explorer 8.
Now saving in a separate thread so that the GUI won't get locked up for large objects.
Added "Always at the end" option to force scripts to run at the end of a scraping session, even if it gets stopped prematurely.
The prompt to save dialog box only shows on exit when a change has actually been made.
Added a keyboard shortcut to the extractor pattern text box such that when text is highlighted and the Control/Command-T key combination is pressed an extractor pattern token will be generated. This is the equivalent of using the corresponding menu item when the right-click pop-up menu is invoked.
Improved error reporting.
Added local script variables to the breakpoint frame.
When in workbench mode screen-scraper will now breakpoint on a script error.
Now displaying a message on the "Last Response" tab when tidying fails.
Fixed a bug related to saving in the last alpha.
DataRecord keys are now sorted when displaying them in the workbench.
Added session.setRandomizeUserAgent(boolean) method.
Now logging a warning when an extractor pattern token has no regex.
Made a fix related to displaying local variables in the breakpoint window.
Made a fix for "deflate" content encoding.
Fixed a bug related to web servers that use an older version of SSL.
Added the following methods: session.setStopScrapingOnScriptError, session.setStopScrapingOnMaxRequestAttemptsReached, session.setStopScrapingOnExtractorPatternTimeout, scrapeableFile.getMaxRequestAttemptsReached, scrapeableFile.getExtractorPatternTimedOut.
Fixed a bug related to prompting for save upon exit.
Deprecated proxy scripting. Can be re-enabled via the AllowProxyScripting property.
Fixed a minor memory leak in the workbench.
Updated the .NET driver to work with COM-based applications.
Added initial support for memory profiling.
Fixed a bug related to duplicate token editor windows.
Add buttons to wrap text and find within the request/response of a proxy transaction.
Now using %20 instead of + to represent a space character when encoding GET/POST parameters.
Now correctly displaying encoded GET/POST parameters in scrapeable file proxy comparer.
Added search term to the top of the proxy search results window.
Now determining whether or not to save on individual key strokes.
Fixed a bug related to displaying the start page and handling history.
Fixed a bug related to deleting multiple items.
Fixed a few minor memory leaks.
Now stripping internal anchors off of redirect URL's.
Now reloading the page when a scraping session is removed via the web UI.
Fixed a bug in the workbench where the left pane would start out too narrow.
Now including sequence in proxy search results.
Fixed an issue related to renaming scraping sessions.
Added a couple of check boxes to wrap text to the proxy panels.
Made a fix to ensure consistency in line wrapping the last response text box.
Now centering the search result in the proxy.
Fixed text related to the edition to be more consistent.
Fixed a bug related to stopping scraping when an infinite redirect is encountered.
Made a fix related to duplicate proxy session names.
Added sutil.getNumRunnableScrapingSessions and sutil.getNumRunningScrapingSessions.
Modified various methods to be Professionl or Enterprise edition only.
Updated session.downloadFile to use current proxy settings.
Updated tool-tips in script editor to accurately reflect API changes.

Public Release 5.5 (03.30.11)

Extractor Token Delimiters: Removed magic duplication of token delimiters ('~@' and '@~') when a token is edited along side an empty extractor token.
stringToFloat method: Added a stringToFloat method in sutil object.
getConnectionTimeout method: Added a getConnectionTimeout method in the session object.
setConnectionTimeout method: Added a setConnectionTimeout method in the session object.
getLoggingLevel method: Added a getLoggingLevel method in the session object
setLoggingLevel method: Added a setLoggingLevel method in the session object
DataSet Viewer: Added a memory so that it will open in list or grid first based on its state the last time it was open.
Duplicate Scraping Sessions: Renaming a scraping session to the same name as another is no longer permitted when selecting to rename it from the right click menu.
Extractor Pattern Pasting: If you paste a copied extractor pattern that has been deleted it no longer adds a bunch of scripts.
Mac Export: When overwriting a file on export, Mac's second prompt before overwriting was removed. Once is enough.
Regular expression textbox: The enter textbox in the general tab of the extractor token editing window now resizes with the window.
Tidy Fix: Basic edition was stuck with Tidy turned on even when you turned it off. That was been fixed so that tidy responds to the off request.
Window Artifacts: When a pop-up window is closed the work bench will redraw itself to remove artifacts from the pop-up.
Runnable Scraping Session: Fixed error that stopped runnable scraping sessions from running when session was not passed.
Detect JS Cookie: Detect JS Cookies button was added to help identify where cookies are being set using JavaScript.
Extractor Token Delimiters: Fixed so that when two extractor tokens share a delimiter only the first is recognized as a token.
removeHTTPHeader method: Added the removeHTTPHeader method to the scrapeableFile object.
Regular Expression Hover: When you place your mouse over an extractor token it now displays the regular expression associated with it in a tool tip. There si a delay on it so that it is not overly annoying.
Script Instances Window: Now in the script instances window it specifies if the script is enabled or disabled. The window has also been updated to size itself so that it remove the horizontal scrollbar.
Paste Extractor Pattern: To avoid the issue of having it look like your screen goes blank when you paste an extractor pattern with lots of sub-extractor patterns, when you paste an extractor pattern the screen no longer jumps to the bottom of the new extractor pattern. It now doesn't jump at all.
Folder Delete: Minor fix so that if you delete a folder immediately after importing into it the delete will take effect the first time.
No to All: If you import a scraping session with attached scripts you can now choose the option No to All to have screen-scraper not replace any of the conflicting scripts.
Proxy Port Resets: If you change the port on a running proxy session it will change without having to stop and restart the proxy session.
Paste Extractor Pattern: When you paste an extractor pattern you now jump to the top of the new pattern.
Jython Libraries Updated: The jython libraries that process python were updated from 2.1 to 2.5.1. The standard python libraries were added at lib/jython-lib. The folders lib/ext, lib/jython-lib, and lib/jython-lib/site-packages are now included in python's search path.
No to All Improvement: If a script is not to be overwritten on import the No to All will avoid the warning that you cannot overwrite the script, as you are not trying to.
Secondary Server for Anonymization Service: We have added a secondary server to handle automatic anonymizations. By default you will continue to use the current server, if you would like to change to the other server you can add the property AnonymizationURLPrepend to your screen-scraper.properties file. The only acceptable values are http://anon.screen-scraper.com and http://anon2.screen-scraper.com.
Compare Last Request and Proxy Transaction Window Scrollbar: The scroll bar is particularly added for POST data that gets very long. Instead of disappearing off the bottom of the window a scroll bar will now appear.
Mail Server Settings: The form fields in settings for a mail server were removed in the Professional edition since those methods are Enterprise only.
Regular Expression @ Fixed: In updating one of the extractors got messed up so that it got confused by the at sign (@). It has been set right again.
Regex Stuck on Screen: When you navigated away from screen-scraper the Regex toll tip would get stuck on the screen. The screen now redraws so that it disappears.
Edit Token Option: The edit token option is now only available when the token is a valid token.
Window Sizes: The DataSet and Compare Last Request and Proxy Transaction windows now retain their size from the last time they were open.
NTLM Authentication Refresh: Workbench was retaining information incorrectly from last NTLM request. This caused it to not log in correctly when run again. It now clears the HTTP state between scrapes.
Jython Library Load File: The file created to take care of adding the python libraries did not deploy with the other updates. This caused python scripts to fail entirely. It has now been resolved.
Code Completion: The code completion has been brought up to date with previously undocumented methods as well as the current alpha methods.
REST Interface: Updates were made to the rest interface to facilitate tracking a session and starting a session with passed variables.
Mail Updates: Added support for connecting to a mail server using TLS/SSL. The mail server port can also now be specified.
Logging Updates: The various logging methods can now take Objects and not just Strings.
Mac Fix: With the update of JRE 1.6.0_22 on Mac OS X the "sss" file extension was being trunctaed when exporting.
Anonymization Fix: In some cases anonymous proxies would spawn, but never become available. Added code to handle this situation.
Made a bug fix that arose when available anonymous proxy servers was depleted to zero.
Now disallowing running multiple screen-scraper interfaces simultaneously. For example, previously the screen-scraper workbench could be run concurrently with the server. This ended up causing database corruption in some cases, though, so we're now disallowing it.
When clicking a search result after performing a find in a proxy session the HTTP transactions table will now scroll to the corresponding transaction.
When clicking a search result after performing a find in a proxy session if the associated proxy session isn't visible in the right pane it now will be.
In exporting objects if an XML comment was found in any of the text fields the resulting exported file would contain an invalid sequence of characters.
Fixed a scrolling bug related to displaying script instances associated with extractor patterns.
Removed a log message that was appearing each time a redirect occurred.
screen-scraper will now display a "start page" when the workbench initially launches.
Based on feedback, now allowing running the screen-scraper workbench and server simultaneously by adding the "AllowMultipleSimultaneousInstances" property to the screen-scraper.properties file.
Fixed a bug where screen-scraper would freeze up when very large requests were included in proxy sessions and scrapeable files.
Fixed a bug on Mac OS X where an overwrite prompt was not being given in exporting scraping sessions.
Fixed a message formatting issue in certain script errors.
Fixed an issue with anonymous proxies being terminated externally.
When creating a new extractor pattern token screen-scraper will now attempt to guess the regular expression that should be used.
Fixed a bug related to editing extractor pattern tokens.
Fixed a bug related to highlighting of data records in the last response tab.
Optimized highlighting of data records in the last response tab.
Fixed a bug related to selecting extractor pattern tokens.
Token editor now saves and closes when the return key is hit.
Fixed a bug related to finding script instances.
Updated proxy to use HttpClient 4.
Fixed a bug related to the recent update to the proxy.
Including a UseGlobalExternalProxyForAllScrapingSessions property in the screen-scraper.properties file will now case global proxy settings to apply to all scraping sessions.
Fixed a minor bug related to invalid extractor pattern token names.
Undo in certain text boxes can now be triggered properly via keyboard shortcut on a Mac.
Now notifying the user if there are no matches when the "Highlight Extracted Data" button is pressed.
The "Last Response" tab can now be displayed in a separate window.
Fixed a bug related to the anonymization service.
The DataManager now handles reserved words correctly.
Fixed a bug related to data extraction timeout.
Fixed a bug related to requests being recorded with redirects.
Fixed a bug related to hitting the "Enter" key in the find dialog box.
You can now wrap text in the last request and last response panels.
Rearranged elements on the last response panel so that overlapping shouldn't occur.
The delay on the script auto-complete box can now be set via the "AutoCompleteDelay" property in the "screen-scraper.properties" file.
Rearranged elements in the proxy "Progress panel" so that they don't overlap.
Now dismissing the splash screen before the start page loads.
The name text box is now highlighted when proxy sessions, scraping sessions, and scripts are created.
Adjusted a few visual elements related to proxy sessions so that they resize correctly.
Now filtering out "sitecheck" requests made by Opera.
Table columns in the "HTTP Transactions" table are now being sized correctly even when the table is empty.
Fixed a bug where less-than symbols weren't always showing up in the tool-tip for extractor pattern tokens.
Restored the horizontal scroll bar in the last response tab.
Fixed an error that caused screen-scraper to disallow testing extractor patterns.
Fixed a minor bug related to Java keystores.
Fixed a bug related to the data set list view not displaying correctly.
Fixed an issue where anonymous proxy pool would not automatically repopulate when proxies were terminated automatically.
Fixed an issue in Linux where the extractor pattern panel was a bit too large.
Fixed an issue in Linux where the scraping session log panel was a bit too large.
Altered how character sets are handled in terms of how specifically set character sets override more global settings.
Long parameter values can now be edited in a separate text box.
Fixed an issue with extractor pattern token tooltips.
Fixed an issue with sub-extractor panels not sequencing after deletion.
screen-scraper will now display an error message when an invalid regular expression is entered for an extractor pattern token.
Fixed an issue with resizing the proxy transaction compare window.
Fixed a bug where the paste sub-extractor pattern was becoming enabled after a sub-extractor pattern had been deleted.
Fixed a bug where data record highlighting wouldn't work correctly with very large HTML pages.
Fixed a bug where parameters sent in a multi-part request were causing invalid responses.
The position of the divider bar on the split pane for proxy sessions is now retained.
Numeric columns in tables are now rendered using the default font.
Fixed a minor bug related to editing extractor pattern tokens.
No longer truncating HTML in the "Last Response" tab.
Minor bug fix to the DataManager.
Fixed a bug related to setting the originator edition when exporting.
The cursor now returns to normal after attempting to highlight data records for a pattern that doesn't match.
Fixed a bug where data records were not highlighting in the last response the very first time.
Fixed an issue where scrollbars weren't appearing in the proxy/scrapeable file compare window.
Now displaying an error message when applying invalid extractor patterns.
Fixed a minor memory leak in the workbench.
Fixed a bug related to highlighting data records.
Fixed a bug where the scrapeable file view wasn't updating correctly in some cases.
The "Generate scrapeable files in..." menu will now scroll when it contains many items.
The term "sutil" will now appear in blue in the script editor.
When exporting an object it will now be selected in the tree.
Fixed a bug related to the proxy / scrapeable file comparer.
Updated the PHP driver so that it now detects when it can't connect to the screen-scraper server.
The "Runnable" tab in the web interface will now show the most recently run instance of a particular scraping session.
Enhanced error message when screen-scraper is inhibited by a local firewall.
Fixed a link to sub-extractor pattern help.

Public Release 5.0 (06.30.10)

feature: added REST interface
feature: can now filter out less useful proxy transactions
feature: added DataManager to facilitate saving data to a database
feature: generate multiple scrapeable files from proxy session
feature: made button bar persistent for extractor patterns
feature: retained number of lines to display for scraping session log between sessions
feature: updated scrapeable file icons to indicate when they are and are not invoked in sequence
feature: added a delete option for scraping sessions to web interface
feature: enhanced data set viewer with list view and colored tokens
feature: improved script error messages
feature: added a method to allow HTTP parameters to be removed from scrapeable files
feature: added logging levels to scraping session
feature: added ability to compare request in scrapeable file with transaction in proxy session
feature: enhanced breakpoint window to show more information, such as current script and number of scripts on the stack
feature: added syntax highlighting to extractor pattern pane
feature: added ability to pause/breakpoint a scraping session with a button
feature: extracted data can now be highlighted in last response tab
feature: pane now scrolls down when an extractor pattern is added
feature: character set can now be determined on a scraping session and scrapeable file level
feature: added ability to limit length of response for a scrapeable file
feature: enhanced handling of database backups over time
feature: can now add more session variables to a scheduled scraping session in the web interface
feature: added ability to clear completed scraping sessions from web interface
feature: enhanced a few default regular expressions
feature: properties file can now be reloaded from the web interface
feature: can now copy and paste sub-extractor patterns
feature: added ability to trim white space from extracted data
feature: added a couple of new options to invoking scripts from an extractor pattern
feature: added sutil to handle more general methods
feature: provided a way to null out session variables for tokens that didn't match
feature: provided a way to save data sets without appending to an existing data set
feature: added session.setMaxConcurrentFileDownloads
feature: added ability to install multiple screen-scraper services in Windows
feature: now higlighting selected words in script text pane
feature: added code completion and macros to script pane
feature: now using syntax highlighting in last response tab
feature: added alternative HTML tidier
feature: added notes column to proxy
feature: added getv and setv to session object
feature: now limiting script stack size in order to avoid memory problems
feature: added ability to force files to be regarded as non-binary
feature: added scrapeableFile.connectionTimedOut
feature: added find feature in proxy session
bugfix: rearranged and made redundant some GUI elements to make working with scrapeable files easier
bugfix: extractor pattern token window no longer scrolls to the bottom when a new token is added
bugfix: scrollable panes no longer scroll to the bottom when first viewed
bugfix: now retaining scroll position in panes when user selects various tabs
bugfix: the find dialog box now appears within screen-scraper's frame by default
bugfix: improved default open/save dialog box on Windows and Mac OS X
bugfix: added message if DATARECORD is absent when a sub-extractor pattern is added
bugfix: fixed resizing of child elements in breakpoint window
bugfix: fixed a bug where scraping session notes couldn't be deleted
bugfix: now clearing main panel when a folder gets deleted
bugfix: fixed a bug where a copied extractor pattern would retain script instances
bugfix: main panel is now getting cleared when a script gets deleted
bugfix: now updating list of scraping sessions for proxy session when a scraping session is renamed
bugfix: session.getNotes() was generating an exception
bugfix: pop-up windows are now appearing closer to the mouse cursor
bugfix: fixed an issue where scrapeable files couldn't be generated from certain proxy transactions
bugfix: made various fixes to proxy so that it more accurately identifies binary and non-binary responses
bugfix: fixed a bug where extractor patterns weren't being generated from selected HTML
bugfix: now clearing lower pane when proxy transactions are deleted
bugfix: fixed dataSet.writeToFile so that column headers are updated correctly
bugfix: now remembering wrap text state in scripts
bugfix: now properly resequencing scrapeable files upon deletion
bugfix: now accurately indicating when a request is multi-part
bugfix: fixed an issue where logs were being truncated
bugfix: improved handling of international characters in RemoteScrapingSession
bugfix: fixed an issue on import when character set wasn't indicated
bugfix: improved handling of hard returns in extractor patterns containing embedded variables
bugfix: improved error message on export
bugfix: improved handling of null values with data records
bugfix: will now recreate log file and continue logging when log file gets deleted
bugfix: fixed an issue where extractor patterns weren't getting highlighted properly after edit
bugfix: improved handling of large proxy transactions
bugfix: fixed an issue when resolving certain URL's from relative to absolute
bugfix: now exporting scripts that are invoked via session.executeScript
bugfix: improved handling of breakpoints in server mode
bugfix: fixed an issue where script pane wasn't being updated on import
bugfix: fixed an issue where tokens with duplicate names in sub-extractor patterns weren't being saved properly
bugfix: made running time human-readable in web interface
bugfix: user's IP address is now displayed when access is denied
bugfix: fixed a bug where the extracted data window couldn't be displayed while the breakpoint window was visible
bugfix: no longer overwriting .vmoptions files
bugfix: now using scraping session character set when exporting
bugfix: the ? character is now disallowed in object names
bugfix: logging level was always defaulting to debug when invoking scraping sessions from the command line
bugfix: can now copy text from the last request tab
bugfix: now displaying an error when invalid regular expression is entered in token
bugfix: now coloring text in log when tidying fails
bugfix: enhanced resizing of table columns
bugfix: fixed an issue where GUI would freeze up when applying an extractor pattern while scraping session was running
bugfix: made a few minor fixes to the .NET driver
bugfix: enhanced Ruby driver to be more Ruby-like
bugfix: updated XML libraries
bugfix: fixed a bug where an exception was being through when values were blank in file used by session.loadVariables
bugfix: fixed BASE HREF issue when viewing HTML in local web browser
bugfix: now deprecating unstable Windows features, including using IE as the HTTP client and allowing VBScript as a scripting language
bugfix: now allowing parentheses to be used in regular expressions, as well as back references
bugfix: fixed an issue where a script wasn't being deleted when it's parent folder was deleted
bugfix: fixed an issue where headers were being munged in certain redirect responses
bugfix: made visual sequencing of invoked scripts more logical
bugfix: improved progress bar when downloading an update
bugfix: improved placement of pop-up windows in web interface
bugfix: session.loadVariables now allows spaces before and after = symbol
bugfix: text not wrapping by default if checkbox was checked in script pane
bugfix: auto-refresh not occurring in web interface if checkbox is initially checked
bugfix: no longer requiring web.htm in web interface URL
bugfix: now highlighting button corresponding to current section in settings dialog box
bugfix: enhanced icons in Windows
bugfix: added icons for all menu items and buttons
bugfix: updated PHP class for better backward compatibility

Public Release 4.5 (03.04.09)

feature: syntax highlighting in the script editor
feature: added icons to menu items
feature: logging levels (debug, info, warn, error)
feature: general optimization in both workbench and server
feature: session.clearAllSessionVariables
feature: context menu on root folder
feature: scripts now automatically import com.screenscraper.common.*
feature: port conflicts now being output to the error.log file
feature: added method to determine whether or not session is running
feature: emails sent regarding anonymization status
feature: ability to log int values
feature: user notified when database connection is lost
feature: scripts can be force overwritten via a property in the properties file
feature: time stamp added to exported scraping session file
feature: look and feel can be set
feature: memory usage displayed in workbench
feature: memory usage accessible via a method call
feature: user is notified in proxy session if external proxy is set
feature: custom HTTP headers can be added
feature: added "Nickname" property
bugfix: proxy sessions weren't handling web sites utilizing non-standard port numbers
bugfix: fixed quirks with scrapeable file names
bugfix: bug in reordering scrapeable file parameters
bugfix: scraping session start/stop button not updating properly
bugfix: in some cases not displaying correct script in breakpoint window
bugfix: mapping sets and options on tokens not applying in data set window
bugfix: invalid URL message not being trapped properly
bugfix: problem importing scraping sessions with unusual names
bugfix: double prompts when overwriting scripts
bugfix: tidying can now be turned off in basic edition
bugfix: now producing a better error message when exporting a scraping session using an invalid character set
bugfix: highlight was offset in some cases in finding text
bugfix: clicking a proxy transaction with large post data was chewing up CPU
bugfix: session variables can be embedded in extractor patterns (again)
bugfix: visual state for proxy sessions not being retained
bugfix: token options and mappings not applying when data is extracted manually from a script
bugfix: cleared up quirks with script instance drop-down list
bugfix: last number of records scraped inaccurately recorded in web interface
bugfix: notify user when script doesn't import because existing script is not to be overwritten
bugfix: scraping session file locked after export
bugfix: scroll and cursor location not remembered on last response tab
bugfix: context menu in log not displaying correctly
bugfix: settings window is resizable to accommodate different font sizes and screen resolutions
bugfix: parameter type drop-down list disappearing when parameters were deleted

Public Release 4.0 (01.21.08)

feature: web interface for scheduling and managing scrapes
feature: added real-time integration with external applications
feature: automatic anonymization
feature: scrapeable files and extractor patterns can be copied and pasted
feature: added a "notes" to scraping sessions
feature: improved cookie compatibility
feature: added sequence to sub-extractor patterns
feature: scraping sessions can now be run directly from the command line
feature: HTML entities can now be automatically converted from scraped data
feature: cookies can be cleared for a scraping session
feature: last response for a scrapeable file can now be viewed in a browser
feature: current time and elapsed time can be output in a script
feature: greatly improved look 'n feel on mac os x
feature: added new regular expressions
feature: "update.zip" files will be decompressed and imported
feature: objects in the tree can be deleted with the "delete" key
feature: enhanced the "status" bar
feature: the licensed email address now appears in the "about" screen
feature: the default file extension for exported objects is now "sss"
feature: a "start/stop scraping" button was added to the scraping session "log" tab
feature: HTML can be automatically stripped from extracted data
feature: screen-scraper can check for updates on startup
feature: enhanced installers
bugfix: mappings were not being imported properly from exported scraping sessions
bugfix: null interpolated session variables were not being properly handled
bugfix: "deflate" encoding was not being properly handled
bugfix: in some cases sequence numbers were being duplicated for scrapeable files
bugfix: in certain cases folders could not be deleted
bugfix: the proxy server was misidentifying some files as binary
bugfix: the "last response" tab was blanking out prematurely in some cases
bugfix: now catching class loader exceptions for jar files compiled with a higher java version
bugfix: ports weren't being displayed for SSL URL's in the proxy
bugfix: exceptions thrown in scripts were causing some subsequent scripts not to be executed
bugfix: various fixes for windows vista
bugfix: mapping sets were not always being deleted properly
bugfix: multiple command line instances were not being handled properly
bugfix: drag 'n drop to folders in some cases wasn't working
bugfix: double-clicking extractor pattern tokens didn't always allow them to be edited
bugfix: extractor pattern tokens were getting repeated after editing a token
bugfix: too high sequence numbers for extractor patterns was causing them to disappear
bugfix: new scripts weren't being sorted properly
deprecated: embedded session variables in extractor patterns
deprecated: the "Optional?" flag for extractor pattern tokens
deprecated: the "Run Script" button
deprecated: automatic joining of data sets
deprecated: RunnableScrapingSession for everything but enterprise edition

Public Release 3.0 (01.10.07)

feature: added a "Find" feature to the scraping session log and script panel.
feature: the scraping session log can now be limited to a specified number of lines.
feature: the scraping session log can automatically remain scrolled to the end.
feature: scripts can now be called from other scripts.
feature: the database now gets backed up automatically.
feature: screen-scraper can now be registered in a GUI-less environment.
feature: tab state is now preserved when moving between objects.
feature: added context menus for editing commands.
feature: upgraded Mac interface to be like Windows and Linux.
feature: added a new library used to write out XML from scripts.
feature: enhanced firewall handling.
feature: for new installs, the user is now referred to the tutorials.
feature: screen-scraper now checks for blocked ports on startup.
feature: added a method to load and save session state between sessions.
feature: integrated a new HTML renderer.
feature: objects can now be organized into folders.
feature: improved "Strip HTML" feature.
bugfix: fixed an issue related to passing in remote variables containing the ! character.
bugfix: fixed an issue related to truncated error messages in scripts.
bugfix: when invoked from the command line with no parameters the "params" variable was coming through as void.
bugfix: in some cases duplicate scripts were showing up on import.
bugfix: there was an issue related to saving while a command line instance was running.
bugfix: fixed an issue in the proxy related to URL's containing multiple adjacent slash characters.
bugfix: in some cases the database was closing prematurely.
bugfix: fixed an issue related to repainting after an extractor pattern was added.
bugfix: the "breakpoint" window wasn't always updating properly.
bugfix: addressed issues related to database corruption.
bugfix: fixed a bug related to tildes in URL's.
bugfix: made multiple fixes related to international character sets and non-ASCII characters.
bugfix: fixed a few issues related to running screen-scraper in various modes simultaneously.

Public Release 2.7.2 (03.24.06)

bugfix: updated the http-client library to accept all SSL certificates.
bugfix: in certain situations the database was getting closed prematurely when screen-scraper was invoked from the command line.

Public Release 2.7 (03.08.06)

feature: screen-scraper can now generate RSS feeds from scraped data.
feature: added session.addToSessionVariable method.
feature: log messages have been enhanced and clarified.
feature: all of screen-scraper's ports are now settable in the properties file.
feature: the web server can now be disabled.
feature: because of a bug in the third-party library that handles the VBScript engine we included a warning in screen-scraper when using VBScript.
bugfix: hot swapping scraping sessions and scripts has been improved.
bugfix: the server can now be run via the shell scripts on more recent versions of Mac OS X.
bugfix: a few fixes were made to increase database robustness.

Public Release 2.6 (11.01.05)

feature: international character sets are now supported.
feature: files can be uploaded within scrapeable files.
feature: added scrapeableFile.saveFileOnRequest, which allows for binary files to be downloaded via POST requests.
feature: added session.reformatDate, which allows for extracted dates to be reformatted.
bugfix: fixed bugs where harmless SQL errors were being generated.
bugfix: under certain circumstances errors would occur when proxying binary files.

Public Release 2.5 (08.02.05)

feature: automatic hot swap from the "import" folder on start-up
feature: scripts can be stopped mid-stream
feature: tidying settable on a scrapeable file level
feature: external proxy settable on a scraping session level
feature: workbench, server, and command line can be run simultaneously
feature: added a system tray icon for the server when running on Windows
feature: added scrapeableFile.extractData and scrapeableFile.extractOneValue
feature: added "mappings" feature for extractor pattern tokens
feature: implemented saving and loading of state
feature: caching of data sets
feature: filtering duplicates from data sets
feature: regular expressions can now be designated from a drop-down list
feature: HTML can be automatically stripped from extracted data
feature: requests can be made multiple times for a URL in case of failures
change: multiple script instances can be deleted at once
change: text box is highlighted in the "find" dialog box by default
change: changed highlight color for "find" feature
change: "last response" is now cleared before exporting
change: installer now sets working directory and installs COM driver
change: enhanced dataSet.writeToFile
change: added "Strict Mode" cookie policy
change: upgraded some third-party libraries
change: performed a number of code optimizations
bugfix: an error message related to help files was being output to the error log
bugfix: dataset window spawned from "breakpoint" dialog window wasn't getting initial focus
bugfix: resolved database corruption issues
bugfix: server now generates logs by default
bugfix: scrapingSession.downloadFile now makes use of existing cookies

Public Release 2.0 (02.02.05)

feature: option for disabling log file generation when run as server
feature: sending email through scripts
feature: SOAP connection support
feature: updated look and feel
feature: button bar for commonly used tasks
feature: status bar for application messages
feature: screen-scraper is automatically installed as a service in the professional edition
change: single "Import..." menu item instead of choosing between scraping sessions or scripts
change: "Yes to all" on import
change: merge cookie drop downs menu items in scraping session general tab
bugfix: new scripts with the same name will get an icremented number
bugfix: vbscript scripts can no be invoked when in server mode

Public Release 1.5 (09.11.04)

change: HTTPS is now handled with a temporary secure certificate
change: Rename gui to workbench
feature: Cookie handling option in scraping sessions
feature: .Net connector added
feature: Local files can now be scraped
feature: Delete table rows by right-click and pop-up menu
feature: Edit menu w/ copy, paste, etc. for text boxes
feature: Allow selection and deletion of multiple HTTP transactions from table
feature: Undo/redo on text boxes from Edit menu
feature: Search function in "Last Response" tab
feature: Script instances can be enabled/disabled
feature: Save and restore last window size.
feature: Data sets can be written to a delimited file
feature: Basic, Digest or NTLM Authentication handling in scraping session
feature: Hot deploy by copying scraping sessions and independent scripts to import dir
feature: Breakpoint debugging in scripts
feature: Extensibility by adding custom jars to the ext dir
bugfix: Extractor pattern token data is now saved by default when editor window is closed
bugfix: Confirm overwrite on export
bugfix: When an error occurs in getting the html page the http code is displayed in the log such as 404, etc.
bugfix: "Chunked" tranfer encoding now handled properly in proxy server
bugfix: New scraping sessions and scripts default names will increment

Public Release 1.2 (06.02.04)

Numerous bug fixes and optimizations
Sub-extractor patterns
More flexible cookie handling
New methods added to built-in screen-scraper objects

Release 1.1.5 (10.01.03)

Several bug fixes and a few minor feature enhancements.
Two new tutorials are now available.

Release 1.1 (09.02.03)

Numerous bug fixes and minor feature enhancements
Internal scripts can now be written in Interpreted Java, JavaScript, JScript, Perl, Python, or VBScript.
The current scrapeable file can now be accessed within a script, also allowing access to the full data scraped for a page.
A method can be called to determine if an error occurred while the file was being requested.
Scraping sessions can be paused in a script.
Maximum number of concurrent scraping sessions can be set via a property.
The connection timeout can now be set via a property.

Release 1.0 (07.31.03)

Numerous bug fixes and minor feature enhancements
Improvements to sever security
Extracted data can automatically be saved into session variables
Extracted data can be joined or appended to existing data sets
Significant improvements to the install procedure
Imrpovements to documentation
Self-updating when new versions become available
Improved usability of running screen-scraper as a server

Release 0.9.5b (06.12.03)

Various improvements in documentation
Several bug fixes and minor feature enhancements were made.
Several optimization and memory leak issues resolved.
Data set and data record objects can be accessed from remote sources (e.g. ASP or PHP scripts.
A lock file now gets generated when screen-scraper starts up in order to allow only one instance to be run at a time, avoiding potential database corruption.
Basic authentication parameters are now associated directly with a scrapeable file.

Release 0.8.7b (05.27.03)

Includes several bug fixes and feature enhancements.
Allows screen-scraper to import and export objects.
Improved support for external proxies, including those that make use of NTLM.

Release 0.8.6b (03.04.03)

Fixes several miscellaneous bugs.
screen-scraper can now clean up HTML using JTidy in order to facilitate data extraction.

Release 0.8.5b (02.18.03)

Fixed a bug in the proxy server that garbled some URL query strings.

Release 0.8.5a (02.08.03)

screen-scraper now uses HttpClient (http://jakarta.apache.org/commons/httpclient/) to handle all of the HTTP transactions, which allows for a broader range of sites to be correctly scraped.

Release 0.8.4b (01.15.03)

added ability to invoke screen-scraper from the command line
added ability for screen-scraper as a server
creating language bindings for Java, PHP, and COM
when viewing the last response from scrapeable files HTTP headers are now
patterns can be formed by highlighting HTML
extractor tokens can be created from highlighting HTML

Release 0.8.2b (11.17.02)

context-sensitive documentation added
several bug fixes and feature enhancements
added support for an external proxy server
added "settings" dialog

Release 0.8b (10.22.02)

initial release