Thoughts, tips, and updates on screen-scraping
Once in a while when you’re scraping you may request a file that ends up being really large, but you actually only need to pull data from the top portion of the file. If it’s a big file it can end up slowing down the scraping process quite a bit. Not too long ago (somewhere … Read moreCapping response length
Within screen-scraper you have the ability to call outside programs directly from your scripts. The following is an example scraping session that makes use of Tesseract OCR and Imagemagick in order to take an image from the internet and attempt to read the text of the image. As is, the scraping session is intended to … Read moreUsing OCR with screen-scraper
Any reasonably-sized software development project benefits greatly from some type of version control system, such as CVS, Subversion, or Git. Internally we use Subversion, and I thought it might be helpful to share a bit how we go about it. What I describe here is primarily applicable to a project where you have many scrapes … Read moreHow we use version control
Many have probably noticed that when a scraping session is exported from screen-scraper all of the scripts invoked from within that scraping session get exported along with it. All of the scripts, that is, except those that get invoked via the session.executeScript method. The exporter isn’t quite smart enough to actually parse the text of … Read moreExporting scraping sessions that use session.executeScript
We previously listed some means to try to stop screen-scraping, but since it is an ongoing topic for us, it bears revisiting. Any site can be scraped, but some require such an influx of time and resources as to make it prohibitively expensive. Some of the common methods to do so are: Turing tests The … Read moreFurther thoughts on hindering screen-scraping
Some of the sites we aspire to scrape contain vast, huge amounts of data. In such cases, an attempt to scrape data from it may run fine for a time, but eventually stop prematurely with the following message printed to the log: The error message was: The application script threw an exception: java.lang.OutOfMemoryError: Java heap … Read moreTechniques for Scraping Large Datasets
Microsoft ASP.NET sites have consistently proven to be some of the most difficult to scrape. This is due to their unconventional nature and cryptic information passed between your browser and the server. You’ll know you’re at an ASP.NET site when your URLs end in .aspx, your links look like this: javascript:__doPostBack(‘gvLicensing’,’Select$0′) And your POSTs look … Read moreScraping ASP.NET Sites
Once screen-scraper extracts data from a web site, typically that data is sent somewhere else. Data is probably most commonly written out to a file, but may also be saved to a database or even submitted to another web site. You can always handle the scraped data in screen-scraper scripts, but what if you want … Read moreHandling scraped data in real time
In certain cases a scrape needs to be anonymized in order to get the data you’re after. Generally this means sending the HTTP requests through one or more proxy servers, over which you may or may not have control (see How to surf and screen-scrape anonymously for more on this). Up to this point, this … Read moreAnonymization through proxy servers
Well, this is a topic I’ve been meaning to address for quite a while, and a recent support request on the topic pushed me to finally get it done. What I’ll describe in this article applies to our own screen-scraper software, but would also apply to most any other screen or web scraping software you … Read moreHow to surf and screen-scrape anonymously