Fatima Athar

May 24, 2019

Scrapy and Realtime Crawling: To Do or Not To Do?

If you read our recent blog post, you know we’re big on using Scrapy for a lot of our large-scale website crawling needs. But there’s another […]
May 24, 2019

Large-scale website scraping issues and the tools to resolve them

In 2011, a team of scientists led by Martin Hilbert discovered that the total amount of data in the world in 2007 was 295 optimally compressed […]