Behind the Scraper: What is it, How it works, Practical Approaches and Pitfalls

Nimra EjazPosted on May 12, 2025

17-18 Min Read Time

Data is needed everywhere. In every emerging technology, data is the basis of the innovations. 

But how can we get the required data? 

What are the resources from which we can collect data, and what are the methods that can help us in extracting data quickly and efficiently? Where can we store extracted data, and how can we secure it? 

In this blog, we are going to talk about data and its major extraction method - scraping.

Scraping

In scraping, we develop spiders. We can think of it as an algorithm that is meant to extract desired data dynamically, even when you are not at your computer, until the source isn’t changed by its makers.

If it’s changed, then we need to fix the spider again and again, and the process can go in a loop sometimes, when makers can’t make up their minds on the final version.

Sometimes it may seem like a waste of effort to develop scrapers for such sites. However, we have a solution for this problem as well. We can directly target the maker’s source so we can get data from there instead of constantly accessing a changing in-hand platform. 

The Objective

The goal of this extraction is to collect the right data at the right time under the right circumstances. Therefore, our results remain useful for the objective for which the data is being collected. Nowadays, the right data is the underlying power of every latest discovery or invention in every aspect. 

We need data because we are inquisitive to know what’s happening in the real world, and whether targeting a particular problem will be helpful for anyone or not. 

Extraction Method:

The target is to collect structured, unstructured, and semi-structured data from various sources, like text files or databases. It can also include JSON files, CSV, Excel sheets, repositories, APIs, and websites.

But before getting into technical details, here are the meanings of a few technical terms that are widely used. 

1. Fetching: Download the source.
2. Parsing: Identifying the content's structure.
3. Extracting: Collect the desired data from the parsed content.
4. Cleaning: Clean and transform the extracted data into the required format. 
5. Storing: Store the data in a reliable place.
6. Utilizing: Use this data for further analysis or processing.

Thanks to multiple open source frameworks available, which can provide you with support for overcoming all the hurdles using their built-in templates, middlewares, pipelines, and settings. 

Notable tools and libraries 

A few of them are discussed below

The Right Data Extraction Technique

Choosing the right tool depends on several factors. These include the source of the data. The complexity of the source also plays a role. Another key consideration is the amount of data we need to extract. The format in which we need to store the data is equally important.

Specific requirements for the extraction process will influence the decision. The frequency at which the data needs to be updated must be accounted for. Environmental and circumstantial factors may further shape the choice. Available resources, such as budget or infrastructure, are critical to evaluate.

Team expertise will determine which tools are practical to implement. Project timelines add another constraint. Lastly, the cost per add-on facility could impact the final selection.

For this blog, we will focus specifically on web scraping as the primary method.

Web Scraping Applications

Web scraping is contributing to several fields where we need validation before reaching some inferences. The huge amount of new and historical web data is gathered for AI analysis, interpretations, researching new concepts, and training machine learning models. We can also evaluate new trends and preferences through the audience’s interests, their sharing trends, and chatbot conversations. 

Most of the time, investors need data to analyze the target audience and their interest in the investment, and to get the analysts' reports and other economic indicators. It also includes tracking competitors’ websites, monitoring their hot-selling items, sales, prices, stocks, or identifying target public preferences in a particular region. 

These analyses are significantly impacting all aspects of the ecosystem. These developments are benefiting both the businesses and the public.

Ethical Extraction

Contrary to the successful extraction of desired data, there is always a thin line between accessing someone's property ethically and making trouble for them to benefit from their own property. Whenever we are accessing data for visualization, we can’t extract a small chunk of data, because we can’t infer from a piece. 

Here is a point to remember: a huge amount of data doesn’t mean the right data, which will make the right interferences, but every time we need enough right data so it can make the results worth considering. Gathering that enough amount of data means accessing that source sufficient times, making the servers slow for them and their customers, or even putting them in trouble to stress about making their security strong enough so that not only hackers but also scrapers couldn’t wrongly benefit from their property. 

The scraping could be ethical if we gather data that is publicly available, and access it in the way it’s intended to be used, like their own target audience is accessing or by following the robots.txt instructions (which parts of the site could be accessed legally). Additionally, we can’t extract data during their active business hours, making it unavailable for their audience. Even in off hours, we can’t engage their servers entirely by hitting it with millions of requests in a short time.

Prerequisites

Scraping is not an easy task in the first place. Along with performing the reverse engineering process, we also need to take care of getting rid of bot detection and extracting all the data in minimal time and cost. This requires a lot of creativity. 

While making sure to have a strong understanding of all kinds of emerging technologies coming in for developing all those source platforms. We also need to be aware of all e-commerce storefronts. Platforms can’t be reverse processed until we have a fair understanding of how all these things are developed in the first place. 

If you have a fair understanding of web, HTML DOM, CSS, and XPath selectors, you wouldn’t struggle. 

Let’s dive a little deeper to understand the core features of web scraping.

1. Requests: To fetch the required web page, web scraping tools send HTTP, HTTPS requests to the target website’s server.
2. Responses: Once the web page is fetched, the HTML, XML, or JSON content is parsed. Parse the response using different libraries to load it in a navigable format. 
3. Data Extraction: To extract the specific data from the page, we can use selectors like CSS or XPath.
4. Data cleaning and transformation: The extracted data can be cleaned and transformed to remove unwanted elements and deduplicate the data or convert it into a more usable structure.
5. Storage: Finally, the structured data is stored in the desired and optimized format (i.e, JSON, database). This data can be directly used in business intelligence BI tools.

Data Extraction with Scrapy

In today’s world, using the right tool and framework is crucial. By taking an optimized approach, we can produce the best results. 

For instance, let’s talk about Scrapy. It provides a complete framework to extract data from the web. It takes care of all scraping needs, whether it’s related to crawling websites, processing the responses, or handling common scraping challenges. It provides everything under the same roof by being a complete package rather than just a parsing library.

It includes:

• Built-in features (Request queuing, scheduling, middlewares, item pipelines, data exports)
• Speed and scalability
• Asynchronous processing
• Robust error handling and a retry mechanism

Spider Creation

Install python3-venv and create a Python virtual environment.

$ cd /scrapy_tutorials
$ python3 -m venv venv
$ source venv/bin/activate

Then, install Scrapy in the above virtual environment.

$ apt-get install python3-pip
$ sudo pip3 install scrapy

Let’s create our first Scrapy project. So in this case, as we're going to be scraping a website, we will call our project ecco_parser. 

scrapy startproject ecco_parser

Pause for a second here and understand how Scrapy structures its projects. Enter this command.

$ cd/ecco_parser
$ tree

You should be able to see something like this:

├── scrapy.cfg
└── ecco_parser
   ├── __init__.py
   ├── items.py
   ├── middlewares.py
   ├── pipelines.py
   ├── settings.py
   └── spiders
       └── __init__.py

This is the template project that Scrapy has automatically created for us when we ran the startproject command. Most of these files will be of no use for now, but we will see a quick overview of each:

• settings.py: It contains all the project settings.
• items.py: It is a model for the extracted data. We can also define our custom model, inheriting the Scrapy built-in Item class.
• pipelines.py: This part is responsible for cleaning and storage of the extracted data. All the items yielded by the spider will pass to it. 
• middlewares.py: It contains the logic of how the request is made and how Scrapy handles the response. The functioning can be modified according to our requirements.
• scrapy.cfg: It contains the deployment settings.

Now Creating Our Spider

Scrapy provides a number of different spider types. Some of the most commonly used types are listed below:

• Spider: This generic Scrapy version will take a list of start_urls and execute the overridden parse method for each URL.
• CrawlSpider: It will crawl the whole website and will parse every link it finds within the domain.
• SitemapSpider: Some sites do have their sitemap.xml file, which contains almost every needed URL. This spider is specially designed for parsing sitemaps.

We are going to use the Spider, the most popular one. 

Simply run the genspider command:

# syntax --> scrapy genspider <name_of_spider> <website_domain>
$ scrapy genspider ecco us.ecco.com

A new spider will have been created and added to the spiders folder:

import scrapy
class EccoSpider(scrapy.Spider):
   name = 'ecco'
   allowed_domains = ['us.ecco.com']
   start_urls = ['https://us.ecco.com/']
   def parse(self, response):
       pass

This spider class contains all these attributes (name, allowed_domains, start_urls) and a method named parse:

• Name: The spider name, which we have entered while running the above spider creation command.
• Allowed_domains: The domains of the spider. All the URLs outside these will be discarded by the spider
• Start_urls: All the initial URLs that will be scraped by Scrapy.
• Parse - a class method, which will be processed every time control is passed to Spider from Scrapy after receiving the response.

To start using this Spider, we need to update the parse method so that we can extract our products.

For instance, if we want to parse a product page, we can set the URL in the list of start_urls and then extract the desired data from the response by writing custom code in the parse method.

This is the complete view of the product in the browser.

You can see if we right-click on any color and open Inspect, we can easily find the class or id where these color details have been stored. You can clearly see that on the page, our color is selected, and respectively, in inspect mode, the selected color’s container is highlighted. We can see that in the selector “div.css-1ssc1jb“ all the color information has been stored. 

Now we want to extract the URL for all the product siblings (colors), and these URLs are stored in the href, which is located in the child anchor tag of this selected div. Therefore, our final selector would be as follows, which will extract all the color URLs.

“response.css('div.css-1ssc1jb a::attr(href)').getall()“ 

For extracting the title, the selector could be:

response.css('.css-16974uh [data-testid="product-card-titleandprice"] .chakra-heading .chakra-text::text').get().

It will extract the intended data, but we could extract the same data with the shorter selector.

response.css(' [data-testid="product-card-titleandprice"] .chakra-text::text').get().

Additionally, we don’t need to extract the whole data through selectors every time. Most of the time, there is a container JSON available in the page source, which we can view by right-clicking the site page and then selecting the “view page source” option.

You can analyze the page source and get an idea of which JSON you can get the maximum required data. Often, there are several JSONs available, but only one with the whole data, or we need to access 3 to 4 JSONs for extracting the entire data, or we need to do some kind of post-processing or mapping to get the required data from multiple JSONs. 

In our case, we can see that the whole data is stored in a container with the following ID, so we can extract the data using this JSON selector.:

“id="__NEXT_DATA__” 

The response is:

(response.css('#__NEXT_DATA__::text').get()) 

Another way to extract the data is:

response.css('script:contains(pageProps)::text').get().

Let’s discuss some of the hurdles we could face while scraping a site:
 

Scraping Hinders

Once you start scraping the source by hitting it again and again, the site could easily detect that you are not a user but a bot. Also, you need to climb all the protection barriers, whether it belongs to bypassing login walls or you have to prove being a human by pressing a button. 

You can’t shake hands with any platform if you can’t prove you are their honest customer, and not in favor of their competitors. If a platform has zero trust policies, you need to handle that as well. 

Every time you try accessing something that is restricted in your area then with dynamic bots you also have to deal with all authentication keys while staying in your area. Without moving a step, the bot has to pretend that it is now accessing the platform from a whitelisted region. 

Yes, many times you also need to develop algorithms to solve random puzzles on the site, but make sure that while trying to access new data, you will not lose your previously collected data, and store it in a reliable storage area. 

Final Thoughts

Web scraping is not just about extracting data—it’s about unlocking opportunities. By combining technical expertise with ethical practices, we can harness data to drive innovation while respecting digital boundaries. Whether you’re a developer, analyst, or entrepreneur, remember: the right data, collected the right way, can transform industries.