Web Scraping Using Python: A Step-by-Step Tutorial for Beginners

If you have ever copied data from a website row by row, you already know why web scraping using Python is so powerful. Instead of spending hours on manual copy-paste, you can write a short Python script that fetches a page, reads its HTML, and extracts exactly the information you need in seconds.

In this beginner-friendly tutorial, you will learn python web scraping with two essential libraries: Requests and BeautifulSoup. We will cover everything from installation to real code, from parsing HTML to saving data in a CSV file, and from handling pagination to scraping ethically. By the end, you will have a complete, runnable project you can adapt to almost any static website.

Whether you are exploring data science, building a price tracker, or automating research, this guide is the right place to start.

What is Web Scraping?

Web scraping is the automated process of extracting information from websites. A scraper program sends an HTTP request to a web page, downloads the HTML, and then parses that HTML to pull out structured data such as text, links, prices, images, or tables.

For example, instead of opening 50 product pages and noting prices by hand, a web scraper can visit all 50 pages, extract the price element, and save the results into a spreadsheet or a database.

Common real-world uses of web scraping include:

• Price comparison and monitoring
• Job listing aggregation
• News and sentiment analysis
• Real estate listing research
• Academic and market research datasets
• Lead generation and competitive analysis

Because Python has a clean syntax and rich ecosystem of libraries, it is one of the most popular languages for web scraping today.

Is Web Scraping Legal? (Ethics & robots.txt)

This is the most common question beginners ask: Is web scraping legal?

The short answer is: it depends on how and what you scrape.
Web scraping itself is not illegal, but it must be done responsibly. Before you scrape any website, follow these guidelines:

• Read the website’s Terms of Service — Some sites explicitly prohibit scraping.
• Check robots.txt — This file tells bots which pages they are allowed to visit. You can usually find it at https://example.com/robots.txt.
• Do not overload the server — Add delays between requests so you do not harm the website’s performance.
• Only scrape public, non-personal data — Avoid collecting private, copyrighted, or personally identifiable information without permission.
• Respect rate limits and CAPTCHAs — If a site tries to block you, stop and reconsider your approach.

The robots.txt standard is documented in RFC 9309, which is widely accepted as the reference for robot exclusion rules.

Ethical rule of thumb: scrape only what you need, request data slowly, and stop if the site owner asks you to.

Build an AI-First Career, Master the Complete Skillset

Choose from our industry-leading programs designed for career success

NSDC Certified

Modern Software and AI Engineering Program

Master full-stack development with AI integration

12 MonthsDuration

AI-LedCurriculum

Career SupportSupport

+1000 more

Go to Program

NSDC Certified

Modern Data Science and ML with specialisation in AI

Advanced data science techniques with AI specialization

12 MonthsDuration

AI-LedCurriculum

Career SupportSupport

+1000 more

Go to Program

NSDC Certified

Advanced AIML with Specialisation in Agentic AI

Deep dive into AIML with focus on Agentic systems

12 MonthsDuration

AI-LedCurriculum

Career SupportSupport

+1000 more

Go to Program

NSDC Certified

DevOps, Cloud & AI Platform Engineering

Build and manage AI-powered cloud infrastructure

12 MonthsDuration

AI-LedCurriculum

Career SupportSupport

+1000 more

Go to Program

NSDC Certified

AI Engineering Advanced Certification by IIT-Roorkee

Premier AI engineering certification from IIT-Roorkee

3 MonthsDuration

AI-LedCurriculum

Career SupportSupport

Go to Program

NSDC Certified

AI Forward Deployed Engineer Program

Full-stack engineering, production AI and client-facing consulting

12 MonthsDuration

AI-LedCurriculum

Career SupportSupport

+1000 more

Go to Program

Tools for Web Scraping in Python

Python offers several libraries for web scraping. The right tool depends on the type of website and the size of your project.

For this tutorial, we focus on Requests + BeautifulSoup because they are lightweight, beginner-friendly, and sufficient for most static websites.

If you are new to Python, you may want to explore the Python for Beginners course or Advanced Python topics to strengthen your fundamentals before moving to larger projects.

Step 1: Install Requests and BeautifulSoup

To start scraping, you need two third-party libraries: requests and beautifulsoup4. Open your terminal or command prompt and run:
pip install requests beautifulsoup4

You may also want pandas and lxml for storing data and faster parsing:
pip install pandas lxml

Once installed, import them in your Python script:
import requests
from bs4 import BeautifulSoup
import pandas as pd

That is all you need for a basic scraper.

Step 2: Send an HTTP Request

Every web page starts with an HTTP request. When you type a URL in your browser, the browser sends a GET request to a server and receives HTML in return. With Python, the requests library does the same thing.

A simple request looks like this:

import requests

url = "https://quotes.toscrape.com/"
response = requests.get(url)

print(response.status_code)
print(response.text[:500])

The status_code tells you whether the request succeeded. A code of 200 means OK; 404 means page not found; 403 or 429 often mean the site is blocking or rate-limiting you.

Many websites also check the User-Agent header to identify bots. To look more like a real browser, add a custom header:

headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
}

response = requests.get(url, headers=headers)
print(response.status_code)

If you want to understand how HTTP works under the hood, read our guide on the Hypertext Transfer Protocol.

Step 3: Parse HTML with BeautifulSoup

Once you have the HTML response, you need to parse it. BeautifulSoup converts messy HTML into a structured tree you can search and navigate.
from bs4 import BeautifulSoup

soup = BeautifulSoup(response.text, "html.parser")
print(soup.title.text)

The first argument is the raw HTML string, and the second is the parser. For most cases, "html.parser" works well. Alternatives include "lxml" (faster) and "html5lib" (more forgiving).

You can now inspect the page structure using BeautifulSoup methods or your browser’s developer tools. Right-click any element on a page and choose Inspect to see the HTML behind it.

For full method documentation, refer to the official BeautifulSoup documentation.

Step 4: Extract Data with find() and find_all()

BeautifulSoup provides two powerful methods for selecting elements:

• find() — returns the first matching element.
• find_all() — returns a list of all matching elements.

You can search by tag name, CSS class, ID, or attribute.

Hiring Partners:

GoogleAmazonMicrosoftFlipkartAdobe1200+ more

Example: Scrape quotes from quotes.toscrape.com

Open the site in your browser and inspect a quote. You will see HTML like this:

<div class="quote">
<span class="text">“The world as we have created it is a process of our thinking.”</span>
<small class="author">Albert Einstein</small>
<a class="tag">world</a>
</div>

Here is the Python code to extract all quotes, authors, and tags:

import requests
from bs4 import BeautifulSoup

url = "https://quotes.toscrape.com/"
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
}
response = requests.get(url, headers=headers)
soup = BeautifulSoup(response.text, "html.parser")

quotes = soup.find_all("div", class_="quote")

for quote in quotes:
text = quote.find("span", class_="text").text
author = quote.find("small", class_="author").text
tags = [tag.text for tag in quote.find_all("a", class_="tag")]
print(f"{text} — {author}")
print(f"Tags: {', '.join(tags)}\n")

Output:

“The world as we have created it is a process of our thinking.” — Albert Einstein
Tags: change, world

... (more quotes)

You can also extract attributes such as href from links:

link = soup.find("a", class_="next")
next_page = link["href"] if link else None
print(next_page)

For a more security-focused example that uses BeautifulSoup with forms, take a look at our article on attacking web forms with BeautifulSoup and Requests.

Step 5: Store the Data (CSV / pandas)

After extracting data, you usually want to save it. The two most common formats are CSV and pandas DataFrames.

Save to CSV using the csv module

import csv

with open("quotes.csv", "w", newline="", encoding="utf-8") as f:
writer = csv.writer(f)
writer.writerow(["Quote", "Author", "Tags"])
for quote in quotes:
text = quote.find("span", class_="text").text
author = quote.find("small", class_="author").text
tags = ", ".join([tag.text for tag in quote.find_all("a", class_="tag")])
writer.writerow([text, author, tags])

Save to a pandas DataFrame

import pandas as pd

data = []
for quote in quotes:
text = quote.find("span", class_="text").text
author = quote.find("small", class_="author").text
tags = ", ".join([tag.text for tag in quote.find_all("a", class_="tag")])
data.append({"Quote": text, "Author": author, "Tags": tags})

df = pd.DataFrame(data)
df.to_csv("quotes.csv", index=False)
print(df.head())

pandas is especially useful when you need to clean, filter, or analyze scraped data before saving it. If you want to build a career around this kind of data work, the Scaler Data Science Course covers Python, pandas, and end-to-end data pipelines in depth.

Handling Pagination and Dynamic Content

Most real websites do not show all their data on one page. You need to handle pagination by following "Next" links or modifying URL parameters.

Example: scrape multiple pages

import requests
from bs4 import BeautifulSoup
import pandas as pd

base_url = "https://quotes.toscrape.com"
page = "/page/1/"
all_quotes = []

while page:
response = requests.get(base_url + page, headers=headers)
soup = BeautifulSoup(response.text, "html.parser")
quotes = soup.find_all("div", class_="quote")

for quote in quotes:  
    text \= quote.find("span", class\_="text").text  
    author \= quote.find("small", class\_="author").text  
    tags \= ", ".join(\[tag.text for tag in quote.find\_all("a", class\_="tag")\])  
    all\_quotes.append({"Quote": text, "Author": author, "Tags": tags})

next\_link \= soup.find("li", class\_="next")  
page \= next\_link.find("a")\["href"\] if next\_link else None

df = pd.DataFrame(all_quotes)
df.to_csv("all_quotes.csv", index=False)
print(f"Scraped {len(df)} quotes.")

Always add a short delay between pages to avoid hammering the server:
import time
time.sleep(1) # wait 1 second between requests

JavaScript-rendered pages

Some websites load content with JavaScript after the initial page request. Requests only fetches the raw HTML, so it will not see dynamically generated data. In those cases, use Selenium or Playwright to control a real browser.

If you are interested in scraping with another language, you can also read about web scraping in R.

Web Scraping Best Practices

Writing a scraper is easy; writing a responsible scraper is what separates beginners from professionals. Follow these best practices:

• Respect robots.txt — Check it before you start scraping a site.
• Use a real User-Agent — Some sites block default requests headers.
• Add delays between requests — Use time.sleep() to reduce server load.
• Handle errors gracefully — Wrap requests in try/except blocks.
• Cache responses when possible — Avoid re-downloading the same page.
• Keep selectors simple and robust — Prefer stable class names over brittle XPath.
• Do not scrape personal or copyrighted data without permission.
• Log your activity — Track what you scraped and when.

Example error handling:

import time

for page_num in range(1, 6):
url = f"https://quotes.toscrape.com/page/{page\_num}/"
try:
response = requests.get(url, headers=headers, timeout=10)
response.raise_for_status()
soup = BeautifulSoup(response.text, "html.parser")
# ... extract data ...
except requests.exceptions.RequestException as e:
print(f"Failed to fetch {url}: {e}")
time.sleep(2)