🐍 Lesson 27: Python Web Scraping with BeautifulSoup & Requests

Welcome to Lesson 27! Today we’ll learn how to scrape data from websites using Python. Web scraping is a powerful technique used in automation, research, data science, SEO, and even AI training. Whether you're interested in gathering market data, researching trends, or monitoring competitors, web scraping will help you get the information you need.

---

⭐ What You Will Learn in This Lesson

• How to install and use BeautifulSoup and Requests
• How to fetch and parse a webpage
• How to extract specific data, such as links, headings, and text
• The importance of respecting website scraping policies

---

👥 Who Is This Lesson For?

• Anyone interested in automating data collection
• Beginners who want to learn about web scraping and data extraction
• Python developers looking to gather data for machine learning or research
• Anyone interested in SEO and competitor monitoring

---

🌐 What Is Web Scraping?

Web scraping refers to the process of fetching a webpage and extracting specific information like:

• Headlines
• Prices
• Links
• Images
• Product details

---

📦 1. Installing Required Libraries

pip install requests
pip install beautifulsoup4

We will use the requests module to fetch the webpage and BeautifulSoup to parse the HTML content.

---

📦 2. Fetching a Webpage

import requests

url = "https://example.com"
response = requests.get(url)

print(response.text)  # HTML content

The requests.get() method fetches the HTML content of the given URL.

---

📦 3. Parsing HTML with BeautifulSoup

from bs4 import BeautifulSoup

soup = BeautifulSoup(response.text, "html.parser")

print(soup.title.text)

With BeautifulSoup, we can parse the HTML and easily navigate it to extract the data we need, such as the title of the page.

---

📦 4. Extracting All Links

links = soup.find_all("a")

for link in links:
    print(link.get("href"))

Use find_all() to retrieve all anchor (a) tags, which contain links to other pages.

---

📦 5. Extracting Specific Data

Example: Extract all headings from a webpage:

headings = soup.find_all("h2")

for h in headings:
    print(h.text)

Here, we're extracting all h2 headings from the page. You can apply the same method for other tags as well.

---

📦 6. Extracting Items by Class

product_titles = soup.find_all("div", class_="product-title")

for title in product_titles:
    print(title.text.strip())

You can also target specific elements using their class_ attribute, which allows you to extract data from specific sections of a page.

---

⚠ Important Note

Always check a website’s robots.txt and terms of service to ensure scraping is allowed. Web scraping should be ethical and legal, respecting website rules and data privacy regulations.

---

🧩 Why Web Scraping Matters

• Automate data collection for research, analysis, or reporting
• Build datasets for machine learning or AI training
• Monitor competitor prices and track market trends
• Gather SEO ranking data for optimization
• Extract valuable business insights from the web

---

🧪 Practice

1. Scrape the title of any public webpage.
2. Extract all the links from a news website.
3. Scrape all h1, h2, and h3 headings from a page.
4. Find all items belonging to a specific class (e.g., article-title) on a webpage.

---

❓ Common Mistakes

• Not respecting a website's robots.txt file
• Scraping too many requests too quickly, which can lead to IP blocking
• Not handling errors like network timeouts and missing elements

---

❓ Frequently Asked Questions (FAQ)

1. Is web scraping legal?

Web scraping is legal as long as it doesn’t violate a website’s terms of service or data privacy regulations. Always check the robots.txt file before scraping.

2. Can I scrape data from any website?

Not all websites allow scraping. You should always check the website’s robots.txt or terms of service to ensure you're allowed to scrape their data.

3. What if I scrape too quickly and get blocked?

Web scraping too quickly can result in your IP being blocked. Always use polite scraping techniques, such as adding delays between requests or rotating your IP addresses.

---

🚀 What’s Next?

In the next lesson, you’ll learn about:

• Working with APIs in Python
• Handling JSON data
• How to interact with online services and gather real-time data

---

➡ Next Lesson

Go to Lesson 28 →