This course dives into the process, strategies, and best practices of web scraping. Learn how to use the Python framework, Scrapy, to practice key techniques.
Overview
Syllabus
Introduction
- How to learn to stop worrying and love the bot
- What you should know
- What is web scraping?
- How the internet works: A brief summary
- Hello world with Scrapy
- Challenge: Scraping all data on a page
- Solution: Scraping all data on a page
- Crawling a website
- Recording data
- Scrapy settings file
- Structuring your scrapers for extensibility/reusability
- Challenge: Scraping news sites
- Solution: Scraping news sites
- Submitting a form
- Finding and using hidden APIs
- Sitemaps and robots.txt
- Challenge: Using CNN's sitemap
- Solution: Using CNN's sitemap
- Logging in
- Browser automation with Selenium
- Interacting with a page
- Next steps
Taught by
Ryan Mitchell
Related Courses
-
Programming Applications with Python: Libraries and Web Scraping
-
Programming Applications with Python: Libraries and Web Scraping
-
Web Scraping with Python - Beautiful Soup Crash Course
4.5 -
Automate with Python – Full Course for Beginners
-
Programming Applications with Python
-
Python for Everybody - Full University Python Course
4.7
