Class Central is learner-supported. When you buy through links on our site, we may earn an affiliate commission.

YouTube

Downloading a Billion Files in Python

EuroPython Conference via YouTube

Overview

Limited-Time Offer: Up to 75% Off Coursera Plus!
7000+ certificate courses from Google, Microsoft, IBM, and many more.
This course teaches learners how to concurrently download a large number of files using Python. The learning outcomes include understanding the tradeoffs between multithreading, multiprocessing, and asyncio for downloading files, as well as gaining insights into design best practices, debugging, error handling, and performance optimization. The course covers topics such as network latency, file size, API usage, and various coding techniques for efficient file downloading. The intended audience for this course is Python developers interested in optimizing file download processes.

Syllabus

Introduction
The Task
Understanding the Task
Network Latency
File Size
The API
The Get API
Disclaimers
Synchronous
Multithreading
Coding
Main Loop
Performance
Why is this happening
Things to keep in mind
Multiprocessing
Multiprocessing code
Iterating over pages
Downloader
Speed Improvements
Async IO
List Call
Async IO Task
Different Libraries
UV Loop
Setup
IO HTTP
ItAll Files
Download Files
Summary
Multi Processing
Threading
Workflow
Interprocess communication overhead
Pagination token
Combo results
The real summary
Lessons learned
Thank you

Taught by

EuroPython Conference

Reviews

Start your review of Downloading a Billion Files in Python

Never Stop Learning.

Get personalized course recommendations, track subjects and courses with reminders, and more.

Someone learning on their laptop while sitting on the floor.