How to design Youtube

Gainlo link: http://blog.gainlo.co/index.php/2016/10/22/design-youtube-part/

Features:

1. Upload video - Y
2. Download video- N
3. Stream video - Y
4. Like/Dislike video - Y
5. Comment on video - Y
6. Subscribe to a user - N
7. Recommendation of videos - Y (Basic recommendation system)
8. Security - N
9. View Count of video - Y

Estimation:

Disk Storage Needed:

• Video Size

1. Average video size - 100MB
2. Number of users - 2 Billion
3. Average Uploads per day(Assuming 10% of the users upload every day) - 200 Million
4. Lets say the Max Upload limit for a video is - 1 GB. With these assumptions, the content size per day = (20010^6)(100*10^6) = 20 * 10^15 = 20 Peta Bytes Content size for 10 years = 20 PB * (365 * 10) = 73000 PB = 73 ExaByte

• Image Size:

• User Profile Size

100kb per user * 2 Billion users = (10010^3) * (210^9) = 200 * 10^12 = 200 TB.

• Number of videos streaming at the same time:

Lets say on an average 50% of the users are streaming at the same time.That is ~1Billion users are streaming at a given time.

Design Goals:

1. Latency - This is critical.
2. Consistency - If a few bytes are lost during streaming a video, this should be ok. As user can always go back and look at the video.
3. Availability - This is critical, being able to stream a video anytime is critical for a service like youtube.

High Level design:

• APIs:

1. UploadResult Upload(user_id,Content)
2. FetchResult Fetch(user_id,video_id)
3. LikeResult Like(user_id,video_id)
4. SubscribeResult Subscribe(user_id,to_user_id)

struct UploadResult{ bool success; int failureReason; int video_id; };

struct FetchResult{ bool success; int failureReason; Content data; RecommendedContent recContent; };

struct RecommendedContent{ vector data; int video_id; }

• Message Flow for these APIs.

1. From User to Application Server(Upload/Like/Subscribe) Client<-> LoadBalancer<->ApplicationServer<->Cache<->DB

2. From Application Server to User(Fetch) Client<-> LoadBalancer<->ApplicationServer<->Cache<->DB

Deep Dive: Fault tolerance and Load distribution:

Database:

1. Is Sharding Needed? Yes.
2. NoSQL vs SQL? Pick NoSQL as data is distributed across several clusters.
3. Highly Available Database Design. Consistency Hashing etc.
4. Data storage for Images/videos? Use CDN's to reduce latency and increase availability. Videos are replicated by CDN and it reduces the hops needed to fetch the video for the end user. Popular Vs long tailed videos(videos that have just <20 views a day). CDNs can be used to replicate and store popular videos. CDN for all the long tailed videos can get expensive. long tailed videos, can be hosted directly by youtube.

Recommendation system: