Efficient Architecture 101: Is Your Upload Strategy Actually Efficient?

Before we start...

So, why this series? I've recently jumped straight into the complexities of system design, and I think the best approach to learning is by writing blogs and sharing what I have learned. My goal is to create simple blogs with humor and a lot of diagrams. Please note, I'm still learning, so if you're experienced and find something that can be improved or is incorrect, please share your thoughts.

The Challenge: A Real-World Bottleneck

Not long ago, I was digging through a codebase and found a classic file upload setup. The user would select a file in their browser, the browser would send it to our backend server, and then the server would upload it to a cloud storage bucket like AWS S3.

It worked, but it was far from efficient. The architecture looked something like this:

So, approach has some serious drawbacks

It's Slow for the User: The file has to make two trips over the internet. The user is stuck waiting for both to complete.

It Hogs Server Resources: Imagine 100 users uploading large videos at once. Your server is suddenly trying to process gigabytes of data, choking its memory, CPU, and bandwidth. This is a classic scalability nightmare.

It Can Be More Expensive: You're paying for the bandwidth twice – once for the data coming into your server, and again for it going out to the cloud storage. Ouch.

A Slightly Better Approach (That's Actually Worse)

So, to fix the speed issue, why not just upload directly from the browser? Modern tools like the AWS SDK for JavaScript let you do just that. The flow seems simpler and faster.

But this comes with a giant, flashing, red security warning. To allow the browser to upload, you have to embed your secret access keys directly in the frontend code. This is like leaving the master key to your entire house under the doormat.

Anyone with a little know-how can open their browser's developer tools, find your keys, and gain full, unrestricted access to your storage bucket. It's a disaster waiting to happen.

The Real Solution: Pre-Signed URLs

This is where we get the best of both worlds: the speed of a direct upload and the security of a server-managed process.

A pre-signed URL is a special, temporary, and secure link that your server generates. It gives a user permission to perform a specific action (like uploading a file with a specific name) directly to your cloud storage, but only for a limited time.

the browser first asks the server for permission. The server then generates a temporary, pre-signed URL and sends it back. The browser uses this special URL to upload the file directly to cloud storage. The server's only job is to authorize the request, not to handle the heavy file transfer itself.

So, Why prefer pre-singed url for uploads?

Enhanced Security: Your cloud storage credentials are never exposed on the client-side.

Improved Performance: Files are uploaded directly to cloud storage, reducing latency for the user.

Increased Scalability: Offloads the bandwidth and processing load from your server, allowing it to handle more application requests.

Cost Optimization: Reduces egress bandwidth costs from your server.

Best Practices

Use Short-Lived URLs: Always configure pre-signed URLs with the shortest possible expiration time (e.g., 5-15 minutes) to minimize the risk of misuse.

Utilize Multipart Uploads: For large files, leverage multipart uploads. This involves generating multiple pre-signed URLs to upload the file in chunks, which improves reliability.

Implement Resumable Uploads: In case of network interruptions, a robust implementation should allow the upload to resume from the last successfully uploaded part, rather than restarting the entire process.