Why files are uploaded using client side upload instead of server?

Exploring why client side upload is preferred over server side upload.

File upload is such a common feature in today’s applications that most of us never think about deeply how it might have been developed. For example you want to upload an image or a pdf file to chatgpt and then ask questions about it. How does the application does this feature?

When I joined my first company right after college we had an internal application which was used to upload images to be used in company’s blog. One of my task was to update a code pointer in that feature this is when my assumptions of how uploading works changed completely.

In this blog we will cover how the image upload is done usually and why was that way choosen by most.

So when you want to upload image there are mainly 2–3 ways to do it.

1. Storing the image as a Base64-encoded string in the database.

2. Storing the image in a blob storage service like Amazon S3 and saving a reference (URL) in the database.

While storing images as Base64-encoded strings might seem simple, it has significant drawbacks:

• Increased storage size: Base64 encoding increases the file size by approximately 33%, leading to inefficient storage usage.

• Database limitations: Some databases, such as DynamoDB, have a maximum item size limit (e.g., 512 KB), making it impractical to store large images.

Because of these reasons, most applications opt to store images in a blob storage service like Amazon S3 and keep only a reference (URL) in the database.

1. Upload the image to the backend and then backend uploads it to s3

In this approach we implement an endpoint in the backend and the image is sent as part of the request.

How It Works:

1. The client sends an image file to the backend as part of an HTTP request.

2. The backend receives the file and loads it into memory.

3. The backend establishes a connection with S3 and uploads the image.

4. Once the upload is complete, the backend responds to the client with the image URL.

While this method is straightforward, it has significant downsides in high-traffic scenarios:

1. Memory overhead: If each uploaded image is 5 MB and the server handles 1,000 requests per second, it could use up to 5 GB of memory just for storing images in RAM.

2. CPU overhead: The server performs additional processing to transfer the image to S3, adding unnecessary load.

3. Scalability issues: As traffic increases, the server may become a bottleneck, leading to performance degradation.

Let us explore the other option

2. Get a signed url from s3 and let the client upload.

In this approach instead of sending the image to the backend we communicate with the backend to by sending the some image information.

How It Works:

1. The client sends an API request to the backend with image metadata (e.g., file name, file size, and intended use case).

2. The backend communicates with the blob storage (e.g., S3) and generates a pre-signed upload URL.

3. The backend returns the signed URL to the client.

4. The client uses the signed URL to upload the image directly to S3.

For example, when uploading an image to ChatGPT, the request to the backend might look like this:

So when you upload an image it calls this endpoint

From dev tools of chatgpt

The first call is to get signed url, the second call they upload the image to blob storage.

We provide information like

{ "file_name": "name.png",
 "file_size": 42944,
 "use_case": "multimodal",
 "timezone_offset_min": 480,
 "reset_rate_limits": false }

The backend then interacts with the blob storage and it returns an upload url which is called signed url. This url is then used by the client to upload an image. This will then interact with the object storage using a signed url and upload the image.

A snnipet of python code

    def get_image_urls(self, image_imformation: Dict):
        file_name = image_imformation["file_name"]
        file_type = image_imformation["file_type"]
        
        # Generate presigned URL with proper parameters
        signed_url = self.s3_client.generate_presigned_url(
            'put_object',
            Params={
                'Bucket': bucket_name,
                'Key': self.get_file_path(file_name),
                'Expires': 3600,
                'ContentType': file_type
            },
            ExpiresIn=3600,
            HttpMethod='PUT'
        )
        
        return {
            "signed_url": signed_url,
            "file_name": file_name,
            "file_type": file_type
        }

Why This Approach Solves the Issues of the First Method

• No memory overhead: The backend never handles the image directly, avoiding memory consumption.

• Reduced CPU load: Since the backend doesn’t upload the image itself, it only processes a lightweight API request.

• Better scalability: The backend is no longer a bottleneck, allowing the system to handle more upload requests efficiently.

Conclusion

In applications there are so many small features when you go deep you understand the design choices made keeping scale in mind. I think this is one of them, Using a pre-signed URL approach offloads the actual file transfer to the client, reducing backend resource usage and improving scalability. Most people I know design the correct system but do not know why they did this, it is important to know why a feature is implemented in a certain way then the other.