Recently, we were working on a project where the backend was a Rails API and the front-end application was written in React. We had to implement a feature for a file upload. If this would have been a complete Rails application then we could upload the file using a file_field but implementing the Rails form was not possible in this case. We started exploring our options and came up with two possible solutions:
- Upload a file to Rails API backend and Rails backend uploads file to S3.
- Upload a file directly to S3 using AWS S3 Presigned URLs
Upload a file to Rails API backend and Rails backend uploads file to S3.
There are four major problems with this approach.
- File will be uploaded twice. First time from user's browser to the Rails backend and second time from Rails backend to S3.
- In the case of a huge file, this will increase the memory footprint of the Rails server which can hamper the performance of other APIs.
- This can cause timeouts in case of large files.
- This can possibly increase the data transmission cost depending on the cloud provider.
Upload a file directly to S3 using AWS S3 Presigned URLs
Let's briefly dicuss how the second approch would work.
- User's browser will request for the presigned URL.
- Application server will create a presigned URL using the AWS credentials stored on the application server.
- User's browser will upload the file to S3 using the presigned URL.
Presigned URLs in AWS S3
All objects and buckets in AWS S3 are private by default. The presigned URLs are useful if we want our user/customer to be able to upload a specific object to our bucket, but you don't require them to have AWS security credentials or permissions. If an AWS user has the necessary permission to access/modify an object, they can create a presigned URL that gives access to the S3 object without using the credentials. We should put in necessary precautions to protect the presigned URL since presigned URLs grant access to the S3 bucket. A pre-signed URL uses three parameters to limit access to the user:
- S3 Bucket - S3 Bucket where objects are stored
- Key - Name of the object
- Expiration - Time after which URL will be expired. We should keep this time short if we are using this for file upload using the API. The maximum lifespan of a presigned URL is 7 days.
There are following attributes of a presigned URL -
key: "attachment/<filename>"
x_amz_algorithm: "AWS4-HMAC-SHA256"
x_amz_credential: "AK***********5QF*******/<date>/<region>/s3/aws4_request"
x_amz_date: "<timestamp>"
x_amz_signature: "<signature>"
While creating a presigned URL user has to specify the intent. If a user is creating a presigned URL for GET(read) then URL can not be used for the PUT(write) request and vice-versa.
We need to specify correct CORS permissions for accessing the presigned URLs of S3. We can edit the CORS configuration by editing the permissions in the S3 bucket. CORS policy will be something like:
[
{
"AllowedHeaders": [
"*"
],
"AllowedMethods": [
"PUT",
"POST",
"DELETE",
"GET"
],
"AllowedOrigins": [
"www.example.com"
],
"ExposeHeaders": []
}
]
There are 2 steps to upload a file directly to S3 using presigned URLs.
- Create a presigned URL in Rails API
- Upload the file to S3 using the presigned URL
1. Create a presigned URL in Rails API
Add aws credentials in the Rails.application.credentials. Please refer custom credentials to know more about securing credentials in rails application.
We can use aws-sdk-s3 gem for accessing AWS S3 from our rails application. Let's setup a AWS credentials as given below.
# initilaizer/aws.rb
# Set AWS credentails on App level
Aws.config.update(
credentials: Aws::Credentials.new(
Rails.application.credentials.aws.fetch(:access_key_id),
Rails.application.credentials.aws.fetch(:secret_access_key)
),
region: Rails.application.credentials.aws.fetch(:region),
)
S3_BUCKET = Aws::S3::Resource.new.bucket(
Rails.application.credentials.aws.fetch(:bucket_name)
)
Create an action in the Rails application for generating the presigned URL which will be used by the React frontend to upload the file directly to the S3 bucket.
def presigned_url
return unless params[:file_name]
file_name = params[:file_name].gsub(/\s+/, '') #remove whitespace from the file name.
presigned_request = S3_BUCKET.presigned_post(
key: generate_key(file_name),
success_action_status: '201',
signature_expiration: (Time.now.utc + 1.minute) #Url will be expired after 1 minute
)
render json: {
url: presigned_request.url
s3_upload_params: {
key: presigned_request.fields["key"]
success_action_status: presigned_request.fields["success_action_status"]
x_amz_credential: presigned_request.fields["x-amz-credential"]
x_amz_algorithm: presigned_request.fields["x-amz-algorithm"]
x_amz_date: presigned_request.fields["x-amz-date"]
x_amz_signature: presigned_request.fields["x-amz-signature"]
}
}, status: :ok
end
def generate_key(file_name)
# Append a unique id to distinguish 2 files with same name.
# We can use timestamp as well in place of uuid.
random_file_key = SecureRandom.uuid
"#{Rails.env}/attachment/#{random_file_key}-#{file_name}"
end
This will return following response, which we'll use to upload file from the react app:
{
"url": "https://myproject.us-west.amazonaws.com"
"s3_upload_params": {
"key": "********/filename.jpg",
"success_action_status": "201",
"x_amz_credential": "************",
"x_amz_algorithm": "AWS4-HMAC-SHA256",
"x_amz_date": "**************",
"x_amz_signature": "************"
}
}
We can use Aws::S3::PresignedPost as an alternate API to generate the presigned URL as well. As we have done it below:
post = Aws::S3::PresignedPost.new(creds, region, bucket, {
key: <filename>,
content_length_range: 0..<size>,
acl: 'public-read',
metadata: {
'original-filename' => <filename>
}
})
post.fields
But we'd recommend using the S3_BUCKET.presigned_post approach for the Rails app since AWS is configured on the app and AWS resources are available throughout the application.
2. Upload the file to S3 using the presigned URL
Let's create a react component with the file input field.
import React, { createRef } from 'react';
const UploadFile = (props) => {
const fileUploadRef = createRef();
const inputProps = {
type: 'file',
disabled,
accept,
};
return (
<span className={classes} onClick={() => fileUploadRef.current.click() }>
<input {...inputProps} ref={fileUploadRef} />
</span>
);
}
We have disabled the input field and wrapped it inside a span. This way we can customize the style of the input field. We've used refs to refer to the input field inside the React Component. We have added the onClick property on the span to emulate the click on the span as the click on the input field. This will open the file selector dialog box. Once files are selected inside the dialog box for the upload, we should add one event handler for file change to upload the file.
Let's look at following event handler on the input field:
const handleFileChange = () => {
const selectedFile = fileUploadRef.current.files[0]);
console.log(selectedFile);
// Add logic/(react hook) for uploading the file.
};
return (
<input {...inputProps} ref={fileUploadRef} onChange={handleFileChange}/>
);
We can add our logic to uplaod a file in the same event handler or extract it into a react hook for the reusability. Let's extract this into a separate hook called useFileUpload.
Let's consider follwing code for the hook:
// useFileUpload.js
import axios from "axios";
const useFileUpload = () => {
const handleImageUpload = (file) => {
// send request to the rails API to get the presigned URL and its parameters
axios.get({
url: '/presigned-url',
params: {
// Rails API expects the file name to generate the S3 object key
file_name: file.name,
},
}).then((uploadParams) => {
// If the response is successful then pass these params to uplaod the file on S3.
if (uploadParams) {
// logic of uploading the file is implemented in the uploadToS3
// uploadToS3 expects the presigned url and file object to upload the file.
uploadToS3(uploadParams, file);
}
});
return { handleImageUpload };
};
export default useFileUpload;
Here we have a resuable react hook which accepts a file as input and upload the file on the S3 using the presigned URL. Now, our UploadFile component will look something like this.
// FileUpload.js
import React, { createRef } from 'react';
import useFileUpload from './hooks/useFileUpload';
const UploadFile = (props) => {
const { handleImageUpload } = useFileUpload();
const fileUploadRef = createRef();
const handleButtonClick = () => {
fileUploadRef.current.click();
};
const handleFileChange = () => {
handleImageUpload(fileUploadRef.current.files[0]);
};
const inputProps = {
type: 'file',
onChange: handleFileChange,
disabled,
accept,
};
return (
<span className={classes} onClick={handleButtonClick}>
<input {...inputProps} ref={fileUploadRef} />
</span>
);
};
export default UploadFile;
Now, let's see how uploadToS3 works.
There are 3 main parts of uploadToS3:
- Create form object for the POST request.
const { url, file_key: fileKey, s3_upload_params: fields } = uploadParams;
const formData = new FormData();
formData.append('Content-Type', 'image/jpg');
Object.entries(fields).forEach(([k, v]) => {
formData.append(k, v);
});
formData.append('file', file);
- Send the POST request to S3 at the presigned URL. We've extracted the
urlfrom theuploadParams.
axios.post({
url,
data: formData,
headers: { 'Content-Type': 'multipart/form-data' }});
- Parse the respone using xml2js and take necesary action according to the response status.
xml2js.parseString(awsResponse, (err, result) => {
if (err) {
console.log('send dummy pic until aws file upload is ready');
}
return result;
});
Finally, let's look at the overall implementation of the uploadToS3 for file upload to S3.
import xml2js from 'xml2js';
import axios from "axios";
const uploadToS3 = (uploadParams, file) => {
// extract the URL and file upload params for s3
const { url, file_key: fileKey, s3_upload_params: fields } = uploadParams;
// create new form object to uplaod the file
const formData = new FormData();
// Set content type of the file
// Please note: if content type and files content do not match file uplaod will fail.
formData.append('Content-Type', 'image/jpg');
// Append S3 upload params to the form object.
Object.entries(fields).forEach(([k, v]) => {
formData.append(k, v);
});
// Append the file
formData.append('file', file);
// send request to S3 for the file upload.
axios.post({
url,
data: formData,
headers: { 'Content-Type': 'multipart/form-data' }
}).then((awsResponse) => {
// S3 returns its response in XML(yeah I know :D)
xml2js.parseString(awsResponse, (err, result) => {
// parse it
if (err) {
// Notify the user that file upload has failed.
console.log('something went wrong, error:', err);
}
// if all goes well
return result;
});
});
};
Check out this full demo of uploading file directly to S3 using presigned URLs.
NPM Packages
- Axios: For handling HTTP requests from the React.
- node-xml2js: Simple XML to JavaScript object converter.
Ruby Gems:
You can connect with us on Twitter for your feedback or suggestions.