AWS Lambda Layers

Using Layers, we can add more code and content to our Lambda function. We can create a layer which can hold a lot of our dependencies as a single ZIP file and we can then attach it to our function at runtime.

Child Process NodeJS

Child process is a built-in NodeJS module which allows us to execute shell commands from NodeJS. Since FFmpeg is a command line tool, we can execute its commands via the exec or execFile functions off the child process module.

//using exec command
const util = require("util")
const exec = util.promisify(require("child_process").exec)
const execFile = util.promisify(require("child_process").execFile)(
  //assuming ffmpeg is available
  // stdout will contain the result if operation successfull
(async function() {
    const { stdout, stderr } = await exec("ffmpeg -version");
})()

/opt and /tmp

Once we attach a layer to our Lambda function, the contents of the uploaded ZIP folder are available via the /opt directory. So, to execute FFmpeg, we just use /opt/ffmpeg.

We don't get access to the underlying operating system of the Lambda, however, we do get access to the /tmp directory where we can store our files and read them later on.

General Strategy

(1) Give Lambda permissions to read and write to specific S3 bucket

(2) Read the video file from S3 via nodejs streams

(3) Copy the data into /tmp directory

(4) Use FFmpeg to do the manipulations on video and store the final result back in /tmp directory

(5) Finally, copy the result file from /tmp back to S3

Reading from S3

Using the aws-sdk for nodejs:

// getFromS3.js
const aws = require("aws-sdk")
const s3 = new aws.S3()
const fs = require("fs")

async function getFromS3(s3) {
  return await new Promise((resolve, reject) => {
    console.log("I am running")
    const writerStream = fs.createWriteStream("/tmp/sample.mp4")

    const params = { Bucket: "ffmpeg-test", Key: "sample.mp4" }
    const s3Stream = s3.getObject(params).createReadStream()

    s3Stream
      .pipe(writerStream)
      .on("error", function(err) {
        reject(err)
      })
      .on("close", function() {
        resolve("OK")
      })
  })
}

Executing FFmpeg

FFmpeg should only be issued inside the main function body rather than exporting it out to another file. Lambda has issues finding the ffmpeg reference if its execution command is put in a different file .

// inside main Lambda body
exports.handler = async event => {
  await getFromS3(s3)

  // manipulating sample.mp4 file in tmp directory
  // taking the output from ffmpeg and storing as result.mp4
  const { stdout, stderr } = await ls(
    "/opt/ffmpeg -i /tmp/sample.mp4 /tmp/result.mp4"
  )

  // await sendToS3(s3);
}

Sending Result Back To S3

// sendToS3.js
const fs = require("fs")
const stream = require("stream")

function uploadReadableStream(s3, stream) {
  const params = {
    Bucket: "nishant-ffmpeg-test",
    Key: "result.mp4",
    Body: stream,
  }
  return s3.putObject(params).promise()
}

async function sendToS3(s3) {
  const readerStream = fs.createReadStream("/tmp/result.mp4")
  const results = await uploadReadableStream(s3, readerStream)
  console.log("upload complete", results)
}

Final Considerations

(1) Lambda times out after 15minutes, so it would be difficult to transcode long videos. Typically shorter videos (1-10min) long should be ideal.

(2) FFmpeg manipulations are pretty memory intensive, so always keep an eye on the memory being used per operation and choose the necessary underlying hardware.

(3) Any Lambda can use up to 5 layers at a time

(4) The total unzipped size of the function and all layers can't exceed the unzipped deployment package size limit of 250 MB

FFmpeg Static Binary for Linux

AWS Lambda Layers Documentation

NodeJS Child Process Reference

AWS S3 Requests with NodeJS Streams