Get website metadata from a url with an AWS Lambda function


This is a super simple way to scrape website metadata from a url similar to how Discord and Twitter work when you paste a url.


1. Create a basic lambda function

Choose author from scratch, give your function a name, select node 18.x as your runtime and then create your function.

Next we will create a zip file containing the necessary packages that our function will need. Somewhere on your machine create an empty folder and here we will install just one package: meta-data scraper



After the package is installed, zip the entire folder we just created.


Back in the aws console, select “upload from .zip file” from the “upload from” dropdown on the Code tab. Select the zip we just created. Your function file structure should look like this (minus the index.cjs, we will create that next):



Alt text



2. Add our function code

Create an index.cjs file. Add the following code: 
const getMetaData = require('metadata-scraper')

exports.handler = async function (event) {
  let url = ""

  if (event.body) {
    let body = JSON.parse(event.body);
    url = body.url
  };
  
  try {
    const data = await getMetaData(url)
    return {
      data:data,
      status:200
    }
  } catch (error) {
    return {
      data:null,
      status: 204
    }
  }
}

Click “Deploy” to deploy our new lambda function.


3. Expose our lambda url

Under configuration select url and add a new url. For testing we can allow public access. Add POST to our CORS policy. Copy our new function url.


4. Call our function from our client

This part is up to you, in react this is how I am calling the lambda function: 

const getMetadata = async () => {  
    const functionEndpoint = <our-function-url>;
    const testingEndpoint = <any-website-with-metadata>;

    let options = {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
      },
      body: JSON.stringify({
        url: testingEndpoint
      })
    };

    try {
      let res = await fetch(functionEndpoint, options);
      let response = await res.json();
      console.log(response.data);
    } catch (error) {
      console.log(error)
    }
}



The returned response should contain an object with the metadata