How To Schedule a Job To Use an AWS Lambda Function To Update Content

This example demonstrates a few features that allow you to customize the behavior of your SCP system. Here we'll be leveraging Site Jobs, Integration with AWS Lambda, and Content Blocks to automatically update content from an external source on a regular schedule.

Our use case is that we wish to display a list of product recalls from the Consumer Product Safety Commission (CPSC), on a content page on the site, and we would like the list of recalls to be automatically updated each hour, using the CPSC's RSS feed.

We'll create Site Job that invokes an AWS Lambda Function each hour, sending it the URL for the RSS feed, and the content block's code and name as parameters. The Lambda Function will pull the XML from the RSS feed, and then turn around and post it to the system as a content block. We'll then write a client-side JS function to parse the content block and display a list of recalls on a content page.

Granted there may be other ways to accomplish this goal, but our approach will let us demonstrate a few of the system features in the same example.

Create the AWS Lambda FunctionCreate an API User in SCP that the Lambda Function Can UseProgram the AWS Lambda FunctionCreate a Site Function in SCP To Communicate With LambdaCreate a Site Job In SCP To Run the Function On a Schedule

Create the AWS Lambda Function

First we'll create an AWS Lambda Function to pull the RSS feed from the external site and post it to our SCP system as a content block. If you are unfamiliar with AWS Lambda, there are a numerous resources and examples out there to learn more about it. You can create the function within any AWS account - it does not need to be related to your SCP system. Within the function we'll add credentials allowing it to post the content to SCP. And within SCP, we'll create a site job that has credentials to call the Lambda Function.

Within your AWS account follow these steps to set up the Lambda Function and an IAM user that is allowed to invoke it.

Create a Lambda Function in the US-East-1 region
On the Lambda Function's details screen, Select "Copy ARN" for the next step.
Go to the IAM interface to create a new user with access to invoke the function:
- Create a new User
- Select no for console access
- For permissions, select "Attach policies directly"
- Choose "Create Policy"
  - For the policy, click to edit it in JSON
  - Paste in this JSON, replacing the resource ARN with the ARN of the new lambda function you copied above:
  { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "lambda:InvokeFunction" ], "Resource": [ "arn:aws:lambda:us-east-1:000000000000:function:xxxxxxxxxxxxxxxxxxxx" ] } ] }
  - Give the new policy a name like "my-lambda-functions"
  - Note later you can use this same user with other lambda functions by adding them to the permissions policy list of ARNs
- Back on the screen where you are creating the user, refresh the permissions policies, and select the new policy you just created
- Add an access key to the new user:
  - Now find the user under IAM -> Users, click into it and select "Security Credentials"
  - Under Access Keys select Create Access Keys. Select "Other" on the next screen
  - Create the access key and make a note of it and the secret key for the next step

Create an API User in SCP that the Lambda Function Can Use

In the administrator in SCP we need an API User that can create and update content blocks. The Lambda Function will use this account to post the RSS feed into SCP as a content block.

Under Operations -> API Users, click the plus icon to create a new user.
For status enter ACTIVE
We recommend adding a Note to explain the purpose of the user. Eg "Used by the UpdateFromRSSFeed Lambda Function."
Click Save Changes
Now that the user is created you need to add a Role to allow it to do things.
Go into the new user's details by clicking "Edit New Record"
From the user's three-dots menu, click Api Roles, and then click the plus icon to add a new role for the user
Select "Content Blocks Editor" for the role. This role allows the user to create and update content blocks, which is what we want.
Create the user's encoded basic authentication token
- SCP supports basic authentication for its API calls. To use basic authentication you simply need to Base-64 encode the user's user name and password in this form: username:password
- So for example, if your user's user name is "FHOWIUHQOWIBNF" and password is "EFBNGWEBUVWEINROWEINWIENGOFNOD", you would need to Base-64 encode this string: "FHOWIUHQOWIBNF:EFBNGWEBUVWEINROWEINWIENGOFNOD"
- You can use any number of tools to do the Base-64 encoding. For this example the result is this string: RkhPV0lVSFFPV0lCTkY6RUZCTkdXRUJVVldFSU5ST1dFSU5XSUVOR09GTk9E
- Save the encoded value for the next step
- NOTE: you could also do this encoding in the Lamdba Function itself, and paste the API user's user name and password into the function. We favor encoding it first, as it is (slightly) more obscure if someone should happen to see the function's code.

Program the AWS Lambda Function

Next add the following code to the AWS Lambda Function. This is the nuts and bolts of the task - as you can see it pulls XML from the feed, encodes it, and then posts it to SCP as a content block. You can paste this code directly into the AWS console under the function's Code tab, or follow their instructions on other ways to update the function's code such as uploading a ZIP file.


x
/**
 * Fetch the xml from a given url, escape it, and post it to an SCP url as a Content Block.
 */
const apiUrl = "https://YOUR-SCP-DOMAIN-NAME/api/contentBlock/createOrUpdate";
const basicAuthToken = "YOUR-API-USERs-ENCODED-BASIC-AUTH-TOKEN";

export const handler = async(event) => {
    let rssFeedURL = event.data.rssFeedURL.settingValue; // Site Job Setting
    let contentBlockCode = event.data.contentBlockCode.settingValue; // Site Job Setting
    let contentBlockName = event.data.contentBlockName.settingValue; // Site Job Setting
    console.info("rssFeedURL", rssFeedURL);
    console.info("contentBlockCode", contentBlockCode);
    console.info("contentBlockName", contentBlockName);
    let intFuncResponse = {};
    try {
        
        // Call the RSS feed
        intFuncResponse.rssUrl = rssFeedURL;
        const res = await fetch(rssFeedURL);
        console.info("Status from RSS feed fetch", res.status);
        let rssContent = await res.text();
        // console.info("Content from RSS feed", rssContent);
        intFuncResponse.statusFromRSSFeed = res.status;
        
        // Escape the XML here so ultimately it can be included in an attribute on a content page
        rssContent = rssContent.replace(/&/g, '&amp;')
               .replace(/</g, '&lt;')
               .replace(/>/g, '&gt;')
               .replace(/"/g, '&quot;')
               .replace(/'/g, '&apos;');
        
        let apiData = { "code": contentBlockCode, "name": contentBlockName, "content": rssContent };
        let apiString = JSON.stringify(apiData);
        
        const  apiCallHeaders = new Headers();
        apiCallHeaders.append("Content-Type", "application/json");
        apiCallHeaders.append("Authorization", "Basic " + basicAuthToken);

        const apiCallOptions = {
            method: "POST",
            headers: apiCallHeaders,
            body: apiString
        };
        
        intFuncResponse.apiUrl = apiUrl;
        const apiResponse = await fetch(apiUrl, apiCallOptions);
        console.info("Status from API call", apiResponse.status);
        intFuncResponse.statusFromAPICall = apiResponse.status;
        
        return intFuncResponse;
    }
    catch (e) {
        console.error(e);
        intFuncResponse.error = e;
        return intFuncResponse;
    }
};

Note the following about this Lambda Function:

For this example to work for you, replace "YOUR-SCP-DOMAIN-NAME" with the domain name you use for your SCP site, and replace "YOUR-API-USERs-ENCODED-BASIC-AUTH-TOKEN" with the basic auth token you created in the previous step.
In this example we are using Node.js 18, but you can use whatever runtime environment you like that lambda supports, including Java, Python, etc. SCP is agnostic and unaware of what goes on in your function.
Note that the rssFeedURL, contentBlockCode and contentBlockName are incoming parameters for the function. We will be sending them into the function from a Site Job (which we are about to set up, below). But you should be able to test the function in the AWS console by hard-coding those parameters. At this point it's not a bad idea to test it to make sure the content block is being created/updated.

Create a Site Function in SCP To Communicate With Lambda

Click the plus icon from the main Site Functions screen, which is found under the Operations -> Site Functions and Jobs.
For Function Event Type select JOB for this example. You may also associate the IF with an "entity event" to run whenever a given entity is created, updated or deleted
For Function Location select AWS_LAMBDA
For Function Endpoint, enter the ARN for the lambda function (click "Copy ARN" from the function details screen)
For Authentication Type select AWS_CREDENTIALS
For Key Value, enter the Access Key for the IAM user from the previous step
For Key Secret enter the "Secret access key" for the IAM user from the previous step
Here is a screenshot (without credentials):

Create a Site Job In SCP To Run the Function On a Schedule

Select the plus icon from the main Site Job screen, which is found under the Operations -> Site Functions and Jobs.
Give the job a name and Frequency.
For the Site Function, select the new Site Function you created in the previous step.

After creating the Site Job, create Site Job Settings under it to represent the parameters for the Lambda Function

The Lambda function in this example refers to three incoming parameters:


xxxxxxxxxx
    let rssFeedURL = event.data.rssFeedURL.settingValue; // Site Job Setting
    let contentBlockCode = event.data.contentBlockCode.settingValue; // Site Job Setting
    let contentBlockName = event.data.contentBlockName.settingValue; // Site Job Setting

These parameters can be set up as Site Job Settings. They will automatically be sent into the Lamdba function by the system.
From the Site Job's three-dots menu, select Site Job Settings and click the plus icon to add the first one:
Repeat for the other two parameters. Note that the code for the setting corresponds to the name of the parameter under the "event.data" object in the Lambda function.

After creating the job you can test it manually by selecting "Run Now..." from the Job's three-dots menu
After each run, the site job's "previousResults" setting gets updated with output from the Lamdba function. For additional troubleshooting, set up logging to CloudWatch in the Lambda function and use the "Monitoring" tab to find the latest logs for it.
Note that by creating a Report of site job settings, you can have the results of the site job runs emailed to you automatically. Refer to the Reports Feature document.