Apiconnect

Lambda Integration in IBM API Connect on AWS

This blog post will provide you an overview of our Lambda integration in API Connect on AWS, what you could use it for and a simple worked example of setting this up. If you’d prefer you can watch me demonstrate this in our video on YouTube.

Lambda is serverless computing platform provided by Amazon Web Services (AWS) which lets you build and deploy your code in a number of different programming languages with easy integrations to AWS services without having to manage infrastructure or servers. This makes it an ideal place to build out your API implementation and when you combine this with our API Connect on AWS SaaS offering you can quickly build, manage and socialise your APIs without worrying about the infrastructure behind them.

Using our Lambda policy in the SaaS service makes this simple and straight forward to integrate these so you can build out your apis around numerous different AWS services then manage them centrally through API Connect, sharing them with consumers through our customisable developer portal.

In order to manage the integration securely we use AWS Security Token Service (STS) so there is no need for you to give API Connect the credentials for your AWS account, you can just create an IAM role and grant permission for API Connect to assume that role.

The API Connect service is running in our AWS account and we have a fixed STS role that the service uses for each region:

Region Role
US East arn:aws:iam::623947394061:role/ibm-apiconnect-us-east-a
Frankfurt arn:aws:iam::623947394061:role/ibm-apiconnect-eu-central-a
London arn:aws:iam::623947394061:role/ibm-apiconnect-eu-west-a

In your AWS account you would set up a role that has permission to invoke your Lambda function(s) and set up the principal for the trust policy to be our service role and the external ID to be your provider org ID of your API Connect service instance. You can set this role to grant permission for just a single lambda function or multiple depending on what you need.

Within API Connect you will find the Lambda policy in the API Assembly editor on the Gateway tab along with all the other policies you can use to build out your API. The preferred model is to configure the lambda policy using the STS Assume Role option, however we do also support providing an Access Key ID and Secret Access Key. Then along with the credentials you just need to specify the name of the function and the region you have the lambda function deployed in.

To do this for yourself you can follow along with my simple worked example:

1. Create Lambda function

From the AWS Console, select Lambda then Create function - either start from scratch or make use of the ‘Hello World’ blueprint example. If you start from scratch and want to use the simple echo function I demonstrate in the video, give your function a name and select NodeJS as the Runtime - you can leave all the other settings the same unless your organisation has specific requirements around roles and permissions. Then click Create function.
Here is the code for the simple example from the video, which you can copy and paste into your function - alternatively you can use an existing function or write something more useful yourself:

export const handler = async(event) => {
    // TODO implement something more interesting here!  
    const response = event;
    return response;
};

After the code is updated in the function, click Deploy. Take a note of the function name and region your function is deployed in - both are part of the displayed function ARN - arn:aws:lambda:{{region}}:{{account}}:function:{{functionName}}

2. Create a role

Now you have the function created you will need to create a role to grant API Connect permission to assume which can invoke the function - for this, go to IAM and select Roles within the AWS Console.

  • Select Create role
  • Select Custom trust policy
  • Update the trust policy to have set the Principal to the API Connect role for the region you are using (see the table above) and set the “Condition” to require an external ID of the provider org you are using in the API Connect service- you should have something like the following:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Statement1",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::623947394061:role/ibm-apiconnect-eu-central-a"
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringEquals": {
                    "sts:ExternalId": "ac51909e-a379-4153-a0e8-ef4a25a83405"
                }
            }
        }
    ]
}

Next, create a permission policy for the role to allow it to invoke lambda functions - something like this (you can also specify the function within the Resource field to limit which lambda functions API Connect can invoke):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "lambda:InvokeFunction",
            "Resource": "*"
        }
    ]
}

Now you have the Lambda function and Role configured we’re ready to start making use of this function within API Connect

3. Create your API in API Connect

To do this, open API Manager - if you don’t already have a trial instance to use, you can create a free trial now.

  • Select Develop APIs and Products.
  • Create a new API using the Add button and selecting API (from REST, GraphQL or SOAP).
  • Select New OpenAPI.
  • Complete the details for the name and path and click Next.
  • Confirm the security options and click Next.

Now you have completed the guide to create the basic API you can set up the OpenAPI specification for it as you would like before moving on to configure the implementation. To configure the implementation, select the Gateway tab, remove the default Invoke policy by hovering over it until you see the bin icon - and then click it.

You can now drag the Lambda policy across in its place and fill in the details from your Lambda policy as the parameters (role, function name and region).

Now you have the Lambda policy in place you can Save the API, and toggle the Online indicator to make the API available to test and then try it out under the Test tab.

There we have an API implemented end to end from a serverless function in AWS Lambda through to defining the OpenAPI spec for it and managing it in API Connect. You could now go on to share this API with Consumers through the Developer Portal, or continue to build out the functionality with additional policies first.

Try it for yourself by signing up for our free 30 day trial today.

Invoke IBM Cloud Functions from API Connect

Cloud functions offer an easy way to build functionality and only pay for the time they are actually running without having to worry about how it will be hosted - you just right the code, save it and it is ready to be invoked. The function can be invoked through an event trigger or as a REST API - in this case I’m going to show how you can trigger a function as part of your API Connect assembly flow - providing an easy way to manage and socialise the functionality you have built.

To call an IBM Cloud function you first need to exchange your IBM Cloud API Key for an IAM access key using the IAM Authentication API.

This is a case of setting up and sending a post request to IBM Cloud IAM, to do this I used a set-variable policy to configure the headers and body followed by an invoke to https://iam.cloud.ibm.com/identity/token. The set-variable looks like this:

      - set-variable:
          version: 2.0.0
          title: setup iam request
          actions:
            - set: message.headers.content-type
              value: application/x-www-form-urlencoded
              type: string
            - set: message.headers.accept
              value: application/json
              type: string
            - set: message.body
              type: string
              value: >-
                grant_type=urn:ibm:params:oauth:grant-type:apikey&apikey=$(iam-apikey)​                

Once you have this access key you can use this to call your function using the endpoint provided in the UI:

{

For the authentication to the function URL we need to set the Authorization header to Bearer {access_key} where access_key is from the body of the previous request, so to obtain this we can use a Parse policy to parse the returned json, and then another set-variable policy to configure the headers for the Function invoke. This time the set-variable looks like this:

      - set-variable:
          version: 2.0.0
          title: setup function call
          actions:
            - set: message.headers.authorization
              value: Bearer $(iam.body.access_token)
              type: string
            - set: message.body
              value: $(request.body)
              type: any
            - set: message.headers.accept
              value: application/json
              type: string

As you can see we’re setting the authorization and accept headers, and passing the original request body from the inbound request. We follow this with an invoke of the function URL and a map, because the JSON returned from the function API includes details on invocation as well as the defined response:

{

The end to end flow looks like this:

{

Here is the complete sample swagger file which can be used in any API Connect deployment, including our new API Connect service on AWS

API Connect Reserved instance provides the ability to add remote API gateways so that you can co-locate the gateway service with your back-end systems for improved performance. With the new announcement of IBM Cloud Satellite, you can make use of this to securely expand your API Connect footprint across cloud providers and into the on-premise datacenter close to where your applications are running.

Create your Satellite location

To create a Satellite location you will need 3 hosts for the control plane, and at least one host to deploy DataPower on. For each of the hosts you will need to do the following:

  • Set up the host prior based on the host requirements for Satellite.

  • Register the host for RedHat updates using subscription-manager register - then apply a subscription to it in the RedHat Customer Portal

  • Refresh the packages and enable the repositories

subscription-manager refresh
subscription-manager repos --enable rhel-server-rhscl-7-rpms
subscription-manager repos --enable rhel-7-server-optional-rpms
subscription-manager repos --enable rhel-7-server-rh-common-rpms
subscription-manager repos --enable rhel-7-server-supplementary-rpms
subscription-manager repos --enable rhel-7-server-extras-rpms
  • Run the attach script obtained from the IBM Cloud Satellite UI to attach the host.

For the three hosts to form the control plane, you will need to assign them to the Satellite control plane through the UI or CLI.

Install DataPower in the Satellite location

  • Download the DataPower rpms from your reserved instance.
  • Install DataPower on your RHEL VM. In order to do this along with the pre-reqs I used the following commands:
yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
yum install schroot ipvsadm telnet
yum install idg_cloud1.10011.common.x86_64.rpm idg_cloud1.10011.image.x86_64.rpm
  • Create a link endpoint pointing to the API Connect Gateway management endpoint (usually port 3000)

Configure DataPower for API Connect

  • Follow the steps to configure your DataPower for use with API Connect.
  • For the certificates, in order to avoid a hostname mismatch create a self-signed keypair for the management interface for the hostname generated for your link endpoint.

Configure Certificate Manager service

Add Remote gateway to your reserved instance

  • Register your gateway in your API Connect Reserved Instance, filling in the details as follows:
    • Management endpoint - enter the full URL and port generated by IBM Cloud Satellite for the Link Endpoint prefixed with https://
    • Certificate - the certificate to present for the mutual TLS communication with the Gateway management
    • CA bundle - the certificate in certificate manager to use to verify the gateway management endpoint (either your CA or the certificate itself)
    • Base URL - the endpoint you want APIs to be called through - should map to the API gateway address configured in the API Connect Gateway service either directly or through a load balancer.

Currently the v10 Reserved Instance of API Connect doesn’t yet have a simple approach for headless use of the CLI toolkit. The following details how to use an IBM Cloud IAM bearer token with the API Connect CLI and REST API in a headless environment such as a CI/CD pipeline.

For interactive use of the API Connect CLI, you can login using the --sso option, retreive an api key with your browser and provide that to the CLI for example:

apic-slim login --server {apic-api-endpoint} --sso

However if you want to use the CLI in a non-interactive context such as a CI/CD pipeline you need to retrieve an IBM Cloud IAM Bearer token for the toolkit to use. This can be obtained using ibmcloud iam oauth-tokens and then placed in ~/.apiconnect/token for the apic CLI to use.

The token file needs to contain:

{apic-api-endpoint}/api: |
  refresh_token: ""
  access_token: {access_token}   

This can be done programmatically using something like this:

ibmcloud iam login --apikey {api-key}
ic iam oauth-tokens | sed 's/IAM token:  Bearer /api.9a6e-bd639816.us-south.apiconnect.cloud.ibm.com\/api: |\n  refresh_token: ""\n  access_token: /' > ~/.apiconnect/token

apic orgs --my --server {apic-api-endpoint}
production    [state: enabled]   https://api...apiconnect.cloud.ibm.com/api/orgs/9123ae60-427c-4997-8a6b-ddd75b169bfb
test-porg     [state: enabled]   https://api...apiconnect.cloud.ibm.com/api/orgs/b73708ea-a7b5-4d27-b562-80767e0b238e

Invoking the API Connect REST APIs

To invoke the API Connect REST APIs in Reserved Instance v10, you can use an IBM Cloud IAM token which can be obtained using the ibmcloud iam oauth-tokens CLI command or with an API call as detailed in the IAM API documentation. This token can then be used as a bearer token to invoke the API Connect REST APIs.

The full process looks like this:

Firstly use your IBM Cloud API key to retrieve an IAM token

curl -X POST \
  "https://iam.cloud.ibm.com/identity/token" \
  --header 'Content-Type: application/x-www-form-urlencoded' \
  --header 'Accept: application/json' \
  --data-urlencode "grant_type=urn:ibm:params:oauth:grant-type:apikey" \
  --data-urlencode "apikey=[your api key]"

{"access_token":"[access token would be here]","refresh_token":"not_supported","token_type":"Bearer","expires_in":3600,"expiration":1615370557,"scope":"ibm openid"}

Then take the value of the access_token and use it to call the API Connect API e.g.

curl https://{{ api_host }}/api/orgs \
  -H "Accept: application/json" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer [access_token]

{
  "total_results": 2,
  "results": [
    {
      "type": "org",
      "org_type": "provider",
    ...
    }
  ]
}

Even though a lot has changed within the API Connect product and the types and numbers of stacks we’re running since I first posted an overview of monitoring API Connect , the main areas we monitor haven’t.

We are still using Grafana as a central location for dashboarding and analysing data across different data sources but some of the tools we’re using to collect the data have changed. Having access to all the data in a single UI is really powerful, especially when troubleshooting or investigating events across the systems, being able to identify correlations between data from external load balancing, response times parsed from logs and pod utilisation metrics can really help narrrow in on specific components and how they impact the wider solution.

Metrics

Metrics flow

For metrics we’re making use of the IBM Cloud Monitoring with Sysdig to gather metrics from across the kubernetes deployment, including metrics from kubernetes itself and recognisable container applications such as nginx. We also supplement this with our own custom metrics exporter, Trawler, which we built for API Connect to extract key application specific data and expose them to a prometheus compatible monitoring tool or send them to graphite. Examples of data gathererd are counts of objects within API Manager and DataPower and analytics call counts. For endpoint and availability monitoring we are continuing to use Hem which is a simple python application to call HTTP(s) endpoints and send the metrics to our graphtie stack. All of these then come together to view within our grafana dashboards - and to be used within new exploratory dashboards whilst problem solving as needed.

Logging

Logging flow

For our logging infrastructure, we continue to use Elastic, making use of the filebeat agent within our clusters to gather and tag the container logs, then some custom parsing in logstash to parse out the significant elements from the different logs so that we can easily correlate these with events going on in the system. A lot of the time this data is then viewed in timeseries graphs within grafana, but also linked to Kibana views to dig deeper in the logs themselves.