Tag Archives: amazon

Microservice using AWS Serverless Application Model and Couchbase

January 5, 2017couchbase, techtipamazon, aws, couchbase, lambda, rest, serverlessarungupta

Amazon Web Services introduced Serverless Application Model, or SAM, a couple of months ago. It defines simplified syntax for expressing serverless resources. SAM extends AWS CloudFormation to add support for API Gateway, AWS Lambda and Amazon DynamoDB. This blog will show how to create a simple microservice using SAM. Of course, we’ll use Couchbase instead of DynamoDB!

This blog will also use the basic concepts explained in Microservice using AWS API Gateway, AWS Lambda and Couchbase. SAM will show the ease with which the entire stack for microservice can be deployed and managed.

As a refresher, here are key components in the architecture:

Client could be curl, AWS CLI/Console, Postman client or any other tool/API that can invoke a REST endpoint.
AWS API Gateway is used to provision APIs. The top level resource is available at path /books. HTTP GET and POST methods are published for the resource.
Each API triggers a Lambda function. Two Lambda functions are created, book-list function for listing all the books available and book-create function to create a new book.
Couchbase is used as a persistence store in EC2. All the JSON documents are stored and retrieved from this database.

Other blogs on serverless:

Microservice using AWS API Gateway, AWS Lambda and Couchbase
AWS IoT Button, Lambda and Couchbase
Serverless FaaS with Lambda and Java

Let’s get started!

Serverless Application Model (SAM) Template

An AWS CloudFormation template with serverless resources conforming to the AWS SAM model is referred to as a SAM file or template. It is deployed as a CloudFormation stack.

Let’s take a look at our SAM template:

This template is available at github.com/arun-gupta/serverless/blob/master/aws/microservice/template.yml.

AWSTemplateFormatVersion : '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Microservice using API Gateway, Lambda and Couchbase
Resources:
  MicroserviceGetAllGateway:
    Type: AWS::Serverless::Function
    Properties:
      Handler: org.sample.serverless.aws.couchbase.gateway.BucketGetAll
      Runtime: java8
      CodeUri: s3://serverless-microservice/microservice-http-endpoint-1.0-SNAPSHOT.jar
      Timeout: 30
      MemorySize: 1024
      Environment:
        Variables:
          COUCHBASE_HOST: ec2-35-163-21-104.us-west-2.compute.amazonaws.com
      Role: arn:aws:iam::598307997273:role/microserviceRole
      Events:
        GetResource:
          Type: Api
          Properties:
            Path: /books
            Method: get
  MicroservicePostGateway:
    Type: AWS::Serverless::Function
    Properties:
      Handler: org.sample.serverless.aws.couchbase.gateway.BucketPost
      Runtime: java8
      CodeUri: s3://serverless-microservice/microservice-http-endpoint-1.0-SNAPSHOT.jar
      Timeout: 30
      MemorySize: 1024
      Environment:
        Variables:
          COUCHBASE_HOST: ec2-35-163-21-104.us-west-2.compute.amazonaws.com
      Role: arn:aws:iam::598307997273:role/microserviceRole
      Events:
        GetResource:
          Type: Api
          Properties:
            Path: /books
            Method: post

AWSTemplateFormatVersion : '2010-09-09'

Transform: AWS::Serverless-2016-10-31

Description: Microservice using API Gateway, Lambda and Couchbase

Resources:

MicroserviceGetAllGateway:

Type: AWS::Serverless::Function

Properties:

Handler: org.sample.serverless.aws.couchbase.gateway.BucketGetAll

Runtime: java8

CodeUri: s3://serverless-microservice/microservice-http-endpoint-1.0-SNAPSHOT.jar

Timeout: 30

MemorySize: 1024

Environment:

Variables:

COUCHBASE_HOST: ec2-35-163-21-104.us-west-2.compute.amazonaws.com

Role: arn:aws:iam::598307997273:role/microserviceRole

Events:

GetResource:

Type: Api

Properties:

Path: /books

Method: get

MicroservicePostGateway:

Type: AWS::Serverless::Function

Properties:

Handler: org.sample.serverless.aws.couchbase.gateway.BucketPost

Runtime: java8

CodeUri: s3://serverless-microservice/microservice-http-endpoint-1.0-SNAPSHOT.jar

Timeout: 30

MemorySize: 1024

Environment:

Variables:

COUCHBASE_HOST: ec2-35-163-21-104.us-west-2.compute.amazonaws.com

Role: arn:aws:iam::598307997273:role/microserviceRole

Events:

GetResource:

Type: Api

Properties:

Path: /books

Method: post

SAM template Specification provide complete details about contents in the template. The key parts of the template are:

Defines two resources, both of Lambda Function type identified by AWS::Serverless::Function attribute. Name of the Lambda function is defined by Resources.<resource>.
Class for each handler is defined by the value of Resources.<resource>.Properties.Handler attribute
Java 8 runtime is used to run the Function defined by Resources.<resource>.Properties.Runtime attribute
Code for the class is uploaded to an S3 bucket, in our case to s3://serverless-microservice/microservice-http-endpoint-1.0-SNAPSHOT.jar
Resources.<resource>.Properties.Environment.Variables.COUCHBASE_HOST attribute value defines the host where Couchbase is running. This can be easily deployed on EC2 as explained at Setup Couchbase.
Each Lambda function is triggered by an API. It is deployed using AWS API Gateway. The path is defined by Events.GetResource.Properties.Path. HTTP method is defined using Events.GetResource.Properties.Method attribute.

Java Application

The Java application that contains the Lambda functions is at github.com/arun-gupta/serverless/tree/master/aws/microservice/microservice-http-endpoint.

Lambda function that is triggered by HTTP GET method is shown:

public class BucketGetAll implements RequestHandler<GatewayRequest, GatewayResponse> {

    @Override
    public GatewayResponse handleRequest(GatewayRequest request, Context context) {
        try {
            N1qlQuery query = N1qlQuery
                    .simple(select("*")
                            .from(i(CouchbaseUtil.getBucketName()))
                            .limit(10));

            String result = CouchbaseUtil.getBucket().query(query).allRows().toString();

            return new GatewayResponse(200, result, GatewayResponse.HEADERS_JSON);
        } catch (ConfigurationException e) {
            return new GatewayResponse(400, e.getMessage(), GatewayResponse.HEADERS_TEXT);
        }
    }
}

public class BucketGetAll implements RequestHandler<GatewayRequest, GatewayResponse> {

@Override

public GatewayResponse handleRequest(GatewayRequest request, Context context) {

try {

N1qlQuery query = N1qlQuery

.simple(select("*")

.from(i(CouchbaseUtil.getBucketName()))

.limit(10));

String result = CouchbaseUtil.getBucket().query(query).allRows().toString();

return new GatewayResponse(200, result, GatewayResponse.HEADERS_JSON);

} catch (ConfigurationException e) {

return new GatewayResponse(400, e.getMessage(), GatewayResponse.HEADERS_TEXT);

}

A little bit of explanation:

Each Lambda function needs to implement the interface com.amazonaws.services.lambda.runtime.RequestHandler.
API Gateway and Lambda integration require a specific input format and output format. These formats are defined as GatewayRequest and GatewayResponse classes.
Function logic uses Couchbase Java SDK to query the Couchbase database. N1QL query is used to query the database. The results and exception are then wrapped in GatewayRequest and GatewayResponse.

Lambda function triggered by HTTP POST method is pretty straightforward as well:

public class BucketPost implements RequestHandler<GatewayRequest, GatewayResponse> {

    @Override
    public GatewayResponse handleRequest(GatewayRequest request, Context context) {

        try {
            JsonDocument document = CouchbaseUtil.getBucket().upsert(Book.fromStringToJson(request.getBody()));
            return new GatewayResponse(200, document.content().toString(), GatewayResponse.HEADERS_JSON);
        } catch (Exception ex) {
            return new GatewayResponse(400, ex.getMessage(), GatewayResponse.HEADERS_TEXT);
        }
    }
}

public class BucketPost implements RequestHandler<GatewayRequest, GatewayResponse> {

@Override

public GatewayResponse handleRequest(GatewayRequest request, Context context) {

try {

JsonDocument document = CouchbaseUtil.getBucket().upsert(Book.fromStringToJson(request.getBody()));

return new GatewayResponse(200, document.content().toString(), GatewayResponse.HEADERS_JSON);

} catch (Exception ex) {

return new GatewayResponse(400, ex.getMessage(), GatewayResponse.HEADERS_TEXT);

}

A bit of explanation:

Incoming request payload is retrieved from GatewayRequest
Document inserted in Couchbase is returned as response.
Like the previous method, Function logic uses Couchbase Java SDK to query the Couchbase database. The results and exception are then wrapped in GatewayRequest and GatewayResponse.

Build the Java application as:

mvn -f microservice-http-endpoint/pom.xml clean package

mvn -f microservice-http-endpoint/pom.xml clean package

Upload Lambda Function to S3

SAM template reads the code from an S3 bucket. Let’s create a S3 bucket:

aws s3 mb s3://serverless-microservice --region us-west-2

aws s3 mb s3://serverless-microservice --region us-west-2

us-west-2 region is one of the supported regions for API Gateway. S3 bucket names are globally unique but their location is region specific.

Upload the code to S3 bucket:

aws s3 cp microservice-http-endpoint/target/microservice-http-endpoint-1.0-SNAPSHOT.jar s3://serverless-microservice/microservice-http-endpoint-1.0-SNAPSHOT.jar

aws s3 cp microservice-http-endpoint/target/microservice-http-endpoint-1.0-SNAPSHOT.jar s3://serverless-microservice/microservice-http-endpoint-1.0-SNAPSHOT.jar

The code is now uploaded to S3 bucket. SAM template is ready to be deployed!

Deploy SAM Template

Deploy the SAM template:

aws cloudformation deploy \
--template-file template.yml \
--stack-name microservice-gateway \
--region us-west-2

aws cloudformation deploy \

--template-file template.yml \

--stack-name microservice-gateway \

--region us-west-2

It shows the output:

Waiting for changeset to be created..
Waiting for stack create/update to complete
Successfully created/updated stack - microservice-gateway

Waiting for changeset to be created..

Waiting for stack create/update to complete

Successfully created/updated stack - microservice-gateway

This one command deploys Lambda functions and REST Resource/APIs that trigger these Lambda functions.

Invoke the Microservice

API Gateway publishes a REST API that can be invoked by curl, wget, AWS CLI/Console, Postman or any other app that can call a REST API. This blog will use AWS Console to show the interaction.

API Gateway home at us-west-2.console.aws.amazon.com/apigateway/home?region=us-west-2#/apis shows:

AWS SAM Microservice API

Click on the API to see all the APIs in this resource:

AWS SAM Microservice API Resources

Click on POST to see the default page for POST method execution:

AWS SAM Microservice API POST

Click on Test to test the API:

Add the payload in Request Body and click on Test to invoke the API. The results are shown as below:

Now click on GET to see the default execution page:

AWS SAM Microservice API GET

Click on Test to test the API:

AWS SAM Microservice API GET Input

No request body is needed, just click on Test the invoke the API. The results are as shown:

Output from the Couchbase database is shown in the Response Body.

References

Deploying Lambda-based Applications
Serverless Architectures
AWS API Gateway
Creating a simple Microservice using Lambda and API Gateway
Couchbase Server Docs
Couchbase Forums
Follow us at @couchbasedev

Source: blog.couchbase.com/2017/january/microservice-aws-serverless-application-model-couchbase

Microservice using AWS API Gateway, AWS Lambda and Couchbase

December 31, 2016couchbaseamazon, apigateway, aws, couchbase, ec2, lambda, serverlessarungupta

This blog has explained the following concepts for serverless applications so far:

Serverless FaaS with AWS Lambda and Java
AWS IoT Button, Lambda and Couchbase

The third blog in serverless series will explain how to create a simple microservice using Amazon API Gateway, AWS Lambda and Couchbase.

Read previous blogs for more context on AWS Lambda.

Amazon API Gateway is a fully managed service that makes it easy for developers to create, publish, maintain, monitor, and secure APIs at any scale. Amazon API Gateway handles all the tasks involved in accepting and processing up to hundreds of thousands of concurrent API calls, including traffic management, authorization and access control, monitoring, and API version management.

Here are the key components in this architecture:

Client could be curl, AWS CLI, Postman client or any other tool/API that can invoke a REST endpoint.
API Gateway is used to provision APIs. The top level resource is available at path /books. HTTP GET and POST methods are published for the resource.
Each API triggers a Lambda function. Two Lambda functions are created, book-list function for listing all the books available and book-create function to create a new book.
Couchbase is used as a persistence store in EC2. All the JSON documents are stored and retrieved from this database.

Let’s get started!

Create IAM Role

IAM roles will have policies and trust relationships that will allow this role to be used in API Gateway and execute Lambda function.

Let’s create a new IAM role:

aws iam create-role \
--role-name microserviceRole \
--assume-role-policy-document file://./trust.json

aws iam create-role \

--role-name microserviceRole \

--assume-role-policy-document file://./trust.json

--assume-role-policy-document defines the trust relationship policy document that grants an entity permission to assume the role. trust.json is at github.com/arun-gupta/serverless/blob/master/aws/microservice/trust.json and looks like:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": [
          "lambda.amazonaws.com",
          "apigateway.amazonaws.com"
        ]
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

{

"Version": "2012-10-17",

"Statement": [

{

"Sid": "",

"Effect": "Allow",

"Principal": {

"Service": [

"lambda.amazonaws.com",

"apigateway.amazonaws.com"

]

"Action": "sts:AssumeRole"

}

]

}

This trust relationship allows Lambda functions and API Gateway to assume this role during execution.

Associate policies with this role as:

aws iam put-role-policy \
--role-name microserviceRole \
--policy-name microPolicy \
--policy-document file://./policy.json

aws iam put-role-policy \

--role-name microserviceRole \

--policy-name microPolicy \

--policy-document file://./policy.json

policy.json is at github.com/arun-gupta/serverless/blob/master/aws/microservice/policy.json and looks like:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:*"
      ],
      "Resource": "arn:aws:logs:*:*:*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "apigateway:*"
      ],
      "Resource": "arn:aws:apigateway:*::/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "execute-api:Invoke"
      ],
      "Resource": "arn:aws:execute-api:*:*:*"
    },
    {
      "Effect": "Allow",
      "Action": [
          "lambda:*"
      ],
      "Resource": "*"
    }
  ]
}

{

"Version": "2012-10-17",

"Statement": [

{

"Effect": "Allow",

"Action": [

"logs:*"

"Resource": "arn:aws:logs:*:*:*"

{

"Effect": "Allow",

"Action": [

"apigateway:*"

"Resource": "arn:aws:apigateway:*::/*"

{

"Effect": "Allow",

"Action": [

"execute-api:Invoke"

"Resource": "arn:aws:execute-api:*:*:*"

{

"Effect": "Allow",

"Action": [

"lambda:*"

"Resource": "*"

}

]

}

This generous policy allows any permissions over logs generated in CloudWatch for all resources. In addition it allows all Lambda and API Gateway permissions to all resources. In general, only required policy would be given to specific resources.

Create Lambda Functions

Detailed steps to create Lambda functions are explained in Serverless FaaS with AWS Lambda and Java. Let’s create the two Lambda functions as required in our case:

aws lambda create-function \
--function-name MicroserviceGetAll \
--role arn:aws:iam::598307997273:role/microserviceRole \
--handler org.sample.serverless.aws.couchbase.BucketGetAll \
--zip-file fileb:///Users/arungupta/workspaces/serverless/aws/microservice/microservice-http-endpoint/target/microservice-http-endpoint-1.0-SNAPSHOT.jar \
--description "Microservice HTTP Endpoint - Get All" \
--runtime java8 \
--region us-west-1 \
--timeout 30 \
--memory-size 1024 \
--environment Variables={COUCHBASE_HOST=ec2-52-53-193-176.us-west-1.compute.amazonaws.com} \
--publish

aws lambda create-function \

--function-name MicroserviceGetAll \

--role arn:aws:iam::598307997273:role/microserviceRole \

--handler org.sample.serverless.aws.couchbase.BucketGetAll \

--zip-file fileb:///Users/arungupta/workspaces/serverless/aws/microservice/microservice-http-endpoint/target/microservice-http-endpoint-1.0-SNAPSHOT.jar \

--description "Microservice HTTP Endpoint - Get All" \

--runtime java8 \

--region us-west-1 \

--timeout 30 \

--memory-size 1024 \

--environment Variables={COUCHBASE_HOST=ec2-52-53-193-176.us-west-1.compute.amazonaws.com} \

--publish

Couple of key items to note in this function are:

IAM role microserviceRole created in previous step is explicitly specified here
Handler is org.sample.serverless.aws.couchbase.BucketGetAll class. This class queries the Couchbase database defined using the COUCHBASE_HOST environment variable.

Create the second Lambda function:

aws lambda create-function \
--function-name MicroservicePost \
--role arn:aws:iam::598307997273:role/microserviceRole \
--handler org.sample.serverless.aws.couchbase.BucketPost \
--zip-file fileb:///Users/arungupta/workspaces/serverless/aws/microservice/microservice-http-endpoint/target/microservice-http-endpoint-1.0-SNAPSHOT.jar \
--description "Microservice HTTP Endpoint - Post" \
--runtime java8 \
--region us-west-1 \
--timeout 30 \
--memory-size 1024 \
--environment Variables={COUCHBASE_HOST=ec2-52-53-193-176.us-west-1.compute.amazonaws.com} \
--publish

aws lambda create-function \

--function-name MicroservicePost \

--role arn:aws:iam::598307997273:role/microserviceRole \

--handler org.sample.serverless.aws.couchbase.BucketPost \

--zip-file fileb:///Users/arungupta/workspaces/serverless/aws/microservice/microservice-http-endpoint/target/microservice-http-endpoint-1.0-SNAPSHOT.jar \

--description "Microservice HTTP Endpoint - Post" \

--runtime java8 \

--region us-west-1 \

--timeout 30 \

--memory-size 1024 \

--environment Variables={COUCHBASE_HOST=ec2-52-53-193-176.us-west-1.compute.amazonaws.com} \

--publish

The handler for this function is org.sample.serverless.aws.couchbase.BucketPost class. This class creates a new JSON document in the Couchbase database identified by COUCHBASE_HOST environment variable.

The complete source code for these classes is at github.com/arun-gupta/serverless/tree/master/aws/microservice/microservice-http-endpoint.

API Gateway Resource

Create an API using Amazon API Gateway and Test It and Build an API to Expose a Lambda Function provide detailed steps and explanation on how to use API Gateway and Lambda Functions to build powerful backend systems. This blog will do a quick run down of the steps in case you want to cut the chase.

Let’s create API Gateway resources.

The first step is to create an API:

aws apigateway \
create-rest-api \
--name Book

aws apigateway \

create-rest-api \

--name Book

This shows the output as:

{
    "name": "Book", 
    "id": "lb2qgujjif", 
    "createdDate": 1482998945
}

{

"name": "Book",

"id": "lb2qgujjif",

"createdDate": 1482998945

}

The value of id attribute is API ID. In our case, this is lb2qgujjif.

Find ROOT ID of the created API as this is required for the next AWS CLI invocation:

aws apigateway get-resources --rest-api-id lb2qgujjif

aws apigateway get-resources --rest-api-id lb2qgujjif

This shows the output:

{
    "items": [
        {
            "path": "/", 
            "id": "hgxogdkheg"
        }
    ]
}

{

"items": [

{

"path": "/",

"id": "hgxogdkheg"

}

]

}

Value of id attribute is ROOT ID. This is also the PARENT ID for the top level resource.

Create a resource

aws apigateway create-resource \
--rest-api-id lb2qgujjif \
--parent-id hgxogdkheg \
--path-part books

aws apigateway create-resource \

--rest-api-id lb2qgujjif \

--parent-id hgxogdkheg \

--path-part books

This shows the output:

{
    "path": "/books", 
    "pathPart": "books", 
    "id": "vrpkod", 
    "parentId": "hgxogdkheg"
}

{

"path": "/books",

"pathPart": "books",

"id": "vrpkod",

"parentId": "hgxogdkheg"

}

Value of id attribute is RESOURCE ID.

API ID and RESOURCE ID are used for subsequent AWS CLI invocations.

API Gateway POST Method

Now that the resource is created, let’s create HTTP POST method on this resource.

Create a POST method

aws apigateway put-method \
--rest-api-id lb2qgujjif \
--resource-id vrpkod \
--http-method POST \
--authorization-type NONE

aws apigateway put-method \

--rest-api-id lb2qgujjif \

--resource-id vrpkod \

--http-method POST \

--authorization-type NONE

to see the response:

{
    "apiKeyRequired": false, 
    "httpMethod": "POST", 
    "authorizationType": "NONE"
}

{

"apiKeyRequired": false,

"httpMethod": "POST",

"authorizationType": "NONE"

}

Set Lambda function as destination of the POST method:

aws apigateway put-integration \
--rest-api-id lb2qgujjif \
--resource-id vrpkod \
--http-method POST \
--type AWS \
--integration-http-method POST \
--uri arn:aws:apigateway:us-west-1:lambda:path/2015-03-31/functions/arn:aws:lambda:us-west-1:<act-id>:function:MicroservicePost/invocations

aws apigateway put-integration \

--rest-api-id lb2qgujjif \

--resource-id vrpkod \

--http-method POST \

--type AWS \

--integration-http-method POST \

--uri arn:aws:apigateway:us-west-1:lambda:path/2015-03-31/functions/arn:aws:lambda:us-west-1:<act-id>:function:MicroservicePost/invocations

Make sure to replace <act-id> with your AWS account id. API ID and RESOURCE ID from previous section are used here as well. --uri is used to specify the URI of integration input. The format of the URI is fixed. This CLI will show the result as:

{
    "httpMethod": "POST", 
    "passthroughBehavior": "WHEN_NO_MATCH", 
    "cacheKeyParameters": [], 
    "type": "AWS", 
    "uri": "arn:aws:apigateway:us-west-1:lambda:path/2015-03-31/functions/arn:aws:lambda:us-west-1:<act-id>:function:MicroservicePost/invocations", 
    "cacheNamespace": "vrpkod"
}

{

"httpMethod": "POST",

"passthroughBehavior": "WHEN_NO_MATCH",

"cacheKeyParameters": [],

"type": "AWS",

"uri": "arn:aws:apigateway:us-west-1:lambda:path/2015-03-31/functions/arn:aws:lambda:us-west-1:<act-id>:function:MicroservicePost/invocations",

"cacheNamespace": "vrpkod"

}

Set content-type of POST method response:

aws apigateway put-method-response \
--rest-api-id lb2qgujjif \
--resource-id vrpkod \
--http-method POST \
--status-code 200 \
--response-models "{\"application/json\": \"Empty\"}"

aws apigateway put-method-response \

--rest-api-id lb2qgujjif \

--resource-id vrpkod \

--http-method POST \

--status-code 200 \

--response-models "{\"application/json\": \"Empty\"}"

to see the response:

{
    "responseModels": {
        "application/json": "Empty"
    }, 
    "statusCode": "200"
}

{

"responseModels": {

"application/json": "Empty"

"statusCode": "200"

}

Set content-type of POST method integration response:

aws apigateway put-integration-response \
--rest-api-id lb2qgujjif \
--resource-id vrpkod \
--http-method POST \
--status-code 200 \
--response-templates "{\"application/json\": \"Empty\"}"

aws apigateway put-integration-response \

--rest-api-id lb2qgujjif \

--resource-id vrpkod \

--http-method POST \

--status-code 200 \

--response-templates "{\"application/json\": \"Empty\"}"

to see the response:

{
    "statusCode": "200", 
    "responseTemplates": {
        "application/json": "Empty"
    }
}

{

"statusCode": "200",

"responseTemplates": {

"application/json": "Empty"

}

Deploy the API

aws apigateway create-deployment \
--rest-api-id lb2qgujjif \
--stage-name test

aws apigateway create-deployment \

--rest-api-id lb2qgujjif \

--stage-name test

to see the response

{
    "id": "9wi991", 
    "createdDate": 1482999187
}

{

"id": "9wi991",

"createdDate": 1482999187

}

Grant permission to allow API Gateway to invoke Lambda Function:

aws lambda add-permission \
--function-name MicroservicePost \
--statement-id apigateway-test-post-1 \
--action lambda:InvokeFunction \
--principal apigateway.amazonaws.com \
--source-arn "arn:aws:execute-api:us-west-1:<act-id>:lb2qgujjif/*/POST/books"

aws lambda add-permission \

--function-name MicroservicePost \

--statement-id apigateway-test-post-1 \

--action lambda:InvokeFunction \

--principal apigateway.amazonaws.com \

--source-arn "arn:aws:execute-api:us-west-1:<act-id>:lb2qgujjif/*/POST/books"

Also, grant permission to the deployed API:

aws lambda add-permission \
--function-name MicroservicePost \
--statement-id apigateway-test-post-2 \
--action lambda:InvokeFunction \
--principal apigateway.amazonaws.com \
--source-arn "arn:aws:execute-api:us-west-1:<act-id>:lb2qgujjif/test/GET/books"

aws lambda add-permission \

--function-name MicroservicePost \

--statement-id apigateway-test-post-2 \

--action lambda:InvokeFunction \

--principal apigateway.amazonaws.com \

--source-arn "arn:aws:execute-api:us-west-1:<act-id>:lb2qgujjif/test/GET/books"

Test the API method:

aws apigateway test-invoke-method \
--rest-api-id lb2qgujjif \
--resource-id vrpkod \
--http-method POST \
--path-with-query-string "" \
--body "{\"id\": \"1\", \"bookname\": \"test book\", \"isbn\": \"123\", \"cost\": \"1.23\"}"

aws apigateway test-invoke-method \

--rest-api-id lb2qgujjif \

--resource-id vrpkod \

--http-method POST \

--path-with-query-string "" \

--body "{\"id\": \"1\", \"bookname\": \"test book\", \"isbn\": \"123\", \"cost\": \"1.23\"}"

to see the response:

{
    "status": 200, 
    "body": "Empty", 
    "log": "Execution log for request test-request\nThu Dec 29 08:16:05 UTC 2016 : Starting execution for request: test-invoke-request\nThu Dec 29 08:16:05 UTC 2016 : HTTP Method: POST, Resource Path: /books\nThu Dec 29 08:16:05 UTC 2016 : Method request path: {}\nThu Dec 29 08:16:05 UTC 2016 : Method request query string: {}\nThu Dec 29 08:16:05 UTC 2016 : Method request headers: {}\nThu Dec 29 08:16:05 UTC 2016 : Method request body before transformations: {\"id\": \"1\", \"bookname\": \"test book\", \"isbn\": \"123\", \"cost\": \"1.23\"}\nThu Dec 29 08:16:05 UTC 2016 : Endpoint request URI: https://lambda.us-west-1.amazonaws.com/2015-03-31/functions/arn:aws:lambda:us-west-1:598307997273:function:MicroservicePost/invocations\nThu Dec 29 08:16:05 UTC 2016 : Endpoint request headers: {x-amzn-lambda-integration-tag=test-request, Authorization=****************************************************************************************************************************************************************************************************************************************************************************************************************************************c8bb85, X-Amz-Date=20161229T081605Z, x-amzn-apigateway-api-id=lb2qgujjif, X-Amz-Source-Arn=arn:aws:execute-api:us-west-1:598307997273:lb2qgujjif/null/POST/books, Accept=application/json, User-Agent=AmazonAPIGateway_lb2qgujjif, Host=lambda.us-west-1.amazonaws.com, X-Amz-Content-Sha256=559d0296d96ec5647eef6381602fe5e7f55dd17065864fafb4f581d106aa92f4, X-Amzn-Trace-Id=Root=1-5864c645-8494974a41a3a16c8d2f9929, Content-Type=application/json}\nThu Dec 29 08:16:05 UTC 2016 : Endpoint request body after transformations: {\"id\": \"1\", \"bookname\": \"test book\", \"isbn\": \"123\", \"cost\": \"1.23\"}\nThu Dec 29 08:16:10 UTC 2016 : Endpoint response body before transformations: \"{\\\"cost\\\":\\\"1.23\\\",\\\"id\\\":\\\"1\\\",\\\"bookname\\\":\\\"test book\\\",\\\"isbn\\\":\\\"123\\\"}\"\nThu Dec 29 08:16:10 UTC 2016 : Endpoint response headers: {x-amzn-Remapped-Content-Length=0, x-amzn-RequestId=0b25323b-cd9f-11e6-8bd4-292925ba63a9, Connection=keep-alive, Content-Length=78, Date=Thu, 29 Dec 2016 08:16:10 GMT, Content-Type=application/json}\nThu Dec 29 08:16:10 UTC 2016 : Method response body after transformations: Empty\nThu Dec 29 08:16:10 UTC 2016 : Method response headers: {X-Amzn-Trace-Id=Root=1-5864c645-8494974a41a3a16c8d2f9929, Content-Type=application/json}\nThu Dec 29 08:16:10 UTC 2016 : Successfully completed execution\nThu Dec 29 08:16:10 UTC 2016 : Method completed with status: 200\n", 
    "latency": 5091, 
    "headers": {
        "X-Amzn-Trace-Id": "Root=1-5864c645-8494974a41a3a16c8d2f9929", 
        "Content-Type": "application/json"
    }
}

{

"status": 200,

"body": "Empty",

"log": "Execution log for request test-request\nThu Dec 29 08:16:05 UTC 2016 : Starting execution for request: test-invoke-request\nThu Dec 29 08:16:05 UTC 2016 : HTTP Method: POST, Resource Path: /books\nThu Dec 29 08:16:05 UTC 2016 : Method request path: {}\nThu Dec 29 08:16:05 UTC 2016 : Method request query string: {}\nThu Dec 29 08:16:05 UTC 2016 : Method request headers: {}\nThu Dec 29 08:16:05 UTC 2016 : Method request body before transformations: {\"id\": \"1\", \"bookname\": \"test book\", \"isbn\": \"123\", \"cost\": \"1.23\"}\nThu Dec 29 08:16:05 UTC 2016 : Endpoint request URI: https://lambda.us-west-1.amazonaws.com/2015-03-31/functions/arn:aws:lambda:us-west-1:598307997273:function:MicroservicePost/invocations\nThu Dec 29 08:16:05 UTC 2016 : Endpoint request headers: {x-amzn-lambda-integration-tag=test-request, Authorization=****************************************************************************************************************************************************************************************************************************************************************************************************************************************c8bb85, X-Amz-Date=20161229T081605Z, x-amzn-apigateway-api-id=lb2qgujjif, X-Amz-Source-Arn=arn:aws:execute-api:us-west-1:598307997273:lb2qgujjif/null/POST/books, Accept=application/json, User-Agent=AmazonAPIGateway_lb2qgujjif, Host=lambda.us-west-1.amazonaws.com, X-Amz-Content-Sha256=559d0296d96ec5647eef6381602fe5e7f55dd17065864fafb4f581d106aa92f4, X-Amzn-Trace-Id=Root=1-5864c645-8494974a41a3a16c8d2f9929, Content-Type=application/json}\nThu Dec 29 08:16:05 UTC 2016 : Endpoint request body after transformations: {\"id\": \"1\", \"bookname\": \"test book\", \"isbn\": \"123\", \"cost\": \"1.23\"}\nThu Dec 29 08:16:10 UTC 2016 : Endpoint response body before transformations: \"{\\\"cost\\\":\\\"1.23\\\",\\\"id\\\":\\\"1\\\",\\\"bookname\\\":\\\"test book\\\",\\\"isbn\\\":\\\"123\\\"}\"\nThu Dec 29 08:16:10 UTC 2016 : Endpoint response headers: {x-amzn-Remapped-Content-Length=0, x-amzn-RequestId=0b25323b-cd9f-11e6-8bd4-292925ba63a9, Connection=keep-alive, Content-Length=78, Date=Thu, 29 Dec 2016 08:16:10 GMT, Content-Type=application/json}\nThu Dec 29 08:16:10 UTC 2016 : Method response body after transformations: Empty\nThu Dec 29 08:16:10 UTC 2016 : Method response headers: {X-Amzn-Trace-Id=Root=1-5864c645-8494974a41a3a16c8d2f9929, Content-Type=application/json}\nThu Dec 29 08:16:10 UTC 2016 : Successfully completed execution\nThu Dec 29 08:16:10 UTC 2016 : Method completed with status: 200\n",

"latency": 5091,

"headers": {

"X-Amzn-Trace-Id": "Root=1-5864c645-8494974a41a3a16c8d2f9929",

"Content-Type": "application/json"

}

Value of status attribute is 200 and indicates this was a successful invocation. Value of log attribute shows the log statement from CloudWatch Logs. Detailed logs can also be obtained using aws logs filter-log-events --log-group /aws/lambda/MicroservicePost.

This command stores a single JSON document in Couchbase. This can be easily verified using the Couchbase CLI Tool cbq.Connect to the Couchbase server as:

cbq -u Administrator -p password -e="http://<COUCHBASE_HOST>:8091"

cbq -u Administrator -p password -e="http://<COUCHBASE_HOST>:8091"

Create a primary index on default bucket as this is required to query the bucket with no clauses:

cbq> create primary index default_index on default;
{
    "requestID": "13b539f9-7fff-4386-92f4-cea161a7aa08",
    "signature": null,
    "results": [
    ],
    "status": "success",
    "metrics": {
        "elapsedTime": "1.917009047s",
        "executionTime": "1.916970061s",
        "resultCount": 0,
        "resultSize": 0
    }
}

cbq> create primary index default_index on default;

{

"requestID": "13b539f9-7fff-4386-92f4-cea161a7aa08",

"signature": null,

"results": [

"status": "success",

"metrics": {

"elapsedTime": "1.917009047s",

"executionTime": "1.916970061s",

"resultCount": 0,

"resultSize": 0

}

Write a N1QL query to access the data:

cbq> select * from default limit 10;
{
    "requestID": "d7b1c3f9-6b4e-4952-9a1e-9faf5169926e",
    "signature": {
        "*": "*"
    },
    "results": [
        {
            "default": {
                "bookname": "test",
                "cost": "1.23",
                "id": "1",
                "isbn": "123"
            }
        }
    ],
    "status": "success",
    "metrics": {
        "elapsedTime": "24.337755ms",
        "executionTime": "24.289796ms",
        "resultCount": 1,
        "resultSize": 175
    }
}

cbq> select * from default limit 10;

{

"requestID": "d7b1c3f9-6b4e-4952-9a1e-9faf5169926e",

"signature": {

"*": "*"

"results": [

{

"default": {

"bookname": "test",

"cost": "1.23",

"id": "1",

"isbn": "123"

}

"status": "success",

"metrics": {

"elapsedTime": "24.337755ms",

"executionTime": "24.289796ms",

"resultCount": 1,

"resultSize": 175

}

The results show the JSON document that was stored by our Lambda function.

API Gateway GET Method

Let’s create HTTP GET method on the resource:

Create a GET method:

aws apigateway put-method \
--rest-api-id lb2qgujjif \
--resource-id vrpkod \
--http-method GET \
--authorization-type NONE

aws apigateway put-method \

--rest-api-id lb2qgujjif \

--resource-id vrpkod \

--http-method GET \

--authorization-type NONE

Set correct Lambda function as destination of GET:

aws apigateway put-integration \
--rest-api-id lb2qgujjif \
--resource-id vrpkod \
--http-method GET \
--type AWS \
--integration-http-method POST \
--uri arn:aws:apigateway:us-west-1:lambda:path/2015-03-31/functions/arn:aws:lambda:us-west-1:598307997273:function:MicroserviceGetAll/invocations

aws apigateway put-integration \

--rest-api-id lb2qgujjif \

--resource-id vrpkod \

--http-method GET \

--type AWS \

--integration-http-method POST \

--uri arn:aws:apigateway:us-west-1:lambda:path/2015-03-31/functions/arn:aws:lambda:us-west-1:598307997273:function:MicroserviceGetAll/invocations

Set content-type of GET method response:

aws apigateway put-method-response \
--rest-api-id lb2qgujjif \
--resource-id vrpkod \
--http-method GET \
--status-code 200 \
--response-models "{\"application/json\": \"Empty\"}"

aws apigateway put-method-response \

--rest-api-id lb2qgujjif \

--resource-id vrpkod \

--http-method GET \

--status-code 200 \

--response-models "{\"application/json\": \"Empty\"}"

Set content-type of GET method integration response:

aws apigateway put-integration-response \
--rest-api-id lb2qgujjif \
--resource-id vrpkod \
--http-method GET \
--status-code 200 \
--response-templates "{\"application/json\": \"Empty\"}"

aws apigateway put-integration-response \

--rest-api-id lb2qgujjif \

--resource-id vrpkod \

--http-method GET \

--status-code 200 \

--response-templates "{\"application/json\": \"Empty\"}"

Grant permission to allow API Gateway to invoke Lambda Function

aws lambda add-permission \
--function-name MicroserviceGetAll \
--statement-id apigateway-test-getall-1 \
--action lambda:InvokeFunction \
--principal apigateway.amazonaws.com \
--source-arn "arn:aws:execute-api:us-west-1:598307997273:lb2qgujjif/*/GET/books"

aws lambda add-permission \

--function-name MicroserviceGetAll \

--statement-id apigateway-test-getall-1 \

--action lambda:InvokeFunction \

--principal apigateway.amazonaws.com \

--source-arn "arn:aws:execute-api:us-west-1:598307997273:lb2qgujjif/*/GET/books"

Grant permission to the deployed API:

aws lambda add-permission \
--function-name MicroserviceGetAll \
--statement-id apigateway-test-getall-2 \
--action lambda:InvokeFunction \
--principal apigateway.amazonaws.com \
--source-arn "arn:aws:execute-api:us-west-1:598307997273:lb2qgujjif/test/GET/books"

aws lambda add-permission \

--function-name MicroserviceGetAll \

--statement-id apigateway-test-getall-2 \

--action lambda:InvokeFunction \

--principal apigateway.amazonaws.com \

--source-arn "arn:aws:execute-api:us-west-1:598307997273:lb2qgujjif/test/GET/books"

Test the method:

aws apigateway test-invoke-method \
--rest-api-id lb2qgujjif \
--resource-id vrpkod \
--http-method GET

aws apigateway test-invoke-method \

--rest-api-id lb2qgujjif \

--resource-id vrpkod \

--http-method GET

to see the output:

{
    "status": 200, 
    "body": "Empty", 
    "log": "Execution log for request test-request\nSat Dec 31 09:07:48 UTC 2016 : Starting execution for request: test-invoke-request\nSat Dec 31 09:07:48 UTC 2016 : HTTP Method: GET, Resource Path: /books\nSat Dec 31 09:07:48 UTC 2016 : Method request path: {}\nSat Dec 31 09:07:48 UTC 2016 : Method request query string: {}\nSat Dec 31 09:07:48 UTC 2016 : Method request headers: {}\nSat Dec 31 09:07:48 UTC 2016 : Method request body before transformations: \nSat Dec 31 09:07:48 UTC 2016 : Endpoint request URI: https://lambda.us-west-1.amazonaws.com/2015-03-31/functions/arn:aws:lambda:us-west-1:598307997273:function:MicroserviceGetAll/invocations\nSat Dec 31 09:07:48 UTC 2016 : Endpoint request headers: {x-amzn-lambda-integration-tag=test-request, Authorization=******************************************************************************************************************************************************************************************************************************************************************************************************6de147, X-Amz-Date=20161231T090748Z, x-amzn-apigateway-api-id=lb2qgujjif, X-Amz-Source-Arn=arn:aws:execute-api:us-west-1:598307997273:lb2qgujjif/null/GET/books, Accept=application/json, User-Agent=AmazonAPIGateway_lb2qgujjif, X-Amz-Security-Token=FQoDYXdzEHEaDEILpsKTo45Ys1LrFCK3A+KOe5HXOSP3GfVAaRYHe1pDUJGHL9MtkFiPjORLFT+UCKjRqE7UFaGscTVG6PZXTuSyQev4XTyROfPylCrtDomGsoZF/iwy4rlJQIJ7elBceyeKu1OVdaT1A99PVeliaCAiDL6Veo1viWOnP+7c72nAaJ5jnyF/nHl/OLhFdFv4t/hnx3JePMk5YM89/6ofxUEVDNfzXxbZHRpTrG/4TPHwjPdoR5i9dEzWMU6Eo5xD4ldQ/m5B3RmrwpaPOuEq39LhJ8k/Vzo+pAfgJTq5ssbNwYOgh0RPSGVNMcoTkCwk0EMMT5vDbmQqZ2dW1a1tmQg9N2xR+QQy+RKMFgO5YY8fMxHnRSdMuuipxl79G1pktc [TRUNCATED]\nSat Dec 31 09:07:48 UTC 2016 : Endpoint request body after transformations: \nSat Dec 31 09:07:53 UTC 2016 : Endpoint response body before transformations: \"[{\\\"default\\\":{\\\"cost\\\":\\\"1.23\\\",\\\"id\\\":\\\"1\\\",\\\"bookname\\\":\\\"test book\\\",\\\"isbn\\\":\\\"123\\\"}}]\"\nSat Dec 31 09:07:53 UTC 2016 : Endpoint response headers: {x-amzn-Remapped-Content-Length=0, x-amzn-RequestId=99ab09b2-cf38-11e6-996f-f5f07af431af, Connection=keep-alive, Content-Length=94, Date=Sat, 31 Dec 2016 09:07:52 GMT, Content-Type=application/json}\nSat Dec 31 09:07:53 UTC 2016 : Method response body after transformations: Empty\nSat Dec 31 09:07:53 UTC 2016 : Method response headers: {X-Amzn-Trace-Id=Root=1-58677564-66f1e96642b16d2db703126e, Content-Type=application/json}\nSat Dec 31 09:07:53 UTC 2016 : Successfully completed execution\nSat Dec 31 09:07:53 UTC 2016 : Method completed with status: 200\n", 
    "latency": 4744, 
    "headers": {
        "X-Amzn-Trace-Id": "Root=1-58677564-66f1e96642b16d2db703126e", 
        "Content-Type": "application/json"
    }
}

{

"status": 200,

"body": "Empty",

"log": "Execution log for request test-request\nSat Dec 31 09:07:48 UTC 2016 : Starting execution for request: test-invoke-request\nSat Dec 31 09:07:48 UTC 2016 : HTTP Method: GET, Resource Path: /books\nSat Dec 31 09:07:48 UTC 2016 : Method request path: {}\nSat Dec 31 09:07:48 UTC 2016 : Method request query string: {}\nSat Dec 31 09:07:48 UTC 2016 : Method request headers: {}\nSat Dec 31 09:07:48 UTC 2016 : Method request body before transformations: \nSat Dec 31 09:07:48 UTC 2016 : Endpoint request URI: https://lambda.us-west-1.amazonaws.com/2015-03-31/functions/arn:aws:lambda:us-west-1:598307997273:function:MicroserviceGetAll/invocations\nSat Dec 31 09:07:48 UTC 2016 : Endpoint request headers: {x-amzn-lambda-integration-tag=test-request, Authorization=******************************************************************************************************************************************************************************************************************************************************************************************************6de147, X-Amz-Date=20161231T090748Z, x-amzn-apigateway-api-id=lb2qgujjif, X-Amz-Source-Arn=arn:aws:execute-api:us-west-1:598307997273:lb2qgujjif/null/GET/books, Accept=application/json, User-Agent=AmazonAPIGateway_lb2qgujjif, X-Amz-Security-Token=FQoDYXdzEHEaDEILpsKTo45Ys1LrFCK3A+KOe5HXOSP3GfVAaRYHe1pDUJGHL9MtkFiPjORLFT+UCKjRqE7UFaGscTVG6PZXTuSyQev4XTyROfPylCrtDomGsoZF/iwy4rlJQIJ7elBceyeKu1OVdaT1A99PVeliaCAiDL6Veo1viWOnP+7c72nAaJ5jnyF/nHl/OLhFdFv4t/hnx3JePMk5YM89/6ofxUEVDNfzXxbZHRpTrG/4TPHwjPdoR5i9dEzWMU6Eo5xD4ldQ/m5B3RmrwpaPOuEq39LhJ8k/Vzo+pAfgJTq5ssbNwYOgh0RPSGVNMcoTkCwk0EMMT5vDbmQqZ2dW1a1tmQg9N2xR+QQy+RKMFgO5YY8fMxHnRSdMuuipxl79G1pktc [TRUNCATED]\nSat Dec 31 09:07:48 UTC 2016 : Endpoint request body after transformations: \nSat Dec 31 09:07:53 UTC 2016 : Endpoint response body before transformations: \"[{\\\"default\\\":{\\\"cost\\\":\\\"1.23\\\",\\\"id\\\":\\\"1\\\",\\\"bookname\\\":\\\"test book\\\",\\\"isbn\\\":\\\"123\\\"}}]\"\nSat Dec 31 09:07:53 UTC 2016 : Endpoint response headers: {x-amzn-Remapped-Content-Length=0, x-amzn-RequestId=99ab09b2-cf38-11e6-996f-f5f07af431af, Connection=keep-alive, Content-Length=94, Date=Sat, 31 Dec 2016 09:07:52 GMT, Content-Type=application/json}\nSat Dec 31 09:07:53 UTC 2016 : Method response body after transformations: Empty\nSat Dec 31 09:07:53 UTC 2016 : Method response headers: {X-Amzn-Trace-Id=Root=1-58677564-66f1e96642b16d2db703126e, Content-Type=application/json}\nSat Dec 31 09:07:53 UTC 2016 : Successfully completed execution\nSat Dec 31 09:07:53 UTC 2016 : Method completed with status: 200\n",

"latency": 4744,

"headers": {

"X-Amzn-Trace-Id": "Root=1-58677564-66f1e96642b16d2db703126e",

"Content-Type": "application/json"

}

Once again, 200 status code shows a successful invocation. Detailed logs can be obtained using aws logs filter-log-events --log-group /aws/lambda/MicroservicePost.

This blog only shows one simple POST and GET methods. Other HTTP methods can be very easily included in this microservice as well.

API Gateway and Lambda References

Serverless Architectures
AWS API Gateway
Creating a simple Microservice using Lambda and API Gateway
Couchbase Server Docs
Couchbase Forums
Follow us at @couchbasedev

Source: blog.couchbase.com/2016/december/microservice-aws-api-gateway-lambda-couchbase

AWS IoT Button, Lambda and Couchbase

December 25, 2016couchbase, techtipamazon, aws, couchbase, iotarungupta

Getting Started with Serverless FaaS and AWS Lambda shows how to use a simple Java function to store a JSON document to Couchbase using AWS Lambda. This blog builds upon that and shows how an AWS IoT Button can be used as a trigger for that Lambda function.

By end of this blog, you’ll learn:

How to configure AWS IoT Button
Use IoT Button as trigger for Lambda Function
Test IoT button

The overall flow will be:

serverless-iot-couchbase

Iot button click will invoke HelloCouchbaseLambda Lambda function. This function is uses Couchbase Java SDK to create a JSON document in Couchbase.

This blog is also playing catch up with Collecting iBeacon Data with Couchbase and Raspberry Pi IoT Devices by Nic and The CouchCase by Matthew on their summer projects. One last blog will be published in this series. That will show how multiple AWS IoT buttons can be used for some fun.

Let’s get started!

Configure IoT Button

The fastest way to configure IoT button is using the mobile app for iOS or Android.

More details about configuring IoT Button using mobile app.

Here are some snapshots from configuring button using the mobile app.

Bring up the app, click on + to start configuring a new button:

Enter button’s serial number:

Configure the button with wifi network:

Upload all the certificates etc:

After this, the button is configured and ready to use. This blog skipped the part where a template Lambda Function is associated with the button click.

If mobile app cannot be used then the button can be configured manually.

Use IoT Button as Trigger for Lambda Function

The aws lambda create-event-source-mapping CLI allows to create an event source for Lambda function. As of AWS CLI version 1.11.21, only Amazon Kinesis stream or an Amazon DynamoDB stream can be used. But for this blog, we’ll use IoT button as a trigger. And this has to be configured using AWS Lambda Console.

IoT Button is only supported in a limited number of regions. For example, it is not supported in the us-west-1 region but us-west-2 region works.

The list of regions not supported are greyed out in the following list:

Lambda Function can be triggered by several events. Lambda Function is invoked when any of these events occur. By default, no triggers are associated with a Lambda Function. For our HelloCouchbaseLambda function, these can be seen at us-west-2.console.aws.amazon.com/lambda/home?region=us-west-2#/functions/HelloCouchbaseLambda?tab=triggers.

Click on Add trigger to add a new trigger:

Select on the empty square to create a new trigger, and select AWS IoT:

For the button previously registered, get the serial number from us-west-2.console.aws.amazon.com/iotv2/home?region=us-west-2#/thinghub:

Specify the serial number of the button in the AWS IoT trigger:

Click on Submit to create the trigger:

And this confirms that the trigger has been added.

Test IoT Button

Before testing the button, let’s login to the Couchbase instance and verify the number of JSON documents in the bucket:

This can be verified at http://<EC2-IP-Address>:8091/index.html#sec=buckets. As expected, no documents exists in the bucket.

Press the button once, and refresh the page. It shows that one document is now stored in the bucket. This is verified in the Couchbase Web Console:

Click on Documents to see the complete list of documents:

Click on the document ID to see more details about the document:

Only timestamp is stored in this JSON document.

Now, let’s update HelloCouchbaseLambda code to include request id in the document as well. This can be achieved by adding the following line of code in the Java class:

buttonDocument.setRequestId(context.getAwsRequestId());

buttonDocument.setRequestId(context.getAwsRequestId());

A new deployment package can be built and uploaded using the following command:

mvn clean package; 
aws lambda update-function-code \
--function-name HelloCouchbaseLambda \
--zip-file fileb:///Users/arungupta/workspaces/serverless/aws/hellocouchbase/hellocouchbase/target/hellocouchbase-1.0-SNAPSHOT.jar \
--region us-west-2 \
--publish

mvn clean package;

aws lambda update-function-code \

--function-name HelloCouchbaseLambda \

--zip-file fileb:///Users/arungupta/workspaces/serverless/aws/hellocouchbase/hellocouchbase/target/hellocouchbase-1.0-SNAPSHOT.jar \

--region us-west-2 \

--publish

Now clicking the button will update the number of documents. But the updated document will have an additional attribute populated as shown:

How are you going to take AWS IoT button and use it with Lambda and Couchbase? Let us know at Couchbase Forums.

References

AWS IoT Button
AWS IoT Button Developer Guide
Couchbase Server Docs
Couchbase Forums
Follow us at @couchbasedev

Source: https://blog.couchbase.com/2016/december/aws-iot-button-lambda-couchbase

Serverless FaaS with AWS Lambda and Java

December 24, 2016couchbase, techtipamazon, lambda, serverlessarungupta

What is Serverless Architecture?

Serverless architecture runs custom code in ephemeral containers that are fully managed by a 3rd party. The custom code is typically a small part of a complete application. It is also called as function. This gives another name for serverless architecture as Function as a Service (FaaS). The container is ephemeral because it may only last for one invocation. The container may be reused but that’s not something you can rely upon. As a developer, you upload the code to FaaS platform, the service then handles all the capacity, scaling, patching and administration of the infrastructure to run your code.

An application built using Serverless Architecture follows the event-driven approach. For example, an activity happened in the application such as a click. This is

This is very different from a classical architecture where the application code is typically deployed in an application server such as Tomcat or WildFly. Scaling your application means starting additional instances of the application server or spinning up additional containers with the packaged application server. The Load Balancer need to be updated with the new IP addresses. Operating system need to be patched, upgraded and maintained.

Serverless Architectures explain the difference between the classical programming model and this new serverless architecture.

FaaS platform takes your application is divided into multiple functions. Each function is deployed in FaaS. The service spins up additional compute instances to meet the scalability demands of your application. FaaS platform provides the execution environment and takes care of starting and tearing down the containers to run your function.

Read Serverless Architectures for more details about these images.

One of the big advantages of FaaS is that you are only charged for the compute time, i.e. the time your code is running. There is no charge when your code is not running.

Another way to look at how Functions are different from VMs and Containers:

Note that Linux containers instead of Docker containers are used as an implementation for AWS Lambda.

How is FaaS different from PaaS?

As quoted at Serverless Architectures, a quick answer is provided by the following tweet:

If your PaaS can efficiently start instances in 20ms that run for half a second, then call it serverless. https://t.co/S3YzvqFYLR

— adrian cockcroft (@adrianco) May 28, 2016

In other words most PaaS applications are not geared towards bringing entire applications up and down for every request, whereas FaaS platforms do exactly this.

Abstracting the Back-end with FaaS explain the difference with different *aaS offerings. The image from the blog is captured below:

faas

Serverless Architectures also provide great details about what FaaS is and is not.

AWS Lambda, Google Cloud Functions and Azure Functions are some of the options for running serverless applications.

This blog will show how to write your first AWS Lambda function.

What is AWS Lambda?

AWS Lambda is FaaS service from Amazon Web Services. It runs your code on a high-availability compute infrastructure and performs all of the administration of the compute resources, including server and operating system maintenance, capacity provisioning and automatic scaling, code monitoring and logging.

AWS Lambda charges you for the duration your code runs in increments of 100ms. There is no cost associated with storing the Lambda function in AWS. First million requests per month are free and the pricing after that is nominal. Read more details on Lambda pricing. It also provides visibility into performance by providing real time metrics and logs to AWS CloudWatch. All you need to do is write the code!

Here is a quick introduction:

Also check out What’s New in AWS Lambda from AWS ReInvent 2016:

Also checkout Serverless Architectural Patterns and Best Practices from AWS ReInvent 2016:

The code you run on AWS Lambda is called a Lambda Function. You upload your code as a zip file or design it using the AWS Lambda Management Console. There is a built-in support for AWS SDK and this simplifies the ability to call other AWS services.

In short, Lambda is scalable, serverless, compute in the cloud.

AWS Lambda provides several execution environments:

Node.js – v0.10.36, v4.3.2 (recommended)
Java – Java 8
Python – Python 2.7
.NET Core – .NET Core 1.0.1 (C#)

This blog will show:

Build a Java application that stores a JSON document to Couchbase
Use Maven to create a deployment package for Java application
Create a Lambda Function
Update the Lambda Function

The complete code in this blog is available at github.com/arun-gupta/serverless/tree/master/aws/hellocouchbase.

Java Application for AWS Lambda

First, lets look at a Java application that will be used for this Lambda function. Programming Model for Lambda Functions in Java provide more details about how to write your Lambda function code in Java.

Our Lambda function will implemented the pre-defined interface com.amazonaws.services.lambda.runtime.RequestHandler. The code looks like:

public class HelloCouchbase implements RequestHandler<Request, String> {

    CouchbaseCluster cluster;
    Bucket bucket;
    LambdaLogger logger;

    @Override
    public String handleRequest(Request request, Context context) {
        SimpleDateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.SSS");
        String timestamp = dateFormat.format(Calendar.getInstance().getTime());

        logger = context.getLogger();
        logger.log("Request received: %s" + timestamp);
        ButtonDocument buttonDocument = new ButtonDocument();
        buttonDocument.setId(context.getAwsRequestId());
        buttonDocument.setRequestId(context.getAwsRequestId());
        buttonDocument.setTimestamp(String.valueOf(timestamp));
        
        getBucket().upsert(buttonDocument.toJson());

        return buttonDocument.toString();
    }

public class HelloCouchbase implements RequestHandler<Request, String> {

CouchbaseCluster cluster;

Bucket bucket;

LambdaLogger logger;

@Override

public String handleRequest(Request request, Context context) {

SimpleDateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.SSS");

String timestamp = dateFormat.format(Calendar.getInstance().getTime());

logger = context.getLogger();

logger.log("Request received: %s" + timestamp);

ButtonDocument buttonDocument = new ButtonDocument();

buttonDocument.setId(context.getAwsRequestId());

buttonDocument.setRequestId(context.getAwsRequestId());

buttonDocument.setTimestamp(String.valueOf(timestamp));

getBucket().upsert(buttonDocument.toJson());

return buttonDocument.toString();

}

handleRequest method is where the function code is implemented. Context provides useful information about Lambda execution environment. Some of the information from the context is stored a JSON document. Finally, Couchbase Java SDK API upsert is used to write a JSON document to the identified Couchbase instance. Couchbase on Amazon EC2 provide complete instructions to install Couchbase on AWS EC2.

Information about the Couchbase server is obtained as:

public CouchbaseCluster getCluster() {
    if (null == cluster) {
        logger.log("env: " + System.getenv("COUCHBASE_HOST"));
        cluster = CouchbaseCluster.create(System.getenv("COUCHBASE_HOST"));
    }
    return cluster;
}

public CouchbaseCluster getCluster() {

if (null == cluster) {

logger.log("env: " + System.getenv("COUCHBASE_HOST"));

cluster = CouchbaseCluster.create(System.getenv("COUCHBASE_HOST"));

}

return cluster;

}

This is once again using Couchbase Java API CouchbaseCluster as a main entry point to the Couchbase cluster. The COUCHBASE_HOST environment variable is passed when the Lambda function is created. In our case, this would point to a single node Couchbase cluster running on AWS EC2. Environment variables were recently introduced in AWS Lambda.

Finally, you need to access bucket in the server:

public Bucket getBucket() {
    while (null == bucket) {
        logger.log("Trying to connect to the database");
        bucket = getCluster().openBucket("serverless", 2L, TimeUnit.MINUTES);

        try {
            Thread.sleep(3000);
        } catch (Exception e) {
            logger.log("Thread sleep Exception: " + e.toString());
            throw new RuntimeException(e);
        }
    }

    return bucket;
}

public Bucket getBucket() {

while (null == bucket) {

logger.log("Trying to connect to the database");

bucket = getCluster().openBucket("serverless", 2L, TimeUnit.MINUTES);

try {

Thread.sleep(3000);

} catch (Exception e) {

logger.log("Thread sleep Exception: " + e.toString());

throw new RuntimeException(e);

}

return bucket;

}

The bucket name is serverless and all JSON documents are stored in this.

A simple Hello World application may be used for creating this function as well.

Create AWS Lambda Deployment Package

AWS Lambda function needs a deployment package. This package is either a .zip or .jar file that contains all the dependencies of the function. Our application is packaged using Maven, and so we’ll use a Maven plugin to create a deployment package.

The application has pom.xml with the following plugin fragment:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <version>2.3</version>
    <configuration>
        <createDependencyReducedPom>false</createDependencyReducedPom>
    </configuration>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <goal>shade</goal>
            </goals>
        </execution>
    </executions>
</plugin>

<groupId>org.apache.maven.plugins</groupId>

<artifactId>maven-shade-plugin</artifactId>

<createDependencyReducedPom>false</createDependencyReducedPom>

</configuration>

<phase>package</phase>

<goals>

<goal>shade</goal>

</goals>

</execution>

</executions>

</plugin>

More details about Maven configuration are available in Creating a .jar Deployment Package Using Maven without any IDE. The maven-shade-plugin allows to create an uber-jar including all the dependencies. The shade goal is tied to the package phase. So the mvn package command will generate a single deployment jar.

Package the application using mvn package command. This will show the output:

[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ hellocouchbase ---
[INFO] Building jar: /Users/arungupta/workspaces/serverless/aws/hellocouchbase/hellocouchbase/target/hellocouchbase-1.0-SNAPSHOT.jar
[INFO] 
[INFO] --- maven-shade-plugin:2.3:shade (default) @ hellocouchbase ---
[INFO] Including com.amazonaws:aws-lambda-java-core:jar:1.1.0 in the shaded jar.
[INFO] Including com.couchbase.client:java-client:jar:2.3.6 in the shaded jar.
[INFO] Including com.couchbase.client:core-io:jar:1.3.6 in the shaded jar.
[INFO] Including io.reactivex:rxjava:jar:1.1.8 in the shaded jar.
[INFO] Replacing original artifact with shaded artifact.
[INFO] Replacing /Users/arungupta/workspaces/serverless/aws/hellocouchbase/hellocouchbase/target/hellocouchbase-1.0-SNAPSHOT.jar with /Users/arungupta/workspaces/serverless/aws/hellocouchbase/hellocouchbase/target/hellocouchbase-1.0-SNAPSHOT-shaded.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------

[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ hellocouchbase ---

[INFO] Building jar: /Users/arungupta/workspaces/serverless/aws/hellocouchbase/hellocouchbase/target/hellocouchbase-1.0-SNAPSHOT.jar

[INFO]

[INFO] --- maven-shade-plugin:2.3:shade (default) @ hellocouchbase ---

[INFO] Including com.amazonaws:aws-lambda-java-core:jar:1.1.0 in the shaded jar.

[INFO] Including com.couchbase.client:java-client:jar:2.3.6 in the shaded jar.

[INFO] Including com.couchbase.client:core-io:jar:1.3.6 in the shaded jar.

[INFO] Including io.reactivex:rxjava:jar:1.1.8 in the shaded jar.

[INFO] Replacing original artifact with shaded artifact.

[INFO] Replacing /Users/arungupta/workspaces/serverless/aws/hellocouchbase/hellocouchbase/target/hellocouchbase-1.0-SNAPSHOT.jar with /Users/arungupta/workspaces/serverless/aws/hellocouchbase/hellocouchbase/target/hellocouchbase-1.0-SNAPSHOT-shaded.jar

[INFO] ------------------------------------------------------------------------

[INFO] BUILD SUCCESS

[INFO] ------------------------------------------------------------------------

The target/hello-couchbase-1.0-SNAPSHOT.jar is the shaded jar that will be deployed to AWS Lambda.

More details about creating a deployment package are at Creating a Deployment Package.

Create AWS Lambda Function

Create AWS Lambda Function using AWS CLI. The CLI command in this case looks like:

aws lambda create-function \
--function-name HelloWorld \
--role arn:aws:iam::<account-id>:role/service-role/myLambdaRole \
--zip-file fileb:///Users/arungupta/workspaces/serverless/aws/hellocouchbase/hellocouchbase/target/hellocouchbase-1.0-SNAPSHOT.jar \
--handler org.sample.serverless.aws.couchbase.HelloCouchbaseLambda \
--description "Hello Couchbase Lambda" \
--runtime java8  \
--region us-west-2 \
--timeout 30 \
--memory-size 1024 \
--publish

aws lambda create-function \

--function-name HelloWorld \

--role arn:aws:iam::<account-id>:role/service-role/myLambdaRole \

--zip-file fileb:///Users/arungupta/workspaces/serverless/aws/hellocouchbase/hellocouchbase/target/hellocouchbase-1.0-SNAPSHOT.jar \

--handler org.sample.serverless.aws.couchbase.HelloCouchbaseLambda \

--description "Hello Couchbase Lambda" \

--runtime java8 \

--region us-west-2 \

--timeout 30 \

--memory-size 1024 \

--publish

In this CLI:

create-function creates a Lambda function
--function-name provides the function name. The function name is case sensitive.
--role specifies Amazon Resource Name (ARN) of an IAM role that Lambda assume when it executes your function to access any other AWS resources. If you’ve executed a Lambda function using AWS Console then this role is created for you.
--zip-file points to the deployment package that was created in previous step. fileb is an AWS CLI specific protocol to indicate that the content uploaded is binary.
--handler is the Java class that is called to begin execution of the function
--publish request AWS Lambda to create the Lambda function and publish a version as an atomic operation. Otherwise multiple versions may be created and may be published at a later point.

Lambda Console shows:

Test AWS Lambda Function

Test the AWS Lambda Function using AWS CLI.

aws lambda invoke \
--function-name HelloCouchbaseLambda \
--region us-west-2 \
--payload '' \
hellocouchbase.out

aws lambda invoke \

--function-name HelloCouchbaseLambda \

--region us-west-2 \

--payload '' \

hellocouchbase.out

It shows the output as:

{
    "StatusCode": 200
}

{

"StatusCode": 200

}

The output from the command is stored in hellocouchbase.out and looks like:

"{\"id\":\"e6bbe71a-ca4f-11e6-95a7-95f2ed410493\",\"installationId\":null,\"requestId\":\"e6bbe71a-ca4f-11e6-95a7-95f2ed410493\",\"identityId\":null,\"timestamp\":\"2016-12-25 03:12:01.157\"}"

"{\"id\":\"e6bbe71a-ca4f-11e6-95a7-95f2ed410493\",\"installationId\":null,\"requestId\":\"e6bbe71a-ca4f-11e6-95a7-95f2ed410493\",\"identityId\":null,\"timestamp\":\"2016-12-25 03:12:01.157\"}"

Invoking this function stores a JSON document in Couchbase. Documents stored in Couchbase can be seen using Couchbase Web Console. The password is Administrator and password is the EC2 instance id.

All data buckets in this Couchbase instance are shown below:

Note that the serverless bucket is manually created.

Clicking on Documents shows details of different documents stored in the bucket:

Clicking on each document shows more details about the JSON document:

Lambda function can also be tested using the Console:

Update AWS Lambda Function

If the application logic changes then a new deployment package needs to be uploaded for the Lambda function. In this case, mvn package will create a deployment package and aws lambda CLI command is used to update the function code:

aws lambda update-function-code \
--function-name HelloCouchbaseLambda \
--zip-file fileb:///Users/arungupta/workspaces/serverless/aws/hellocouchbase/hellocouchbase/target/hellocouchbase-1.0-SNAPSHOT.jar \
--region us-west-2 \
--publish

aws lambda update-function-code \

--function-name HelloCouchbaseLambda \

--zip-file fileb:///Users/arungupta/workspaces/serverless/aws/hellocouchbase/hellocouchbase/target/hellocouchbase-1.0-SNAPSHOT.jar \

--region us-west-2 \

--publish

Shows the result:

{
    "CodeSha256": "w510ejw/OoVsQt2JiLG2bPZPAaFvQCRrYYYlQWctCQE=", 
    "FunctionName": "HelloCouchbaseLambda", 
    "CodeSize": 6978108, 
    "MemorySize": 1024, 
    "FunctionArn": "arn:aws:lambda:us-west-2:<account-id>:function:HelloCouchbaseLambda:8", 
    "Environment": {
        "Variables": {
            "COUCHBASE_HOST": "ec2-35-165-249-235.us-west-2.compute.amazonaws.com"
        }
    }, 
    "Version": "8", 
    "Role": "arn:aws:iam::<account-id>:role/service-role/myLambdaRole", 
    "Timeout": 30, 
    "LastModified": "2016-12-25T04:17:38.717+0000", 
    "Handler": "org.sample.serverless.aws.couchbase.HelloCouchbaseLambda", 
    "Runtime": "java8", 
    "Description": "Java Hello Couchbase"
}

{

"CodeSha256": "w510ejw/OoVsQt2JiLG2bPZPAaFvQCRrYYYlQWctCQE=",

"FunctionName": "HelloCouchbaseLambda",

"CodeSize": 6978108,

"MemorySize": 1024,

"FunctionArn": "arn:aws:lambda:us-west-2:<account-id>:function:HelloCouchbaseLambda:8",

"Environment": {

"Variables": {

"COUCHBASE_HOST": "ec2-35-165-249-235.us-west-2.compute.amazonaws.com"

}

"Version": "8",

"Role": "arn:aws:iam::<account-id>:role/service-role/myLambdaRole",

"Timeout": 30,

"LastModified": "2016-12-25T04:17:38.717+0000",

"Handler": "org.sample.serverless.aws.couchbase.HelloCouchbaseLambda",

"Runtime": "java8",

"Description": "Java Hello Couchbase"

}

The function can then be invoked again.

During writing of this blog, this was often used to debug the function as well. This is because Lambda functions do not have any state or box associated with them. And so you cannot log in to a box to check out if the function did not deploy correctly. You can certainly use CloudWatch log statements once the function is working.

AWS Lambda References

Serverless Architectures
AWS Lambda: How it works
Couchbase Server Docs
Couchbase Forums
Follow us at @couchbasedev

Source: https://blog.couchbase.com/2016/december/serverless-faas-aws-lambda-java

Stateful Containers on Kubernetes using Persistent Volume and Amazon EBS

July 30, 2016containers, couchbaseamazon, containers, couchbase, docker, kubernetesarungupta

This blog will show how to create stateful containers in Kubernetes using Amazon EBS.

Couchbase is a stateful container. This means that state of the container needs to be carried with it. In Kubernetes, the smallest atomic unit of running a container is a pod. So a Couchbase container will run as a pod. And by default, all data stored in Couchbase is stored on the same host.

This figure is originally explained in Kubernetes Cluster on Amazon and Expose Couchbase Service. In addition, this figure shows storage local to the host.

Pods are ephemeral and may be restarted on a different host. A Kubernetes Volume outlives any containers that run within the pod, and data is preserved across container restarts. However the volume will cease to exist when a pod ceases to exist. This is solved by Persistent Volumes that provide persistent, cluster-scoped storage for applications that require long lived data.

Creating and using a persistent volume is a three step process:

Provision: Administrator provision a networked storage in the cluster, such as AWS ElasticBlockStore volumes. This is called as PersistentVolume.
Request storage: User requests storage for pods by using claims. Claims can specify levels of resources (CPU and memory), specific sizes and access modes (e.g. can be mounted once read/write or many times write only). This is called as PersistentVolumeClaim.
Use claim: Claims are mounted as volumes and used in pods for storage.

Specifically, this blog will show how to use an AWS ElasticBlockStore as PersistentVolume, create a PersistentVolumeClaim, and then claim it in a pod.

stateful containers

Complete source code for this blog is at: github.com/arun-gupta/couchbase-kubernetes.

Provision AWS Elastic Block Storage

Following restrictions need to be met if Amazon ElasticBlockStorage is used as a PersistentVolume with Kubernetes:

the nodes on which pods are running must be AWS EC2 instances
those instances need to be in the same region and availability-zone as the EBS volume
EBS only supports a single EC2 instance mounting a volume

Create an AWS Elastic Block Storage:

aws ec2 create-volume --region us-west-2 --availability-zone us-west-2a --size 5 --volume-type gp2

aws ec2 create-volume --region us-west-2 --availability-zone us-west-2a --size 5 --volume-type gp2

The region us-west-2 region and us-west-2a availability zone is used here. And so Kubernetes cluster need to start in the same region and availability zone as well.

This shows the output as:

{
    "AvailabilityZone": "us-west-2a", 
    "Encrypted": false, 
    "VolumeType": "gp2", 
    "VolumeId": "vol-47f59cce", 
    "State": "creating", 
    "Iops": 100, 
    "SnapshotId": "", 
    "CreateTime": "2016-07-29T21:57:43.343Z", 
    "Size": 5
}

{

"AvailabilityZone": "us-west-2a",

"Encrypted": false,

"VolumeType": "gp2",

"VolumeId": "vol-47f59cce",

"State": "creating",

"Iops": 100,

"SnapshotId": "",

"CreateTime": "2016-07-29T21:57:43.343Z",

"Size": 5

}

Check if the volume is available as:

aws --region us-west-2 ec2 describe-volumes --volume-id vol-47f59cce

aws --region us-west-2 ec2 describe-volumes --volume-id vol-47f59cce

It shows the output as:

{
    "Volumes": [
        {
            "AvailabilityZone": "us-west-2a", 
            "Attachments": [], 
            "Encrypted": false, 
            "VolumeType": "gp2", 
            "VolumeId": "vol-47f59cce", 
            "State": "available", 
            "Iops": 100, 
            "SnapshotId": "", 
            "CreateTime": "2016-07-29T21:57:43.343Z", 
            "Size": 5
        }
    ]
}

{

"Volumes": [

{

"AvailabilityZone": "us-west-2a",

"Attachments": [],

"Encrypted": false,

"VolumeType": "gp2",

"VolumeId": "vol-47f59cce",

"State": "available",

"Iops": 100,

"SnapshotId": "",

"CreateTime": "2016-07-29T21:57:43.343Z",

"Size": 5

}

]

}

Note the unique identifier for the volume in VolumeId attribute. You can also verify the EBS block in AWS Console:

Start Kubernetes Cluster

Download Kubernetes 1.3.3, untar it and start the cluster on Amazon:

export KUBERNETES_PROVIDER=aws
KUBE_AWS_ZONE=us-west-2a NODE_SIZE=m3.large NUM_NODES=3 ./kubernetes/cluster/kube-up.sh

export KUBERNETES_PROVIDER=aws

KUBE_AWS_ZONE=us-west-2a NODE_SIZE=m3.large NUM_NODES=3 ./kubernetes/cluster/kube-up.sh

Three points to note here:

Zone in which the cluster is started is explicitly set to us-west-1a. This matches the zone where EBS storage volume was created.
By default, each node size is m3.medium. Here is is set to m3.large.
By default, 1 master and 4 worker nodes are created. Here only 3 worker nodes are created.

This will show the output as:

... Starting cluster in us-west-2a using provider aws
... calling verify-prereqs
... calling kube-up
Starting cluster using os distro: jessie
Uploading to Amazon S3
+++ Staging server tars to S3 Storage: kubernetes-staging-0eaf81fbc51209dd47c13b6d8b424149/devel
upload: ../../../../../var/folders/81/ttv4n16x7p390cttrm_675y00000gn/T/kubernetes.XXXXXX.ISohbaGM/s3/bootstrap-script to s3://kubernetes-staging-0eaf81fbc51209dd47c13b6d8b424149/devel/bootstrap-script
Uploaded server tars:
  SERVER_BINARY_TAR_URL: https://s3.amazonaws.com/kubernetes-staging-0eaf81fbc51209dd47c13b6d8b424149/devel/kubernetes-server-linux-amd64.tar.gz
  SALT_TAR_URL: https://s3.amazonaws.com/kubernetes-staging-0eaf81fbc51209dd47c13b6d8b424149/devel/kubernetes-salt.tar.gz
  BOOTSTRAP_SCRIPT_URL: https://s3.amazonaws.com/kubernetes-staging-0eaf81fbc51209dd47c13b6d8b424149/devel/bootstrap-script
INSTANCEPROFILE arn:aws:iam::598307997273:instance-profile/kubernetes-master    2016-07-29T15:13:35Z    AIPAJF3XKLNKOXOTQOCTkubernetes-master   /
ROLES   arn:aws:iam::598307997273:role/kubernetes-master        2016-07-29T15:13:33Z    /       AROAI3Q2KFBD5PCKRXCRM   kubernetes-master
ASSUMEROLEPOLICYDOCUMENT        2012-10-17
STATEMENT       sts:AssumeRole  Allow
PRINCIPAL       ec2.amazonaws.com
INSTANCEPROFILE arn:aws:iam::598307997273:instance-profile/kubernetes-minion    2016-07-29T15:13:39Z    AIPAIYSH5DJA4UPQIP4Bkubernetes-minion   /
ROLES   arn:aws:iam::598307997273:role/kubernetes-minion        2016-07-29T15:13:37Z    /       AROAIQ57MPQYSHRPQCT2Q   kubernetes-minion
ASSUMEROLEPOLICYDOCUMENT        2012-10-17
STATEMENT       sts:AssumeRole  Allow
PRINCIPAL       ec2.amazonaws.com
Using SSH key with (AWS) fingerprint: SHA256:dX/5wpWuUxYar2NFuGwiZuRiydiZCyx4DGoZ5/jL/j8
Creating vpc.
Adding tag to vpc-fa3d6c9e: Name=kubernetes-vpc
Adding tag to vpc-fa3d6c9e: KubernetesCluster=kubernetes
Using VPC vpc-fa3d6c9e
Adding tag to dopt-3aad625e: Name=kubernetes-dhcp-option-set
Adding tag to dopt-3aad625e: KubernetesCluster=kubernetes
Using DHCP option set dopt-3aad625e
Creating subnet.
Adding tag to subnet-e11f5985: KubernetesCluster=kubernetes
Using subnet subnet-e11f5985
Creating Internet Gateway.
Using Internet Gateway igw-5c748f38
Associating route table.
Creating route table
Adding tag to rtb-84fcf1e0: KubernetesCluster=kubernetes
Associating route table rtb-84fcf1e0 to subnet subnet-e11f5985
Adding route to route table rtb-84fcf1e0
Using Route Table rtb-84fcf1e0
Creating master security group.
Creating security group kubernetes-master-kubernetes.
Adding tag to sg-91590bf7: KubernetesCluster=kubernetes
Creating minion security group.
Creating security group kubernetes-minion-kubernetes.
Adding tag to sg-9d590bfb: KubernetesCluster=kubernetes
Using master security group: kubernetes-master-kubernetes sg-91590bf7
Using minion security group: kubernetes-minion-kubernetes sg-9d590bfb
Creating master disk: size 20GB, type gp2
Adding tag to vol-def79e57: Name=kubernetes-master-pd
Adding tag to vol-def79e57: KubernetesCluster=kubernetes
Allocated Elastic IP for master: 52.40.216.69
Adding tag to vol-def79e57: kubernetes.io/master-ip=52.40.216.69
Generating certs for alternate-names: IP:52.40.216.69,IP:172.20.0.9,IP:10.0.0.1,DNS:kubernetes,DNS:kubernetes.default,DNS:kubernetes.default.svc,DNS:kubernetes.default.svc.cluster.local,DNS:kubernetes-master
Starting Master
Adding tag to i-5a7cebf5: Name=kubernetes-master
Adding tag to i-5a7cebf5: Role=kubernetes-master
Adding tag to i-5a7cebf5: KubernetesCluster=kubernetes
Waiting for master to be ready
Attempt 1 to check for master nodeWaiting for instance i-5a7cebf5 to be running (currently pending)
Sleeping for 3 seconds...
Waiting for instance i-5a7cebf5 to be running (currently pending)
Sleeping for 3 seconds...
Waiting for instance i-5a7cebf5 to be running (currently pending)
Sleeping for 3 seconds...
Waiting for instance i-5a7cebf5 to be running (currently pending)
Sleeping for 3 seconds...
 [master running]
Attaching IP 52.40.216.69 to instance i-5a7cebf5
Attaching persistent data volume (vol-def79e57) to master
2016-07-29T22:00:36.909Z        /dev/sdb        i-5a7cebf5      attaching       vol-def79e57
cluster "aws_kubernetes" set.
user "aws_kubernetes" set.
context "aws_kubernetes" set.
switched to context "aws_kubernetes".
user "aws_kubernetes-basic-auth" set.
Wrote config for aws_kubernetes to /Users/arungupta/.kube/config
Creating minion configuration
Creating autoscaling group
 0 minions started; waiting
 0 minions started; waiting
 0 minions started; waiting
 0 minions started; waiting
 3 minions started; ready
Waiting for cluster initialization.

  This will continually check to see if the API for kubernetes is reachable.
  This might loop forever if there was some uncaught error during start
  up.

..........................................................................................................................................................................................................Kubernetes cluster created.
Sanity checking cluster...
Attempt 1 to check Docker on node @ 52.42.0.65 ...not working yet
Attempt 2 to check Docker on node @ 52.42.0.65 ...not working yet
Attempt 3 to check Docker on node @ 52.42.0.65 ...working
Attempt 1 to check Docker on node @ 52.36.195.201 ...working
Attempt 1 to check Docker on node @ 52.43.35.173 ...working

Kubernetes cluster is running.  The master is running at:

  https://52.40.216.69

The user name and password to use is located in /Users/arungupta/.kube/config.

... calling validate-cluster
Found 3 node(s).
NAME                                        STATUS    AGE
ip-172-20-0-26.us-west-2.compute.internal   Ready     1m
ip-172-20-0-27.us-west-2.compute.internal   Ready     1m
ip-172-20-0-28.us-west-2.compute.internal   Ready     1m
Validate output:
NAME                 STATUS    MESSAGE              ERROR
controller-manager   Healthy   ok                   
scheduler            Healthy   ok                   
etcd-0               Healthy   {"health": "true"}   
etcd-1               Healthy   {"health": "true"}   
Cluster validation succeeded
Done, listing cluster services:

Kubernetes master is running at https://52.40.216.69
Elasticsearch is running at https://52.40.216.69/api/v1/proxy/namespaces/kube-system/services/elasticsearch-logging
Heapster is running at https://52.40.216.69/api/v1/proxy/namespaces/kube-system/services/heapster
Kibana is running at https://52.40.216.69/api/v1/proxy/namespaces/kube-system/services/kibana-logging
KubeDNS is running at https://52.40.216.69/api/v1/proxy/namespaces/kube-system/services/kube-dns
kubernetes-dashboard is running at https://52.40.216.69/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard
Grafana is running at https://52.40.216.69/api/v1/proxy/namespaces/kube-system/services/monitoring-grafana
InfluxDB is running at https://52.40.216.69/api/v1/proxy/namespaces/kube-system/services/monitoring-influxdb

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

... Starting cluster in us-west-2a using provider aws

... calling verify-prereqs

... calling kube-up

Starting cluster using os distro: jessie

Uploading to Amazon S3

+++ Staging server tars to S3 Storage: kubernetes-staging-0eaf81fbc51209dd47c13b6d8b424149/devel

upload: ../../../../../var/folders/81/ttv4n16x7p390cttrm_675y00000gn/T/kubernetes.XXXXXX.ISohbaGM/s3/bootstrap-script to s3://kubernetes-staging-0eaf81fbc51209dd47c13b6d8b424149/devel/bootstrap-script

Uploaded server tars:

SERVER_BINARY_TAR_URL: https://s3.amazonaws.com/kubernetes-staging-0eaf81fbc51209dd47c13b6d8b424149/devel/kubernetes-server-linux-amd64.tar.gz

SALT_TAR_URL: https://s3.amazonaws.com/kubernetes-staging-0eaf81fbc51209dd47c13b6d8b424149/devel/kubernetes-salt.tar.gz

BOOTSTRAP_SCRIPT_URL: https://s3.amazonaws.com/kubernetes-staging-0eaf81fbc51209dd47c13b6d8b424149/devel/bootstrap-script

INSTANCEPROFILE arn:aws:iam::598307997273:instance-profile/kubernetes-master 2016-07-29T15:13:35Z AIPAJF3XKLNKOXOTQOCTkubernetes-master /

ROLES arn:aws:iam::598307997273:role/kubernetes-master 2016-07-29T15:13:33Z / AROAI3Q2KFBD5PCKRXCRM kubernetes-master

ASSUMEROLEPOLICYDOCUMENT 2012-10-17

STATEMENT sts:AssumeRole Allow

PRINCIPAL ec2.amazonaws.com

INSTANCEPROFILE arn:aws:iam::598307997273:instance-profile/kubernetes-minion 2016-07-29T15:13:39Z AIPAIYSH5DJA4UPQIP4Bkubernetes-minion /

ROLES arn:aws:iam::598307997273:role/kubernetes-minion 2016-07-29T15:13:37Z / AROAIQ57MPQYSHRPQCT2Q kubernetes-minion

ASSUMEROLEPOLICYDOCUMENT 2012-10-17

STATEMENT sts:AssumeRole Allow

PRINCIPAL ec2.amazonaws.com

Using SSH key with (AWS) fingerprint: SHA256:dX/5wpWuUxYar2NFuGwiZuRiydiZCyx4DGoZ5/jL/j8

Creating vpc.

Adding tag to vpc-fa3d6c9e: Name=kubernetes-vpc

Adding tag to vpc-fa3d6c9e: KubernetesCluster=kubernetes

Using VPC vpc-fa3d6c9e

Adding tag to dopt-3aad625e: Name=kubernetes-dhcp-option-set

Adding tag to dopt-3aad625e: KubernetesCluster=kubernetes

Using DHCP option set dopt-3aad625e

Creating subnet.

Adding tag to subnet-e11f5985: KubernetesCluster=kubernetes

Using subnet subnet-e11f5985

Creating Internet Gateway.

Using Internet Gateway igw-5c748f38

Associating route table.

Creating route table

Adding tag to rtb-84fcf1e0: KubernetesCluster=kubernetes

Associating route table rtb-84fcf1e0 to subnet subnet-e11f5985

Adding route to route table rtb-84fcf1e0

Using Route Table rtb-84fcf1e0

Creating master security group.

Creating security group kubernetes-master-kubernetes.

Adding tag to sg-91590bf7: KubernetesCluster=kubernetes

Creating minion security group.

Creating security group kubernetes-minion-kubernetes.

Adding tag to sg-9d590bfb: KubernetesCluster=kubernetes

Using master security group: kubernetes-master-kubernetes sg-91590bf7

Using minion security group: kubernetes-minion-kubernetes sg-9d590bfb

Creating master disk: size 20GB, type gp2

Adding tag to vol-def79e57: Name=kubernetes-master-pd

Adding tag to vol-def79e57: KubernetesCluster=kubernetes

Allocated Elastic IP for master: 52.40.216.69

Adding tag to vol-def79e57: kubernetes.io/master-ip=52.40.216.69

Generating certs for alternate-names: IP:52.40.216.69,IP:172.20.0.9,IP:10.0.0.1,DNS:kubernetes,DNS:kubernetes.default,DNS:kubernetes.default.svc,DNS:kubernetes.default.svc.cluster.local,DNS:kubernetes-master

Starting Master

Adding tag to i-5a7cebf5: Name=kubernetes-master

Adding tag to i-5a7cebf5: Role=kubernetes-master

Adding tag to i-5a7cebf5: KubernetesCluster=kubernetes

Waiting for master to be ready

Attempt 1 to check for master nodeWaiting for instance i-5a7cebf5 to be running (currently pending)

Sleeping for 3 seconds...

Waiting for instance i-5a7cebf5 to be running (currently pending)

Sleeping for 3 seconds...

Waiting for instance i-5a7cebf5 to be running (currently pending)

Sleeping for 3 seconds...

Waiting for instance i-5a7cebf5 to be running (currently pending)

Sleeping for 3 seconds...

[master running]

Attaching IP 52.40.216.69 to instance i-5a7cebf5

Attaching persistent data volume (vol-def79e57) to master

2016-07-29T22:00:36.909Z /dev/sdb i-5a7cebf5 attaching vol-def79e57

cluster "aws_kubernetes" set.

user "aws_kubernetes" set.

context "aws_kubernetes" set.

switched to context "aws_kubernetes".

user "aws_kubernetes-basic-auth" set.

Wrote config for aws_kubernetes to /Users/arungupta/.kube/config

Creating minion configuration

Creating autoscaling group

0 minions started; waiting

3 minions started; ready

Waiting for cluster initialization.

This will continually check to see if the API for kubernetes is reachable.

This might loop forever if there was some uncaught error during start

up.

..........................................................................................................................................................................................................Kubernetes cluster created.

Sanity checking cluster...

Attempt 1 to check Docker on node @ 52.42.0.65 ...not working yet

Attempt 2 to check Docker on node @ 52.42.0.65 ...not working yet

Attempt 3 to check Docker on node @ 52.42.0.65 ...working

Attempt 1 to check Docker on node @ 52.36.195.201 ...working

Attempt 1 to check Docker on node @ 52.43.35.173 ...working

Kubernetes cluster is running. The master is running at:

https://52.40.216.69

The user name and password to use is located in /Users/arungupta/.kube/config.

... calling validate-cluster

Found 3 node(s).

NAME STATUS AGE

ip-172-20-0-26.us-west-2.compute.internal Ready 1m

ip-172-20-0-27.us-west-2.compute.internal Ready 1m

ip-172-20-0-28.us-west-2.compute.internal Ready 1m

Validate output:

NAME STATUS MESSAGE ERROR

controller-manager Healthy ok

scheduler Healthy ok

etcd-0 Healthy {"health": "true"}

etcd-1 Healthy {"health": "true"}

Cluster validation succeeded

Done, listing cluster services:

Kubernetes master is running at https://52.40.216.69

Elasticsearch is running at https://52.40.216.69/api/v1/proxy/namespaces/kube-system/services/elasticsearch-logging

Heapster is running at https://52.40.216.69/api/v1/proxy/namespaces/kube-system/services/heapster

Kibana is running at https://52.40.216.69/api/v1/proxy/namespaces/kube-system/services/kibana-logging

KubeDNS is running at https://52.40.216.69/api/v1/proxy/namespaces/kube-system/services/kube-dns

kubernetes-dashboard is running at https://52.40.216.69/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard

Grafana is running at https://52.40.216.69/api/v1/proxy/namespaces/kube-system/services/monitoring-grafana

InfluxDB is running at https://52.40.216.69/api/v1/proxy/namespaces/kube-system/services/monitoring-influxdb

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

Read more details about starting a Kubernetes cluster on Amazon.

Couchbase Pod w/o Persistent Storage

Let’s create a Couchbase pod without persistent storage. This means that if the pod is rescheduled on a different host then it will not have access to the data created on it.

Here are quick steps to run a Couchbase pod and expose it outside the cluster:

kubectl.sh run couchbase --image=arungupta/couchbase
kubectl.sh expose deployment couchbase --target-port=8091 --port=8091 --type=LoadBalancer
kubectl.sh describe svc couchbase

kubectl.sh run couchbase --image=arungupta/couchbase

kubectl.sh expose deployment couchbase --target-port=8091 --port=8091 --type=LoadBalancer

kubectl.sh describe svc couchbase

Read more details at Kubernetes cluster at Amazon.

The last command shows the ingress load balancer address. Access Couchbase Web Console at <ip>:8091.

kubernetes-pv-couchbase-amazon-elb

The main page of Couchbase Web Console shows up:

kubernetes-pv-couchbase-amazon-web-console

A default travel-sample bucket is already created by arungupta/couchbase image. This bucket is shown in the Data Buckets tab:

Click on Create New Data Bucket button to create a new data bucket. Give it a name k8s, take all the defaults, and click on Create button to create the bucket:

Created bucket is shown in the Data Buckets tab:

Check status of the pod:

kubectl.sh get po
NAME                         READY     STATUS    RESTARTS   AGE
couchbase-2646907196-memz2   1/1       Running   0          53m

kubectl.sh get po

NAME READY STATUS RESTARTS AGE

couchbase-2646907196-memz2 1/1 Running 0 53m

Delete the pod:

kubectl.sh delete po couchbase-2646907196-memz2
pod "couchbase-2646907196-memz2" deleted

kubectl.sh delete po couchbase-2646907196-memz2

pod "couchbase-2646907196-memz2" deleted

Watch the new pod being created:

kubectl.sh get -w po
NAME                         READY     STATUS        RESTARTS   AGE
couchbase-2646907196-memz2   1/1       Terminating   0          53m
couchbase-2646907196-wo6ve   1/1       Running       0          3s

kubectl.sh get -w po

NAME READY STATUS RESTARTS AGE

couchbase-2646907196-memz2 1/1 Terminating 0 53m

couchbase-2646907196-wo6ve 1/1 Running 0 3s

Access the Web Console again and see that the bucket does not exist:

Let’s clean up the resources created:

kubectl.sh delete svc couchbase
kubectl.sh delete deployment couchbase

kubectl.sh delete svc couchbase

kubectl.sh delete deployment couchbase

Couchbase Pod with Persistent Storage

Now, lets expose a Couchbase pod with persistent storage. As discussed above, lets create a PersistentVolume and claim the volume.

Request storage

Like any other Kubernetes resources, a persistent volume is created by using a resource description file:

kind: PersistentVolume
apiVersion: v1
metadata:
  name: couchbase-pv
  labels:
    type: amazonEBS
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  awsElasticBlockStore:
    volumeID: vol-47f59cce
    fsType: ext4

kind: PersistentVolume

apiVersion: v1

metadata:

labels:

type: amazonEBS

spec:

capacity:

storage: 5Gi

accessModes:

- ReadWriteOnce

awsElasticBlockStore:

volumeID: vol-47f59cce

fsType: ext4

The important pieces of information here are:

Creating a storage of 5 GB
Storage can be mounted by only one node for reading/writing
specifies the volume id created earlier

Read more details about definition of this file at kubernetes.io/docs/user-guide/persistent-volumes/.

This file is available at: github.com/arun-gupta/couchbase-kubernetes/blob/master/pv/couchbase-pv.yml.

The volume itself can be created as:

kubectl create -f couchbase-pv.yml

kubectl create -f couchbase-pv.yml

and shows the output:

persistentvolume "couchbase-pv" created

persistentvolume "couchbase-pv" created

Use claim

A PersistentVolumeClaim can be created using this resource file:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: couchbase-pvc
  labels:
    type: amazonEBS
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi

kind: PersistentVolumeClaim

apiVersion: v1

metadata:

labels:

type: amazonEBS

spec:

accessModes:

- ReadWriteOnce

resources:

requests:

storage: 5Gi

In our case, both PersistentVolume and PersistentVolumeClaim are 5 GB but they don’t have to be.

Read more details about definition of this file at kubernetes.io/docs/user-guide/persistent-volumes/#persistentvolumeclaims.

This file is at github.com/arun-gupta/couchbase-kubernetes/blob/master/pv/couchbase-pvc.yml.

The claim can be created as:

kubectl create -f couchbase-pvc.yml

kubectl create -f couchbase-pvc.yml

and shows the output:

persistentvolumeclaim "couchbase-pvc" created

persistentvolumeclaim "couchbase-pvc" created

Create RC with Persistent Volume Claim

Create a Couchbase Replication Controller using this resource file:

apiVersion: v1
kind: ReplicationController
metadata:
  name: couchbase
spec:
  replicas: 1
  template:
    metadata:
      name: couchbase-rc-pod
      labels:
        name: couchbase-rc-pod
        context: couchbase-pv
    spec:
      containers:
      - name: couchbase-rc-pod
        image: arungupta/couchbase
        volumeMounts:
        - mountPath: "/opt/couchbase/var"
          name: mypd
        ports:
        - containerPort: 8091
        - containerPort: 8092
        - containerPort: 8093
        - containerPort: 11210
      volumes:
      - name: mypd
        persistentVolumeClaim:
          claimName: couchbase-pvc

apiVersion: v1

kind: ReplicationController

metadata:

spec:

replicas: 1

template:

metadata:

labels:

context: couchbase-pv

spec:

containers:

- name: couchbase-rc-pod

image: arungupta/couchbase

volumeMounts:

- mountPath: "/opt/couchbase/var"

ports:

- containerPort: 8091

- containerPort: 8092

- containerPort: 8093

- containerPort: 11210

volumes:

- name: mypd

persistentVolumeClaim:

claimName: couchbase-pvc

Key parts here are:

Resource defines a Replication Controller using arungupta/couchbase Docker image
volumeMounts define which volumes are going to be mounted. /opt/couchbase/var is the directory where Couchbase stores all the data.
volumes define different volumes that can be used in this RC definition

Create the RC as:

kubectl create -f couchbase-rc.yml

kubectl create -f couchbase-rc.yml

and shows the output:

replicationcontroller "couchbase" created

replicationcontroller "couchbase" created

Check for pod as kubectl.sh get -w po to see:

kubectl.sh get -w po
NAME              READY     STATUS              RESTARTS   AGE
couchbase-jx3fn   0/1       ContainerCreating   0          3s
NAME              READY     STATUS    RESTARTS   AGE
couchbase-jx3fn   1/1       Running   0          20s

kubectl.sh get -w po

NAME READY STATUS RESTARTS AGE

couchbase-jx3fn 0/1 ContainerCreating 0 3s

NAME READY STATUS RESTARTS AGE

couchbase-jx3fn 1/1 Running 0 20s

Expose RC as a service:

kubectl.sh expose rc couchbase --target-port=8091 --port=809--type=LoadBalancer
service "couchbase" exposed

kubectl.sh expose rc couchbase --target-port=8091 --port=809--type=LoadBalancer

service "couchbase" exposed

Get all the services:

kubectl.sh get svc
NAME         CLUSTER-IP    EXTERNAL-IP        PORT(S)    AGE
couchbase    10.0.49.129   a6179426155e2...   8091/TCP   19s
kubernetes   10.0.0.1      <none>             443/TCP    1h

kubectl.sh get svc

NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE

couchbase 10.0.49.129 a6179426155e2... 8091/TCP 19s

kubernetes 10.0.0.1 <none> 443/TCP 1h

Describe the service as kubectl.sh describe svc couchbase to see:

Name:                   couchbase
Namespace:              default
Labels:                 context=couchbase-pv
                        name=couchbase-pod
Selector:               context=couchbase-pv,name=couchbase-pod
Type:                   LoadBalancer
IP:                     10.0.49.129
LoadBalancer Ingress:   a6179426155e211e6b664022b850255f-1850736155.us-west-2.elb.amazonaws.com
Port:                   <unset>   8091/TCP
NodePort:               <unset>   31636/TCP
Endpoints:              10.244.1.3:8091
Session Affinity:       None
Events:
  FirstSeen     LastSeen        Count   From                    SubobjectPath   Type            Reason                  Message
  ---------     --------        -----   ----                    -------------   --------        ------                  -------
  31s           31s             1       {service-controller }                   Normal          CreatingLoadBalancer    Creating load balancer
  29s           29s             1       {service-controller }                   Normal          CreatedLoadBalancer     Created load balancer

Name: couchbase

Namespace: default

Labels: context=couchbase-pv

name=couchbase-pod

Selector: context=couchbase-pv,name=couchbase-pod

Type: LoadBalancer

IP: 10.0.49.129

LoadBalancer Ingress: a6179426155e211e6b664022b850255f-1850736155.us-west-2.elb.amazonaws.com

Port: <unset> 8091/TCP

NodePort: <unset> 31636/TCP

Endpoints: 10.244.1.3:8091

Session Affinity: None

Events:

FirstSeen LastSeen Count From SubobjectPath Type Reason Message

--------- -------- ----- ---- ------------- -------- ------ -------

31s 31s 1 {service-controller } Normal CreatingLoadBalancer Creating load balancer

29s 29s 1 {service-controller } Normal CreatedLoadBalancer Created load balancer

Wait for ~3 mins for the load balancer to settle. Access the Couchbase Web Console at <ingress-lb>:8091. Once again, only travel-sample bucket exists. This is created by arungupta/couchbase image used in the RC definition.

Show Stateful Containers

Lets create a new bucket. Give it a name kubernetes-pv, take all defaults and click on Create button to create the bucket.

The bucket now shows up in the console:

Terminate Couchbase pod and see the state getting restored.

Get the pods again:

kubectl.sh get po
NAME              READY     STATUS    RESTARTS   AGE
couchbase-jx3fn   1/1       Running   0          7m

kubectl.sh get po

NAME READY STATUS RESTARTS AGE

couchbase-jx3fn 1/1 Running 0 7m

Delete the pod:

kubectl.sh delete po couchbase-jx3fn
pod "couchbase-jx3fn" deleted

kubectl.sh delete po couchbase-jx3fn

pod "couchbase-jx3fn" deleted

Pod gets recreated:

kubectl.sh get -w po
NAME              READY     STATUS              RESTARTS   AGE
couchbase-jx3fn   1/1       Terminating         0          8m
couchbase-qq6wu   0/1       ContainerCreating   0          4s
NAME              READY     STATUS    RESTARTS   AGE
couchbase-qq6wu   1/1       Running   0          5s

kubectl.sh get -w po

NAME READY STATUS RESTARTS AGE

couchbase-jx3fn 1/1 Terminating 0 8m

couchbase-qq6wu 0/1 ContainerCreating 0 4s

NAME READY STATUS RESTARTS AGE

couchbase-qq6wu 1/1 Running 0 5s

And now when you access the Couchbase Web Console, the earlier created bucket still exists:

That’s because the data was stored in the backing EBS storage.

Cleanup Kubernetes Cluster

Shutdown Kubernetes cluster:

KUBE_AWS_ZONE=us-west-2a NODE_SIZE=m3.large NUM_NODES=3 ./kubernetes/cluster/kube-down.sh 
Bringing down cluster using provider: aws
Deleting ELBs in: vpc-fa3d6c9e
Waiting for ELBs to be deleted
All ELBs deleted
Deleting instances in VPC: vpc-fa3d6c9e
Deleting auto-scaling group: kubernetes-minion-group-us-west-2a
Deleting auto-scaling launch configuration: kubernetes-minion-group-us-west-2a
Deleting auto-scaling group: kubernetes-minion-group-us-west-2a
Deleting auto-scaling group: kubernetes-minion-group-us-west-2a
Waiting for instances to be deleted
Waiting for instance i-8b7aed24 to be terminated (currently shutting-down)
Sleeping for 3 seconds...
Waiting for instance i-8b7aed24 to be terminated (currently shutting-down)
Sleeping for 3 seconds...
Waiting for instance i-8b7aed24 to be terminated (currently shutting-down)
Sleeping for 3 seconds...
Waiting for instance i-8b7aed24 to be terminated (currently shutting-down)
Sleeping for 3 seconds...
Waiting for instance i-8b7aed24 to be terminated (currently shutting-down)
Sleeping for 3 seconds...
Waiting for instance i-8b7aed24 to be terminated (currently shutting-down)
Sleeping for 3 seconds...
Waiting for instance i-8b7aed24 to be terminated (currently shutting-down)
Sleeping for 3 seconds...
Waiting for instance i-8b7aed24 to be terminated (currently shutting-down)
Sleeping for 3 seconds...
Waiting for instance i-8b7aed24 to be terminated (currently shutting-down)
Sleeping for 3 seconds...
Waiting for instance i-8b7aed24 to be terminated (currently shutting-down)
Sleeping for 3 seconds...
Waiting for instance i-8b7aed24 to be terminated (currently shutting-down)
Sleeping for 3 seconds...
Waiting for instance i-8b7aed24 to be terminated (currently shutting-down)
Sleeping for 3 seconds...
Waiting for instance i-8b7aed24 to be terminated (currently shutting-down)
Sleeping for 3 seconds...
Waiting for instance i-8b7aed24 to be terminated (currently shutting-down)
Sleeping for 3 seconds...
Waiting for instance i-8b7aed24 to be terminated (currently shutting-down)
Sleeping for 3 seconds...
Waiting for instance i-8b7aed24 to be terminated (currently shutting-down)
Sleeping for 3 seconds...
Waiting for instance i-8b7aed24 to be terminated (currently shutting-down)
Sleeping for 3 seconds...
Waiting for instance i-8b7aed24 to be terminated (currently shutting-down)
Sleeping for 3 seconds...
Waiting for instance i-8b7aed24 to be terminated (currently shutting-down)
Sleeping for 3 seconds...
All instances deleted
Releasing Elastic IP: 52.40.216.69
Deleting volume vol-def79e57
Cleaning up resources in VPC: vpc-fa3d6c9e
Cleaning up security group: sg-91590bf7
Cleaning up security group: sg-9d590bfb
Cleaning up security group: sg-e97b298f
Deleting security group: sg-91590bf7
Deleting security group: sg-9d590bfb
Deleting security group: sg-e97b298f
Deleting VPC: vpc-fa3d6c9e
Done

KUBE_AWS_ZONE=us-west-2a NODE_SIZE=m3.large NUM_NODES=3 ./kubernetes/cluster/kube-down.sh

Bringing down cluster using provider: aws

Deleting ELBs in: vpc-fa3d6c9e

Waiting for ELBs to be deleted

All ELBs deleted

Deleting instances in VPC: vpc-fa3d6c9e

Deleting auto-scaling group: kubernetes-minion-group-us-west-2a

Deleting auto-scaling launch configuration: kubernetes-minion-group-us-west-2a

Deleting auto-scaling group: kubernetes-minion-group-us-west-2a

Waiting for instances to be deleted