AWS Serverless Lambda Scheduled Events to Store Tweets in Couchbase

couchbase, , , ,

This blog has explained a few Serverless concepts with code samples:

  • Serverless FaaS with AWS Lambda and Java
  • AWS IoT Button, Lambda and Couchbase
  • Microservice using AWS API Gateway, AWS Lambda and Couchbase
  • Microservice using AWS Serverless Application Model and Couchbase

This particular blog entry will show how to use AWS Lambda to store tweets of a tweeter in Couchbase. Here are the high level components:

 

lambda-twitter-couchbase

The key concepts are:

  • Lambda Function deployed using Serverless Application Model
  • Triggered every 3 hours using Scheduled Events
  • Uses Twitter4J API to query new tweets since the last fetch
  • Use Couchbase Java SDK API to store JSON documents in the Couchbase Server

Complete sample code for this blog is available at github.com/arun-gupta/twitter-n1ql.

Serverless Application Model

Serverless Application Model, or SAM, defines simplified syntax for expressing serverless resources. SAM extends AWS CloudFormation to add support for API Gateway, AWS Lambda and Amazon DynamoDB. Read more details in Microservice using AWS Serverless Application Model and Couchbase.

For our application, SAM template is available at github.com/arun-gupta/twitter-n1ql/blob/master/template-example.yml and shown below:

What do we see here?

  • Function is packaged and available in a S3 bucket
  • Handler class is org.sample.twittter.TwitterRequestHandler and is at github.com/arun-gupta/twitter-n1ql/blob/master/twitter-feed/src/main/java/org/sample/twitter/TwitterRequestHandler.java. It looks like:
    By default, this class reads the twitter handle of Donald Trump. More fun on that coming in a subsequent blog.
  • COUCHBASE_HOST and COUCHBASE_BUCKET_PASSWORD are environment variables that provide EC2 host where Couchbase database is running and the password of the bucket.
  • Function can be triggered by different events. In our case, this is triggered every three hours. More details about the expression used here are at Schedule Expressions Using Rate or Cron.

Fetching Tweets using Twitter4J

Tweets are read using Twitter4J API. It is an unofficial Twitter API that provides a Java abstraction over Twitter REST API. Here is a simple example:

 

Twitter4J Docs and Javadocs are pretty comprehensive.

Twitter API allows to read only last 200 tweets. Lambda function is invoked every 3 hours. The tweet frequency of @realDonaldTrump is not 200 every 3 hours, at least yet. If it does reach that dangerous level then we can adjust the rate to trigger Lambda function more frequently.

JSON representation of each tweet is stored in Couchbase server using Couchbase Java SDK. AWS Lambda supports Node, Python and C#. And so you can use Couchbase Node SDK, Couchbase Python SDK or Couchbase .NET SDK to write these functions as well.

Twitter4J API allows to fetch tweets since the id of a particular tweet. This allows to ensure that duplicate tweets are not fetched. This requires us to sort all tweets in a particular order and then pick the id of the most recent tweet. This was solved using the simple N1QL query:

The syntax is very SQL-like. More on this in a subsequent blog.

Store Tweets in Couchbase

The final item is to store the retrieved tweets in Couchbase.

Value of COUCHABSE_HOST environment variable is used to connect to the Couchbase instance. The value of COUCHBASE_BUCKET_PASSWORD environment variable is to connect to the secure bucket where all JSON documents are stored. Its very critical that the bucket be password protected and not directly specified in the source code. More on this in a subsequent blog.

The JSON document is upserted (insert or update) in Couchbase using the Couchbase Java API:

 

This Lambda Function has been running for a few days now and has captured 258 tweets from @realDonaldTrump.

serverless-lambda-couchbase-twitter-bucket

An interesting analysis of his tweets is coming shortly!

Talk to us:

  • Couchbase Forums
  • Couchbase Database Developer Portal
  • @couchbasedev and @couchbase

Complete sample code for this blog is available at github.com/arun-gupta/twitter-n1ql.

Source: https://blog.couchbase.com/2017/january/aws-serverless-lambda-scheduled-events-tweets-couchbase

Couchbase Customer Advisory Note – Security

couchbase

In light of the recent widespread news about security vulnerabilities in MongoDB and Elasticsearch, we want to proactively remind our customers of Security Best Practices for Couchbase.

At this time there have been no known ransomware attacks on Couchbase, and no new security vulnerabilities have been identified in the product. This advisory is in the spirit of ‘forewarned is forearmed’.

Comprehensive security planning is a complex topic, but getting started with Security Basics is not. This Advisory Note is intended as a heads-up and reminder of general security best practices as well as Couchbase security capabilities available to you. First of all, let’s start with the basics. All Couchbase Server installations should ensure that:

  • Proper physical security (server access and backup storage) is maintained.
  • Couchbase Server nodes are behind a firewall so that they are not publically accessible. Here is how to configure network access to Couchbase using IP tables.
  • The server operating system is up to date with the latest security patches.
  • Delete the “default” bucket.
  • Secure in-transit data by using SSL connections for client/server and server/server communication.
  • Use a strong and unique bucket password for all data buckets.
  • Add security to your Couchbase mobile application
  • Encrypt Couchbase Lite databases

Additionally, customers should consult the following Couchbase resources in order to build a comprehensive security plan:

Documentation

  • Couchbase Server Security Considerations
  • Couchbase Server Introduction to Security
  • Couchbase Server Security Best Practices

Blogs

  • Configuring IPSec
  • N1QL Security
  • Skipping Default Bucket Creation
  • Security for Mobile Data Synchronization
  • Decentralized Security with Couchbase Mobile

As always, please reach out to us if you have any questions.

How to contact?

Source: https://blog.couchbase.com/2017/couchbase-customer-advisory-note-security

Starting a Kubernetes 1.5.x cluster

containers,

Kubernetes

Kubernetes 1.5.0 was released just about a month ago! Key theme for the release are:

  • StatefulSets (ex-PetSets)
    • StatefulSets are beta now (fixes and stabilization)
  • Improved Federation Support
    • New command: kubefed
    • DaemonSets
    • Deployments
    • ConfigMaps
  • Simplified Cluster Deployment
    • Improvements to kubeadm
    • HA Setup for Master
  • Node Robustness and Extensibility
    • Windows Server Container support
    • CRI for pluggable container runtimes
    • kubelet API supports authentication and authorization

Read CHANGELOG for complete details.

Up until 1.5.0, starting up a Kubernetes cluster on Amazon Web Services was pretty straight forward.

But with 1.5.0 and 1.5.1, the command fails with the error:

What happened?

Basically, Kubernetes binaries was getting bigger than 1GB. The binary was broken into a basic install bundle and client and server binaries. The updated installation process requires to download the basic install bundle of of 4.57 MB (yes, MB instead of GB). It includes cluster scripts like kubectl, kube-up.sh and kube-down.sh, examples, docs and other scripts. This then downloads client and server binaries. Server binary is the base image that is used to start EC2 instances. But instead of automating the download of binaries, somebody decided to add a README in the server directory.

This was a big user experience change, and no links in the README bundled with the release or the release blog. Ouch!

Anyway, this was filed as #38728 and fixed promptly. But it missed the 1.5.1 release and now finally showed up in the 1.5.2 release today.

So, how do you run a Kubernetes 1.5.2 cluster on AWS?

It is more seamlessly integrated now but you need to hit Enter key a couple of times to accept the default value:

After the usual Kubernetes cluster is created, the output is shown as:

Even though your Kubernetes cluster on AWS starts up fine, but kube-up.sh script is going to be deprecated soon. The recommended way is to use Kubernetes Cluster on Amazon using Kops.

Now that your Kubernetes cluster is up, what do you do next?

  • Follow the detailed steps for Kubernetes for Java Developers workshop.
  • Run a Couchbase cluster in Kubernetes
  • Learn more about Couchbase cluster in Containers

Source: https://blog.couchbase.com/2017/january/starting-kubernetes-1.5.x-cluster

Microservice using AWS Serverless Application Model and Couchbase

couchbase, techtip, , , , ,

Amazon Web Services introduced Serverless Application Model, or SAM, a couple of months ago. It defines simplified syntax for expressing serverless resources. SAM extends AWS CloudFormation to add support for API Gateway, AWS Lambda and Amazon DynamoDB. This blog will show how to create a simple microservice using SAM. Of course, we’ll use Couchbase instead of DynamoDB!

This blog will also use the basic concepts explained in Microservice using AWS API Gateway, AWS Lambda and Couchbase. SAM will show the ease with which the entire stack for microservice can be deployed and managed.

As a refresher, here are key components in the architecture:

serverless-microservice

  • Client could be curl, AWS CLI/Console, Postman client or any other tool/API that can invoke a REST endpoint.
  • AWS API Gateway is used to provision APIs. The top level resource is available at path /books. HTTP GET and POST methods are published for the resource.
  • Each API triggers a Lambda function. Two Lambda functions are created, book-list function for listing all the books available and book-create function to create a new book.
  • Couchbase is used as a persistence store in EC2. All the JSON documents are stored and retrieved from this database.

Other blogs on serverless:

  • Microservice using AWS API Gateway, AWS Lambda and Couchbase
  • AWS IoT Button, Lambda and Couchbase
  • Serverless FaaS with Lambda and Java

Let’s get started!

Serverless Application Model (SAM) Template

An AWS CloudFormation template with serverless resources conforming to the AWS SAM model is referred to as a SAM file or template. It is deployed as a CloudFormation stack.

Let’s take a look at our SAM template:

This template is available at github.com/arun-gupta/serverless/blob/master/aws/microservice/template.yml.

SAM template Specification provide complete details about contents in the template. The key parts of the template are:

  • Defines two resources, both of Lambda Function type identified by AWS::Serverless::Function attribute. Name of the Lambda function is defined by Resources.<resource>.
  • Class for each handler is defined by the value of Resources.<resource>.Properties.Handler attribute
  • Java 8 runtime is used to run the Function defined by Resources.<resource>.Properties.Runtime attribute
  • Code for the class is uploaded to an S3 bucket, in our case to s3://serverless-microservice/microservice-http-endpoint-1.0-SNAPSHOT.jar
  • Resources.<resource>.Properties.Environment.Variables.COUCHBASE_HOST attribute value defines the host where Couchbase is running. This can be easily deployed on EC2 as explained at Setup Couchbase.
  • Each Lambda function is triggered by an API. It is deployed using AWS API Gateway. The path is defined by Events.GetResource.Properties.Path. HTTP method is defined using Events.GetResource.Properties.Method attribute.

Java Application

The Java application that contains the Lambda functions is at github.com/arun-gupta/serverless/tree/master/aws/microservice/microservice-http-endpoint.

Lambda function that is triggered by HTTP GET method is shown:

A little bit of explanation:

  • Each Lambda function needs to implement the interface com.amazonaws.services.lambda.runtime.RequestHandler.
  • API Gateway and Lambda integration require a specific input format and output format. These formats are defined as GatewayRequest and GatewayResponse classes.
  • Function logic uses Couchbase Java SDK to query the Couchbase database. N1QL query is used to query the database. The results and exception are then wrapped in GatewayRequest and GatewayResponse.

Lambda function triggered by HTTP POST method is pretty straightforward as well:

A bit of explanation:

  • Incoming request payload is retrieved from GatewayRequest
  • Document inserted in Couchbase is returned as response.
  • Like the previous method, Function logic uses Couchbase Java SDK to query the Couchbase database. The results and exception are then wrapped in GatewayRequest and GatewayResponse.

Build the Java application as:

Upload Lambda Function to S3

SAM template reads the code from an S3 bucket. Let’s create a S3 bucket:

us-west-2 region is one of the supported regions for API Gateway. S3 bucket names are globally unique but their location is region specific.

Upload the code to S3 bucket:

The code is now uploaded to S3 bucket. SAM template is ready to be deployed!

Deploy SAM Template

Deploy the SAM template:

It shows the output:

This one command deploys Lambda functions and REST Resource/APIs that trigger these Lambda functions.

Invoke the Microservice

API Gateway publishes a REST API that can be invoked by curl, wget, AWS CLI/Console, Postman or any other app that can call a REST API. This blog will use AWS Console to show the interaction.

API Gateway home at us-west-2.console.aws.amazon.com/apigateway/home?region=us-west-2#/apis shows:

AWS SAM Microservice API

Click on the API to see all the APIs in this resource:

AWS SAM Microservice API Resources

Click on POST to see the default page for POST method execution:

AWS SAM Microservice API POST

Click on Test to test the API:

AWS SAM Microservice API POST Input

Add the payload in Request Body and click on Test to invoke the API. The results are shown as below:

AWS SAM Microservice API POST Output

Now click on GET to see the default execution page:

AWS SAM Microservice API GET

Click on Test to test the API:

AWS SAM Microservice API GET Input

No request body is needed, just click on Test the invoke the API. The results are as shown:

AWS SAM Microservice API GET Output

Output from the Couchbase database is shown in the Response Body.

References

  • Deploying Lambda-based Applications
  • Serverless Architectures
  • AWS API Gateway
  • Creating a simple Microservice using Lambda and API Gateway
  • Couchbase Server Docs
  • Couchbase Forums
  • Follow us at @couchbasedev

Source: blog.couchbase.com/2017/january/microservice-aws-serverless-application-model-couchbase

Microservice using AWS API Gateway, AWS Lambda and Couchbase

couchbase, , , , , ,

This blog has explained the following concepts for serverless applications so far:

  • Serverless FaaS with AWS Lambda and Java
  • AWS IoT Button, Lambda and Couchbase

The third blog in serverless series will explain how to create a simple microservice using Amazon API Gateway, AWS Lambda and Couchbase.

Read previous blogs for more context on AWS Lambda.

Amazon API Gateway is a fully managed service that makes it easy for developers to create, publish, maintain, monitor, and secure APIs at any scale. Amazon API Gateway handles all the tasks involved in accepting and processing up to hundreds of thousands of concurrent API calls, including traffic management, authorization and access control, monitoring, and API version management.

Here are the key components in this architecture:

serverless-microservice

  • Client could be curl, AWS CLI, Postman client or any other tool/API that can invoke a REST endpoint.
  • API Gateway is used to provision APIs. The top level resource is available at path /books. HTTP GET and POST methods are published for the resource.
  • Each API triggers a Lambda function. Two Lambda functions are created, book-list function for listing all the books available and book-create function to create a new book.
  • Couchbase is used as a persistence store in EC2. All the JSON documents are stored and retrieved from this database.

Let’s get started!

Create IAM Role

IAM roles will have policies and trust relationships that will allow this role to be used in API Gateway and execute Lambda function.

Let’s create a new IAM role:

--assume-role-policy-document defines the trust relationship policy document that grants an entity permission to assume the role. trust.json is at github.com/arun-gupta/serverless/blob/master/aws/microservice/trust.json and looks like:

This trust relationship allows Lambda functions and API Gateway to assume this role during execution.

Associate policies with this role as:

policy.json is at github.com/arun-gupta/serverless/blob/master/aws/microservice/policy.json and looks like:

This generous policy allows any permissions over logs generated in CloudWatch for all resources. In addition it allows all Lambda and API Gateway permissions to all resources. In general, only required policy would be given to specific resources.

Create Lambda Functions

Detailed steps to create Lambda functions are explained in Serverless FaaS with AWS Lambda and Java. Let’s create the two Lambda functions as required in our case:

Couple of key items to note in this function are:

  • IAM role microserviceRole created in previous step is explicitly specified here
  • Handler is org.sample.serverless.aws.couchbase.BucketGetAll class. This class queries the Couchbase database defined using the COUCHBASE_HOST environment variable.

Create the second Lambda function:

The handler for this function is org.sample.serverless.aws.couchbase.BucketPost class. This class creates a new JSON document in the Couchbase database identified by COUCHBASE_HOST environment variable.

The complete source code for these classes is at github.com/arun-gupta/serverless/tree/master/aws/microservice/microservice-http-endpoint.

API Gateway Resource

Create an API using Amazon API Gateway and Test It and Build an API to Expose a Lambda Function provide detailed steps and explanation on how to use API Gateway and Lambda Functions to build powerful backend systems. This blog will do a quick run down of the steps in case you want to cut the chase.

Let’s create API Gateway resources.

  1. The first step is to create an API:
    This shows the output as:
    The value of id attribute is API ID. In our case, this is lb2qgujjif.
  2. Find ROOT ID of the created API as this is required for the next AWS CLI invocation:
    This shows the output:
    Value of id attribute is ROOT ID. This is also the PARENT ID for the top level resource.
  3. Create a resource
    This shows the output:
    Value of id attribute is RESOURCE ID.

API ID and RESOURCE ID are used for subsequent AWS CLI invocations.

API Gateway POST Method

Now that the resource is created, let’s create HTTP POST method on this resource.

  1. Create a POST method
    to see the response:
  2. Set Lambda function as destination of the POST method:
    Make sure to replace <act-id> with your AWS account id. API ID and RESOURCE ID from previous section are used here as well. --uri is used to specify the URI of integration input. The format of the URI is fixed. This CLI will show the result as:
  3. Set content-type of POST method response:
    to see the response:
  4. Set content-type of POST method integration response:
    to see the response:
  5. Deploy the API
    to see the response
  6. Grant permission to allow API Gateway to invoke Lambda Function:
    Also, grant permission to the deployed API:
  7. Test the API method:
    to see the response:
    Value of status attribute is 200 and indicates this was a successful invocation. Value of log attribute shows the log statement from CloudWatch Logs. Detailed logs can also be obtained using aws logs filter-log-events --log-group /aws/lambda/MicroservicePost.
  8. This command stores a single JSON document in Couchbase. This can be easily verified using the Couchbase CLI Tool cbq.Connect to the Couchbase server as:
    Create a primary index on default bucket as this is required to query the bucket with no clauses:
  9. Write a N1QL query to access the data:
    The results show the JSON document that was stored by our Lambda function.

API Gateway GET Method

Let’s create HTTP GET method on the resource:

  1. Create a GET method:
  2. Set correct Lambda function as destination of GET:
  3. Set content-type of GET method response:
  4. Set content-type of GET method integration response:
  5. Grant permission to allow API Gateway to invoke Lambda Function
  6. Grant permission to the deployed API:
  7. Test the method:
    to see the output:
    Once again, 200 status code shows a successful invocation. Detailed logs can be obtained using aws logs filter-log-events --log-group /aws/lambda/MicroservicePost.

This blog only shows one simple POST and GET methods. Other HTTP methods can be very easily included in this microservice as well.

API Gateway and Lambda References

  • Serverless Architectures
  • AWS API Gateway
  • Creating a simple Microservice using Lambda and API Gateway
  • Couchbase Server Docs
  • Couchbase Forums
  • Follow us at @couchbasedev

Source: blog.couchbase.com/2016/december/microservice-aws-api-gateway-lambda-couchbase

AWS IoT Button, Lambda and Couchbase

couchbase, techtip, , ,

Getting Started with Serverless FaaS and AWS Lambda shows how to use a simple Java function to store a JSON document to Couchbase using AWS Lambda. This blog builds upon that and shows how an AWS IoT Button can be used as a trigger for that Lambda function.

By end of this blog, you’ll learn:

  • How to configure AWS IoT Button
  • Use IoT Button as trigger for Lambda Function
  • Test IoT button

The overall flow will be:

serverless-iot-couchbase

Iot button click will invoke HelloCouchbaseLambda Lambda function. This function is uses Couchbase Java SDK to create a JSON document in Couchbase.

This blog is also playing catch up with Collecting iBeacon Data with Couchbase and Raspberry Pi IoT Devices by Nic and The CouchCase by Matthew on their summer projects. One last blog will be published in this series. That will show how multiple AWS IoT buttons can be used for some fun.

Let’s get started!

Configure IoT Button

The fastest way to configure IoT button  is using the mobile app for iOS or Android.

 

More details about configuring IoT Button using mobile app.

Here are some snapshots from configuring button using the mobile app.

Bring up the app, click on + to start configuring a new button:

aws-iot-button-configure-1

Enter button’s serial number:

aws-iot-button-configure-2

Register the button:

aws-iot-button-configure-3

Configure the button with wifi network:

aws-iot-button-configure-4

Upload all the certificates etc:

aws-iot-button-configure-5

After this, the button is configured and ready to use. This blog skipped the part where a template Lambda Function is associated with the button click.

If  mobile app cannot be used then the button can be configured manually.

Use IoT Button as Trigger for Lambda Function

The aws lambda create-event-source-mapping CLI allows to create an event source for Lambda function. As of AWS CLI version 1.11.21, only Amazon Kinesis stream or an Amazon DynamoDB stream can be used. But for this blog, we’ll use IoT button as a trigger. And this has to be configured using AWS Lambda Console.

IoT Button is only supported in a limited number of regions. For example, it is not supported in the us-west-1 region but us-west-2 region works.

The list of regions not supported are greyed out in the following list:

aws-iot-buttons-supported-region

Lambda Function can be triggered by several events. Lambda Function is invoked when any of these events occur. By default, no triggers are associated with a Lambda Function. For our HelloCouchbaseLambda function, these can be seen at us-west-2.console.aws.amazon.com/lambda/home?region=us-west-2#/functions/HelloCouchbaseLambda?tab=triggers.

AWS Lambda Default Triggers

Click on Add trigger to add a new trigger:

AWS Lambda Add Trigger

Select on the empty square to create a new trigger, and select AWS IoT:

AWS Lambda Add IoT Trigger

For the button previously registered, get the serial number from us-west-2.console.aws.amazon.com/iotv2/home?region=us-west-2#/thinghub:

aws-iot-things-hub

Specify the serial number of the button in the AWS IoT trigger:

aws-iot-add-trigger

Click on Submit to create the trigger:

aws-iot-added-trigger

And this confirms that the trigger has been added.

Test IoT Button

Before testing the button, let’s login to the Couchbase instance and verify the number of JSON documents in the bucket:

aws-iot-button-couchbase-console-default

This can be verified at http://<EC2-IP-Address>:8091/index.html#sec=buckets. As expected, no documents exists in the bucket.

Press the button once, and refresh the page. It shows that one document is now stored in the bucket. This is verified in the Couchbase Web Console:

aws-iot-button-couchbase-console-one-document

Click on Documents to see the complete list of documents:

aws-iot-button-couchbase-one-document-2

Click on the document ID to see more details about the document:

aws-iot-button-couchbase-one-document-details

Only timestamp is stored in this JSON document.

Now, let’s update HelloCouchbaseLambda code to include request id in the document as well. This can be achieved by adding the following line of code in the Java class:

A new deployment package can be built and uploaded using the following command:

Now clicking the button will update the number of documents. But the updated document will have an additional attribute populated as shown:

aws-iot-button-couchbase-second-document-details

How are you going to take AWS IoT button and use it with Lambda and Couchbase? Let us know at Couchbase Forums.

References

  • AWS IoT Button
  • AWS IoT Button Developer Guide
  • Couchbase Server Docs
  • Couchbase Forums
  • Follow us at @couchbasedev

Source: https://blog.couchbase.com/2016/december/aws-iot-button-lambda-couchbase

Serverless FaaS with AWS Lambda and Java

couchbase, techtip, ,

What is Serverless Architecture?

Serverless architecture runs custom code in ephemeral containers that are fully managed by a 3rd party. The custom code is typically a small part of a complete application. It is also called as function. This gives another name for serverless architecture as Function as a Service (FaaS). The container is ephemeral because it may only last for one invocation. The container may be reused but that’s not something you can rely upon. As a developer, you upload the code to FaaS platform, the service then handles all the capacity, scaling, patching and administration of the infrastructure to run your code.

An application built using Serverless Architecture follows the event-driven approach. For example, an activity happened in the application such as a click. This is

This is very different from a classical architecture where the application code is typically deployed in an application server such as Tomcat or WildFly. Scaling your application means starting additional instances of the application server or spinning up additional containers with the packaged application server. The Load Balancer need to be updated with the new IP addresses. Operating system need to be patched, upgraded and maintained.

Serverless Architectures explain the difference between the classical programming model and this new serverless architecture.

FaaS platform takes your application is divided into multiple functions. Each function is deployed in FaaS. The service spins up additional compute instances to meet the scalability demands of your application. FaaS platform provides the execution environment and takes care of starting and tearing down the containers to run your function.

Read Serverless Architectures for more details about these images.

One of the big advantages of FaaS is that you are only charged for the compute time, i.e. the time your code is running. There is no charge when your code is not running.

Another way to look at how Functions are different from VMs and Containers:

vm-containers-serverless

Note that Linux containers instead of Docker containers are used as an implementation for AWS Lambda.

How is FaaS different from PaaS?

As quoted at Serverless Architectures, a quick answer is provided by the following tweet:

In other words most PaaS applications are not geared towards bringing entire applications up and down for every request, whereas FaaS platforms do exactly this.

Abstracting the Back-end with FaaS explain the difference with different *aaS offerings. The image from the blog is captured below:

faas

Serverless Architectures also provide great details about what FaaS is and is not.

AWS Lambda, Google Cloud Functions and Azure Functions are some of the options for running serverless applications.

This blog will show how to write your first AWS Lambda function.

What is AWS Lambda?

AWS Lambda is FaaS service from Amazon Web Services. It runs your code on a high-availability compute infrastructure and performs all of the administration of the compute resources, including server and operating system maintenance, capacity provisioning and automatic scaling, code monitoring and logging.

AWS Lambda charges you for the duration your code runs in increments of 100ms. There is no cost associated with storing the Lambda function in AWS. First million requests per month are free and the pricing after that is nominal. Read more details on Lambda pricing. It also provides visibility into performance by providing real time metrics and logs to AWS CloudWatch. All you need to do is write the code!

Here is a quick introduction:

Also check out What’s New in AWS Lambda from AWS ReInvent 2016:

Also checkout Serverless Architectural Patterns and Best Practices from AWS ReInvent 2016:

The code you run on AWS Lambda is called a Lambda Function. You upload your code as a zip file or design it using the AWS Lambda Management Console. There is a built-in support for AWS SDK and this simplifies the ability to call other AWS services.

In short, Lambda is scalable, serverless, compute in the cloud.

AWS Lambda provides several execution environments:

  • Node.js – v0.10.36, v4.3.2 (recommended)
  • Java – Java 8
  • Python – Python 2.7
  • .NET Core – .NET Core 1.0.1 (C#)

This blog will show:

  • Build a Java application that stores a JSON document to Couchbase
  • Use Maven to create a deployment package for Java application
  • Create a Lambda Function
  • Update the Lambda Function

The complete code in this blog is available at github.com/arun-gupta/serverless/tree/master/aws/hellocouchbase.

Java Application for AWS Lambda

First, lets look at a Java application that will be used for this Lambda function. Programming Model for Lambda Functions in Java provide more details about how to write your Lambda function code in Java.

Our Lambda function will implemented the pre-defined interface com.amazonaws.services.lambda.runtime.RequestHandler. The code looks like:

handleRequest method is where the function code is implemented. Context provides useful information about Lambda execution environment. Some of the information from the context is stored a JSON document. Finally, Couchbase Java SDK API upsert is used to write a JSON document to the identified Couchbase instance. Couchbase on Amazon EC2 provide complete instructions to install Couchbase on AWS EC2.

Information about the Couchbase server is obtained as:

This is once again using Couchbase Java API CouchbaseCluster as a main entry point to the Couchbase cluster. The COUCHBASE_HOST environment variable is passed when the Lambda function is created. In our case, this would point to a single node Couchbase cluster running on AWS EC2. Environment variables were recently introduced in AWS Lambda.

Finally, you need to access bucket in the server:

The bucket name is serverless and all JSON documents are stored in this.

A simple Hello World application may be used for creating this function as well.

Create AWS Lambda Deployment Package

AWS Lambda function needs a deployment package. This package is either a .zip or .jar file that contains all the dependencies of the function. Our application is packaged using Maven, and so we’ll use a Maven plugin to create a deployment package.

The application has pom.xml with the following plugin fragment:

More details about Maven configuration are available in Creating a .jar Deployment Package Using Maven without any IDE. The maven-shade-plugin allows to create an uber-jar including all the dependencies. The shade goal is tied to the package phase. So the mvn package command will generate a single deployment jar.

Package the application using mvn package command. This will show the output:

The target/hello-couchbase-1.0-SNAPSHOT.jar is the shaded jar that will be deployed to AWS Lambda.

More details about creating a deployment package are at Creating a Deployment Package.

Create AWS Lambda Function

Create AWS Lambda Function using AWS CLI. The CLI command in this case looks like:

In this CLI:

  • create-function creates a Lambda function
  • --function-name provides the function name. The function name is case sensitive.
  • --role specifies Amazon Resource Name (ARN) of an IAM role that Lambda assume when it executes your function to access any other AWS resources. If you’ve executed a Lambda function using AWS Console then this role is created for you.
  • --zip-file points to the deployment package that was created in previous step. fileb is an AWS CLI specific protocol to indicate that the content uploaded is binary.
  • --handler is the Java class that is called to begin execution of the function
  • --publish request AWS Lambda to create the Lambda function and publish a version as an atomic operation. Otherwise multiple versions may be created and may be published at a later point.

Lambda Console shows:

servleress-couchbase-lambda-function

Test AWS Lambda Function

Test the AWS Lambda Function using AWS CLI.

It shows the output as:

The output from the command is stored in hellocouchbase.out and looks like:

Invoking this function stores a JSON document in Couchbase. Documents stored in Couchbase can be seen using Couchbase Web Console. The password is Administrator and password is the EC2 instance id.

All data buckets in this Couchbase instance are shown below:

serverless-couchbase-bucket-overview

Note that the serverless bucket is manually created.

Clicking on Documents shows details of different documents stored in the bucket:

serverless-couchbase-bucket-documents

Clicking on each document shows more details about the JSON document:

serverless-couchbase-bucket-document

Lambda function can also be tested using the Console:

serverless-couchbase-console-test

Update AWS Lambda Function

If the application logic changes then a new deployment package needs to be uploaded for the Lambda function. In this case, mvn package will create a deployment package and aws lambda CLI command is used to update the function code:

Shows the result:

The function can then be invoked again.

During writing of this blog, this was often used to debug the function as well. This is because Lambda functions do not have any state or box associated with them. And so you cannot log in to a box to check out if the function did not deploy correctly. You can certainly use CloudWatch log statements once the function is working.

AWS Lambda References

  • Serverless Architectures
  • AWS Lambda: How it works
  • Couchbase Server Docs
  • Couchbase Forums
  • Follow us at @couchbasedev

Source: https://blog.couchbase.com/2016/december/serverless-faas-aws-lambda-java

Kubernetes Monitoring with Heapster, InfluxDB and Grafana

containers, couchbase, ,

Kubernetes provides detailed insights about resource usage in the cluster. This is enabled by using Heapster, cAdvisor, InfluxDB and Grafana.

Heapster is installed as a cluster-wide pod. It gathers monitoring and events data for all pods on each node by talking to the Kubelet. Kubelet itself fetches this data from cAdvisor. This data is persisted in InfluxDB and then visualized using Grafana.

kubernetes-logging

Resource Usage Monitoring provide more details about monitoring resources in Kubernetes.

Heapster, InfluxDB and Grafana are Kubernetes addons. They are enabled by default if you are running the cluster on Amazon Web Services or Google Cloud. But need to be explicitly enabled if the cluster is started using minikube or kops addons.

Start a Kubernetes cluster on Amazon Web Services as:

KUBERNETES_PROVIDER=aws; kube-up.sh

More details about starting a Kubernetes cluster are available at Getting Started with Kubernetes 1.4.

By default, it creates a 4-node Kubernetes cluster in us-west-2a region. More details about the cluster can be seen using the command kubectl cluster-info and it shows the results as:

Note the URL for the Grafana service. Open this URL in a browser window. You’ll be prompted for an invalid certificate warning but this can be safely ignored at this time. In production system, appropriate certificates should be installed.

Then you’ll be prompted for credentials. These can be obtained using kubectl config view command. It will show the output as:

Use the value from username and password fields.

This shows the default dashboard:

kubernetes-grafana-empty-dashboard

It consists of two dashboards – one for cluster and another for pods.

kubernetes-grafana-dashboards

For this blog, a 4-node Couchbase cluster was created following the steps outlined in Create a Couchbase Cluster using Kubernetes.

A cluster-wide dashboard shows CPU, Memory, Filesystem and Network usage across all the hosts and looks like:

kubernetes-grafana-cluster

CPU, memory, filesystem and network usage for all nodes may be seen:

kubernetes-grafana-cluster-per-node

Details for each node may be seen by selecting the node:

kubernetes-grafana-cluster-nodelist

CPU, memory, filesystem and network usage for each node is displayed:

kubernetes-grafana-cluster-one-node

Pods dashboard shows CPU, memory, filesystem and network usage for each pod:

kubernetes-grafana-pods

A different pod may be chosen:

kubernetes-grafana-pods-list

A complete list of all services running in the Kubernetes can be seen using kubectl get services --all-namespaces command. It shows the output as:

A complete list of all the pods running in the Kubernetes cluster can be seen using kubectl get pods --all-namespaces. It shows the output as:

kubectl.sh get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default couchbase-master-rc-q9awd 1/1 Running 17 56m
default couchbase-worker-rc-b1qkc 1/1 Running 15 54m
default couchbase-worker-rc-j1c5z 1/1 Running 17 52m
default couchbase-worker-rc-ju7z3 1/1 Running 15 52m
kube-system elasticsearch-logging-v1-18ylh 1/1 Running 0 1h
kube-system elasticsearch-logging-v1-fupap 1/1 Running 0 1h
kube-system fluentd-elasticsearch-ip-172-20-0-94.us-west-2.compute.internal 1/1 Running 0 1h
kube-system fluentd-elasticsearch-ip-172-20-0-95.us-west-2.compute.internal 1/1 Running 0 1h
kube-system fluentd-elasticsearch-ip-172-20-0-96.us-west-2.compute.internal 1/1 Running 15 1h
kube-system fluentd-elasticsearch-ip-172-20-0-97.us-west-2.compute.internal 1/1 Running 17 1h
kube-system heapster-v1.2.0-1374379659-jms8e 4/4 Running 0 1h
kube-system kibana-logging-v1-fcg4b 1/1 Running 3 1h
kube-system kube-dns-v20-wpip4 3/3 Running 0 1h
kube-system kube-proxy-ip-172-20-0-94.us-west-2.compute.internal 1/1 Running 0 1h
kube-system kube-proxy-ip-172-20-0-95.us-west-2.compute.internal 1/1 Running 0 1h
kube-system kube-proxy-ip-172-20-0-96.us-west-2.compute.internal 1/1 Running 15 1h
kube-system kube-proxy-ip-172-20-0-97.us-west-2.compute.internal 1/1 Running 17 1h
kube-system kubernetes-dashboard-v1.4.0-yxxgx 1/1 Running 0 1h
kube-system monitoring-influxdb-grafana-v4-7asy4 2/2 Running 0 1h

Some references:

  • Kubernetes Resource Monitoring
  • Couchbase Cluster using Kubernetes, Docker Swarm, DC/OS and Amazon ECS
  • Follow us @couchbasedev

Source: blog.couchbase.com/2016/december/kubernetes-monitoring-heapster-influxdb-grafana

Couchbase Cluster on Mesos with DC/OS

containers, couchbase

apache-mesos-logoapache-mesos-marathon-logoDocker LogoCouchbase Logo

Apache Mesos is an open source cluster manager developed at UC Berkeley. It provides resource isolation and sharing across distributed applications. Mesos consists of a master daemon that manages slave daemons running on each cluster node.Mesos frameworks are applications that runs on Mesos and run tasks on these slaves. Marathon is a container orchestration platform running on Mesos.Multiple container formats are supported and Docker is certainly the most common one!

Docker Container using Apache Mesos and Marathon explains how to setup Mesos and Marathon. The setup is quite involving and a bit flaky. It required to download and Install Mesos Master and Slave, ZooKeeper, Docker Engine, and Marathon.

DC/OS is a distributed operating system using Mesos as its kernel. Couchbase on Mesos using DC/OS and Amazon explained how to run a single Couchbase container on DC/OS using CloudFormation templates.

Running a single node Couchbase may work during initial development phase. The need to start a multi-node Couchbase cluster becomes eminent as you move along further in development, and certainly needed during the production phase.

So, you’d like to run a Couchbase cluster on DC/OS?

Couchbase Cluster on DC/OS is complete walk through of how to setup a Couchbase cluster on DC/OS. It walks through the following steps:

  • What is Couchbase?
  • Couchbase Cluster
  • Setup DC/OS on Amazon Web Services
  • Configure CLI and Install Marathon Load Balancer
  • Create Couchbase “startup” service
  • Create Couchbase “node” service
  • Scale Couchbase Cluster
  • Rebalance Couchbase Cluster
  • Conclusion

DC/OS dashboard with a Couchbase cluster looks like:

For further information check out:

  • Couchbase on Containers
  • Couchbase Developer Portal
  • Ask questions on Couchbase Forums or Stack Overflow
  • Download Couchbase

You can also follow us at @couchbasedev and @couchbase.

Source: http://blog.couchbase.com/2016/november/couchbase-cluster-mesos-dcos

Health Check of Docker Containers

containers, couchbase

One of the new features in Docker 1.12 is how health check for a container can be baked into the image definition. And this can be overridden at the command line.

Just like the CMD instruction, there can be multiple HEALTHCHECK instructions in Dockerfile but only the last one is effective.

This is a great addition because a container reporting status as Up 1 hour may return errors. The container may be up but there is no way for the application inside the container to provide a status. This instruction fixes that.

The Dockerfile that builds arungupta/couchbase image is:

It uses configure-node.sh script to configure the server using Couchbase REST API. The new instruction to notice here is HEALTHCHECK.

This instruction can be specified as:

The <options> can be:

  • --interval=DURATION (default 30s)
  • --timeout=DURATION (default 30s)
  • --retries=N (default 3)

The <command> is the command that runs inside the container to check the health.

If health check is enabled, then the container can have three states:

  • starting – Initial status when the container is still starting
  • healthy – If the command succeeds then the container is healthy
  • unhealthy – If a single run of the <command> takes longer than the specified timeout then it is considered unhealthy. If a health check fails then the <command> will run retries number of times and will be declared unhealthy if the <command> still fails.

The commands exit status indicates the health status of the container. The following values are allowed:

  • 0 – container is healthy
  • 1 – container is not healthy

In our instruction, /pools REST API is invoked using curl. If the command fails then an exit status of 1 is returned, and this marks the container unhealthy for that attempt. This command is invoked every 5 seconds. The container is marked unhealthy if the command does not return successfully within 3 seconds.

Run the container as:

Check the status:

Notice how health: starting status is reported in the STATUS column. Checking after a few seconds shows the status:

And now its reported healthy.

More details about this HEALTHCHECK instruction can be found on docs.docker.com.

Now, if you are running an image that does not have HEALTHCHECK instruction then the docker run command can be used to specify similar values. An equivalent runtime command would be:

Last 5 health checks for a container can be obtained using the docker inspect command:

The output is shown as:

 

Source: http://blog.couchbase.com/2016/november/docker-health-check-keeping-containers-healthy

 

Docker for AWS – Getting Started Video

containers, couchbase, ,

Want to create a highly-available Docker cluster on Amazon Web Services? Run multi-container applications on it using Docker Services?

Docker Logo
amazon-web-services-logo
Couchbase Logo

Docker for AWS allows you to exactly do that! This video shows:

  • Create a highly-available Docker cluster on Amazon Web Services (0:00)
  • Check configuration (5:43)
  • Use Docker services to create a Couchbase cluster (8:23)

Enjoy!

couchbase.com/containers provide more details about how to run Couchbase in different container frameworks. More information about Couchbase:

  • Couchbase Developer Portal
  • Couchbase Forums
  • @couchbasedev or @couchbase

Source: blog.couchbase.com/2016/november/docker-for-aws-getting-started-video

Multimaster Kubernetes Cluster on Amazon Using Kops

containers, couchbase

Getting Started with Kubernetes 1.4 using Spring Boot and Couchbase explains how to get started with Kubernetes  1.4 on Amazon Web Services. A Couchbase service is created in the cluster and a Spring Boot application stores a JSON document in the database. It uses kube-up.sh script from the Kubernetes binary download at github.com/kubernetes/kubernetes/releases/download/v1.4.0/kubernetes.tar.gz  to start the cluster. This script is capable of creating a Kubernetes cluster with single master only. This is a fundamental flaw of distributed applications where the master becomes a Single Point of Failure.

Meet kops – short for Kubernetes Operations.

This is the easiest way to get a highly-available Kubernetes cluster up and running. The kubectl script is the CLI for running commands against running clusters. Think of kops as kubectl for cluster.

This blog will show how to create a highly-available Kubernetes cluster on Amazon using kops. And once the cluster is created, then it’ll create a Couchbase service on it and run a Spring Boot application to store JSON document in the database.

Many thanks to justinsb, sarahz, razic, jaygorrell, shrugs, bkpandey and others at Kubernetes slack channel for helping me through the details!

Download kops and kubectl

  • Download Kops latest release. This blog was tested with 1.4.1 on OSX.Complete set of commands for kops can be seen:
  • Download kubectl:
  •  Include kubectl in your PATH.

Create Bucket and NS Records on Amazon

There is a bit of setup involved at this time, and hopefully this will get cleaned up over next releases. Bringing up a cluster on AWS provide detailed steps and more background. Here is what the blog followed:

  • Pick a domain where Kubernetes cluster will be hosted. This blog uses kubernetes.arungupta.me domain. You can pick a top level domain or a sub-domain.
  • Amazon Route 53 is a highly available and scalable DNS service. Login to Amazon Console and created a hosted zone for this domain using Route 53 service.kops-hosted-zoneCreated zone looks like:kops-hosted-zone-createdThe values shown in the Value column are important as they’ll be used later for creating NS records.
  • Create a S3 bucket using Amazon Console to store cluster configuration – this is called state store.kops-s3-bucket
  • The domain kubernetes.arungupta.me is hosted on GoDaddy. For each value shown in the Value column of Route53 hosted zone, create a NS record using GoDaddy Domain Control Center for this domain.Select the type of record:
    kops-godaddy-add-zone-recordFor each value, add the record as shown:kops-godaddy-add-ns-recordCompleted set of records look like:kops-godaddy-ns-records

Start Kubernetes Multimaster Cluster

Let’s understand a bit about Amazon regions and zones:

Amazon EC2 is hosted in multiple locations world-wide. These locations are composed of regions and Availability Zones. Each region is a separate geographic area. Each region has multiple, isolated locations known as Availability Zones.

Amazon Docs

A highly-available Kubernetes cluster can be created across zones but not across regions.

  • Find out availability zones within a region:
  • Create a multi-master cluster:
    Most of the switches are self-explanatory. Some switches need a bit of explanation:

    • Specifying multiple zones using --master-zones (must be odd number) create multiple masters across AZ
    • --cloud=aws is optional if cloud can be inferred from zones
    • --yes is used to specify the immediate creation of cluster. Otherwise only the state is stored in the bucket, and the cluster needs to be created separately.

    Complete set of CLI switches can be seen:

  • Once the cluster is created, get more details about the cluster:

  • Check cluster client and server version:

  • Check all nodes in the cluster:

    Or find out only the master nodes:

  • Check all the clusters:

Kubernetes Dashboard Addon

By default, a cluster created using kops does not have the UI dashboard. But this can be added as an add on:

Now complete details about the cluster can be seen:

And the Kubernetes UI dashboard is at the shown URL. In our case, this is https://api.kubernetes.arungupta.me/ui and looks like:

kops-kubernetes-ui

Credentials for accessing this dashboard can be obtained using the kubectl config view command. The values are shown like:

Deploy Couchbase Service

As explained in Getting Started with Kubernetes 1.4 using Spring Boot and Couchbase, let’s run a Couchbase service:

This configuration file is at github.com/arun-gupta/kubernetes-java-sample/blob/master/maven/couchbase-service.yml.

Get the list of services:

Describe the service:

Get the pods:

Run Spring Boot Application

The Spring Boot application runs against the Couchbase service and stores a JSON document in it.

Start the Spring Boot application:

This configuration file is at github.com/arun-gupta/kubernetes-java-sample/blob/master/maven/bootiful-couchbase.yml.

See list of all the pods:

Check logs of the complete pod:

The updated dashboard now looks like:

kops-dashboard-with-apps

Delete the Kubernetes Cluster

Kubernetes cluster can be deleted as:

couchbase.com/containers provide more details about how to run Couchbase in different container frameworks.

More information about Couchbase:

  • Couchbase Developer Portal
  • Couchbase Forums
  • @couchbasedev or @couchbase

Source: blog.couchbase.com/2016/november/multimaster-kubernetes-cluster-amazon-kops

Docker Container Anti Patterns

containers, couchbase,

This blog will explain 10 containers anti-patterns that I’ve seen over the past few months:

  1. Data or logs in containers – Containers are ideal for stateless applications and are meant to be ephemeral. This means no data or logs should be stored in the container otherwise they’ll be lost when the container terminates. Instead use volume mapping to persist them outside the containers. ELK stack could be used to store and process logs. If managed volumes are used during early testing process, then remove them using -v switch with the docker rm command.
  2.  IP addresses of container – Each container is assigned an IP address. Multiple containers communicate with each other to create an application, for example an application deployed on an application server will need to talk with a database. Existing containers are terminated and new containers are started all the time. Relying upon the IP address of the container will require constantly updating the application configuration. This will make the application fragile. Instead create services. This will provide a logical name that can be referred independent of the growing and shrinking number of containers. And it also provides a basic load balancing as well.
  3. Run a single process in a container – A Dockerfile has use one CMD and ENTRYPOINT. Often, CMD will use a script that will perform some configuration of the image and then start the container. Don’t try to start multiple processes using that script. Its important to follow separation of concerns pattern when creating Docker images. This will make managing your containers, collecting logs, updating each individual process that much harder. You may consider breaking up application into multiple containers and manage them independently.
  4. Don’t use docker exec – The docker exec command starts a new command in a running container. This is useful for attaching a shell using the docker exec -it {cid} bash. But other than that the container is already running the process that its supposed to be running.
  5. Keep your image lean – Create a new directory and include Dockerfile and other relevant files in that directory. Also consider using .dockerignore to remove any logs, source code, logs etc before creating the image. Make sure to remove any downloaded artifacts after they are unzipped.
  6. Create image from a running container – A new image can be created using the docker commit command. This is useful when any changes in the container have been made. But images created using this are non-reproducible. Instead make changes in the Dockerfile, terminate existing containers and start a new container with the updated image.
  7. Security credentials in Docker image – Do not store security credentials in the Dockerfile. They are in clear text and checked into a repository. This makes them completely vulnerable. Use -e to specify passwords as runtime environment variable. Alternatively --env-file can be used to read environment variables from a file. Another approach is to used CMD or ENTRYPOINT to specify a script. This script will pull the credentials from a third party and then configure your application.
  8. latest tag: Starting with an image like couchbase is tempting. If no tags are specified then a container is started using the image couchbase:latest.  This image may not actually be latest and instead refer to an older version. Taking an application into production requires a fully controller environment with exact version of the image. Read this Docker: The latest confusion post by fellow Docker Captain @adrianmouat.  Make sure to always use tag when running a container. For example, use couchbase:enterprise-4.5.1 instead of just couchbase.
  9. Impedance mismatch – Don’t use different images, or even different tags in dev, test, staging and production environment. The image that is the “source of truth” should be created once and pushed to a repo. That image should be used for different environments going forward. In some cases, you may consider running your unit tests on the WAR file as part of maven build and then create the image. But any system integration testing should be done on the image that will be pushed in production.
  10. Publishing ports – Don’t use -P to publish all the exposed ports. This will allow you to run multiple containers and publish their exposed ports. But this also means that all the ports will be published. Instead use -p to publish specific ports.

Adding more based upon discussion on twitter …

  1. Root user – Don’t run containers as root user. The host and the container share the same kernel. If the container is compromised, a root user can do more damage to the underlying hosts. Use RUN groupadd -r couchbase && useradd -r -g couchbase couchbase to create a group and a user in it. Use the USER instruction to switch to that user. Each USER creates a new layer in the image. Avoid switching the user back and forth to reduce the number of layers. Thanks to @Aleksandar_78 for this tip!
  2. Dependency between containers – Often applications rely upon containers to be started in a certain order. For example, a database container must be up before an application can connect to it. The application should be resilient to such changes as the containers may be terminated or started at any time. In this case, have the application container wait for the database connection to succeed before proceeding further. Do not use wait-for scripts in Dockerfile for the containers to startup in a specific order. Particularly waiting for a certain number of seconds for a particular container to start is very fragile. Thanks to @ratnopam for this tip!

What other anti-patterns do you follow?

Docker for Java developers is a self-paced hands-on workshop that explains how to get started with Docker for Java developers.

Interested in a more deep dive tutorial? Watch this 2-hours tutorial from JavaOne!

couchbase.com/containers shows how to run Couchbase in a variety of frameworks.

Source: blog.couchbase.com/2016/october/docker-container-anti-patterns

Docker on Windows 2016 Server

containers, ,

This blog is the first part of a multi-part series. The first part showed how to set up Windows Server 2016 as a VirtualBox VM. This second part will show how to configure Docker on Windows 2016 VM.

  1. Start an elevated PowerShell session:docker-windows-2016-22
  2. Run the script to install Docker:
    This will install the PowerShell module, enable containers feature and install Docker.

    docker-windows-2016-23

    The VM needs to be restarted in order for the containers to be enabled. Refer to Container Host Deployment – Windows Server for more detailed instructions.

  3. The VM reboots. Start a PowerShell and check Docker version using docker version command:docker-windows-2016-24More details about Docker can be found using the docker info command:docker-windows-2016-25
  4. Run your first Docker container using the docker run -it -p 80:80 microsoft/iis command:docker-windows-2016-26This will download the Microsoft IIS server Docker image. This is going to take a while so please be patient!
  5. Once the 8.9 GB image is downloaded (after a while), the IIS server is started for you. Check the list of images using the docker images command and the list of running containers using the docker ps command:docker-windows-2016-27More details about the container can be found using the docker inspect command:

  6. The exact IP address of the container can be found using the command:

    IIS main page is accessible at http://<container-ip>, as shown below:

    docker-windows-2016-28

The next part will show how to create your own Docker image on Windows Server 2016.

Source: blog.couchbase.com/2016/october/docker-on-windows-2016-server

Persisting Couchbase Data Across Container Restarts

containers, couchbase,

Best Practices for Virtualized Platforms provide best practices for running Couchbase on a virtualized platform like Amazon Web Services and Azure. In addition, it also provide some recommendations for running it as Docker container.

One of the recommendations is to map Couchbase node specific data to a local folder. Let’s understand that in more detail.

Implicit Per-Container Storage

If a Couchbase container is started as:

This container:

  • Starts in a detached mode using -d
  • Different query, caching and administration ports are mapped using -p
  • A name is provided using --name
  • Image is couchbase/server:sandbox

By default, the data for the container is stored in a managed volume. Checking volume mounts using the docker inspect command shows:

The data for Couchbase is stored in the container filesystem defined by the value of Source attribute. This can be verified by logging into the root filesystem:

Now you can see the data directory:

A new directory is created for a new run of the container. This directory is still around when the container is stopped and removed but no longer easily accessible. Thus no data is preserved across container restarts.

The volume can be explicitly removed, along with container, using the command:

If the container terminates then the entire state of the application is lost.

Explicit Host Directory Mapping

Now, let’s start a Couchbase container with explicit volume mapping:

This container is very similar to the container started earlier. The main difference is that a directory from host ~/couchbase is mapped to a directory in the container /opt/couchbase/var.

Couchbase container persists any data in /opt/couchbase/var directory in the container filesystem. Now that directory is mapped to a directory on the host filesystem. This allows to persist state of the container outside on the host filesystem. The bypasses the union filesystem used by Docker and exposes the host filesystem to the container. This allows the state to persist across container restarts. The new container only needs to start with the exact same volume mapping.

More details about the container can be seen as:

jq is a JSON processor that needs to be installed separately. And the output is shown as:

This shows the source and destination directory. RW shows that the volume is read/write.

If the container is started using Docker for Mac, then Couchbase Web Console is accessible at http://localhost:8091. The Data Buckets tab shows the default travel-sample bucket:

docker-volume-couchbase-01

Click on Create New Data Bucket to create a new data bucket. Give it the name sample:

docker-volume-couchbase-02

The Data Buckets tab is updated with this newly created bucket:

docker-volume-couchbase-03

Now stop and remove the container:

Start the container again using the same command:

Data Buckets tab will show the same two buckets in the Couchbase Web Console.

 

In this case, if the container is started on a different host then the state would not be available. Or if the host dies then the state is lost.

An alternative and a more robust and foolproof way to manage persistence in containers is using a shared network filesystem such as Ceph, GlusterFS or Network Filesystem. Some other common approaches are to use Docker Volume Plugins like Flocker from ClusterHQ or Software Defined Storage such as PortWorx. All of these storage technique simplify how state of a container can be saved in a multi-container multi-host environment. A future blog will cover these techniques in detail.

Read more details in Managing data in containers.

couchbase.com/containers provide more details about how to run Couchbase in different container frameworks.

More information about Couchbase:

  • Couchbase Developer Portal
  • Couchbase Forums
  • @couchbasedev or @couchbase

Source: blog.couchbase.com/2016/october/persisting-couchbase-data-across-container-restarts