Tag Archives: containers

Gossip-based Kubernetes Cluster on AWS using Kops

Creating a Kubernetes cluster using Kops requires a top-level domain or a sub domain and setting up Route 53 hosted zones. This domain allows the worker nodes to discover the master and the master to discover all the etcd servers. This is also needed for kubectl to be able to talk directly with the master. This worked well but an additional hassle for the developers.

Kubernetes Logo

Kops 1.6.2 adds an experimental support for gossip-based, uses Weave Mesh, discovery of nodes. This makes the process of setting up Kubernetes cluster using Kops DNS-free, and much more simplified.

Let’s take a look!

  1. Install or upgrade kops:
  2. Check the version:
  3. Create an S3 bucket as “state store”:
  4. Create a Kubernetes cluster:
    It shows the output as:
    Wait for a few minutes for the cluster to be created.
  5. Validate the cluster:
  6. Get the list of nodes using kubectl:
  7. Deleting a cluster is pretty straight forward as well:

That’s it!

github.com/arun-gupta/kubernetes-java-sample provide several examples of getting started with Kubernetes.

File issues at github.com/kubernetes/kops/issues.

 

Getting Started with Oracle Container Cloud Service

Oracle Cloud Container Logo

Oracle Container Cloud Service is Oracle’s entry into the the world of managed container service. There are plenty of existing options:

  • Docker for AWS or Azure
  • Amazon Elastic Container Service
  • Google Container Engine
  • Azure Container Service
  • DC/OS by Mesosphere
  • OpenShift by Red Hat

This blog will explain how to get started with Oracle Container Cloud Service. A comparison of different managed services is started at Managed Container Service.

Before we jump into all the details, let’s try to clarify a couple of things about this offering from Oracle.

First, a bit about the name. “Oracle Cloud Container Service” seems more natural and intuitive since its a Container Service in Oracle Cloud. Wonder why is it called “Oracle Container Cloud Service”? Is it because “Oracle Container” is Oracle’s container orchestration framework and its a Cloud Service? Could that mean other orchestration frameworks be offered as a service as well?

Second, don’t confuse it with Oracle Application Container Cloud Service that allows to build cloud-native 12-factor applications using polyglot platform. Now, that confuses me further. Can the Container Service not be used to build 12-factor apps? Are cloud-native and containers mutually exclusive?

Anyway, this is causing more confusion than clarification :) Let’s move on!

One last thing before we dig in. Many thanks to Bruno Borges (@brunoborges) for pushing the buttons for cloud service activation. I don’t know the normal time for the free trial to be activated otherwise. And a much bigger thank to Mike Raab (@mikeraab) for helping me understand the details of Container Service.

Let’s get started!

  1. Get a Free Trial for Oracle Cloud. It takes a few days for the trial to be activated. The trial time bombs after 30 days so make sure you’ve time planned for evaluation. Each free trial comes with 6 OC3 nodes. OC3 is one of the compute node types available on Oracle Cloud. OC3 particularly is 1 OCPU (think vCPU on Amazon Web Services) and 7.5 GB RAM.
  2. Once the account is activated, you get an email as shown:oracle-cloud-welcome-emailThe important piece of information is username, temporary password, identity domain and My Services URL. The My Account URL link is only for account administration.
  3. Click on My Service URL, login using the values from email:oracle-cloud-services-login
    You get an opportunity to change your password afterwards
  4. Oracle Cloud dashboard shows up after logging in:oracle-cloud-services-dashboardA default set of services and their status is shown. The dashboard can also be customized by clicking on Customize Dashboard button on the top right.
  5. Getting to Oracle Container Cloud Service Console is a bit non-intuitive but you get it once you know it. Select Container Cloud Service tab, click on top-right corner and select Open Service Console:oracle-cloud-container-service-console-accessOr you can directly click on the link for Oracle Container Cloud Service Console in the welcome email. Service console looks like:oracle-cloud-container-service-console
  6. Click on Create Service:oracle-cloud-container-service-definitionOracle Container Container Service Instance Details provide more details about each of the field.What is a worker node? We’ll talk about it a bit later. But essentially this is where the container runs. We are asking for only one worker node.

    Its worth noting different capacities for the worker node:
    Oracle Cloud CPUs

    Confirm all the settings:

    oracle-cloud-container-service-definition-confirmation

    and click on Create> to start the service creation.

  7. Wait for about 30 minutes for the service to be created. After that the Service Console looks like:
    oracle-cloud-container-service-console-with-serviceWait, we asked for  one worker node and how come two OCPUs are being consumed.Each Oracle Container Cloud Service has at least two nodes – a manager node and one or more worker nodes. Manager node is responsible for administration of all the workers and and orchestrate containers on different worker nodes. Worker nodes can be organized in different resource pools to meet different workflow needs.

    And, so ~30 minutes are spent provisioning two nodes and installing container service components on each node. This is also evident in the service logs shown in Service Create and Delete History shown in the main Console page:

    No timestamp in the activity feels a bit too clean.

  8. One main question that I kept wondering all along is “when am I ready to deploy the containers?“. Apparently, not yet!A couple of more steps so hang in there …

    In your service, click on the top-right icon to select another menu:
    oracle-cloud-container-console-open

    Select Container Console.  So, now you are transitioning from Oracle Container Cloud Service Console to Container Console. Make sure to use the right terminology otherwise it gets confusing fast.

  9. This attempts to open Container Console but prompts the usual warningoracle-cloud-container-console-open-warning

    Just click on Proceed link. In a typical production setup, this will setup correctly using certificates and so this warning would not happen.

  10. This brings up a login screen:oracle-cloud-container-console-login
  11. Use the username and password specified during service creation earlier. Click on Login to see Container Console:oracle-cloud-container-console-default

Are we there yet?

Yes, now is the time to deploy containers. But we’ll cover that in a subsequent blog!

Just to recap on what is needed to get started with Oracle Container Cloud Service …

  1. Register for Oracle Cloud trial
  2. Login to main Oracle Cloud Dashboard
  3. Create a Oracle Container Cloud Service Instance
  4. Oracle Container Cloud Service Instance Console
  5. Container Console

All the steps need to be done once but a console inside a console inside a dashboard feels like Inception. The good thing is that the IP address of Container Console is a public IP address served by Oracle Cloud and can be used from anywhere.

Oracle Container Cloud Service Docs have lot more details about building and deploying applications using this Console.

In the next blog, we’ll see what it takes to run a Couchbase container using this console? Possibly a cluster of Couchbase across multiple hosts?

Want to learn more about running Couchbase in containers?

  • Couchbase on Containers
  • Couchbase Forums
  • Couchbase Developer Portal
  • @couchhasedev and @couchbase

Source: https://blog.couchbase.com/2017/february/getting-started-oracle-container-cloud-service

Docker Container Anti Patterns

This blog will explain 10 containers anti-patterns that I’ve seen over the past few months:

  1. Data or logs in containers – Containers are ideal for stateless applications and are meant to be ephemeral. This means no data or logs should be stored in the container otherwise they’ll be lost when the container terminates. Instead use volume mapping to persist them outside the containers. ELK stack could be used to store and process logs. If managed volumes are used during early testing process, then remove them using -v switch with the docker rm command.
  2.  IP addresses of container – Each container is assigned an IP address. Multiple containers communicate with each other to create an application, for example an application deployed on an application server will need to talk with a database. Existing containers are terminated and new containers are started all the time. Relying upon the IP address of the container will require constantly updating the application configuration. This will make the application fragile. Instead create services. This will provide a logical name that can be referred independent of the growing and shrinking number of containers. And it also provides a basic load balancing as well.
  3. Run a single process in a container – A Dockerfile has use one CMD and ENTRYPOINT. Often, CMD will use a script that will perform some configuration of the image and then start the container. Don’t try to start multiple processes using that script. Its important to follow separation of concerns pattern when creating Docker images. This will make managing your containers, collecting logs, updating each individual process that much harder. You may consider breaking up application into multiple containers and manage them independently.
  4. Don’t use docker exec – The docker exec command starts a new command in a running container. This is useful for attaching a shell using the docker exec -it {cid} bash. But other than that the container is already running the process that its supposed to be running.
  5. Keep your image lean – Create a new directory and include Dockerfile and other relevant files in that directory. Also consider using .dockerignore to remove any logs, source code, logs etc before creating the image. Make sure to remove any downloaded artifacts after they are unzipped.
  6. Create image from a running container – A new image can be created using the docker commit command. This is useful when any changes in the container have been made. But images created using this are non-reproducible. Instead make changes in the Dockerfile, terminate existing containers and start a new container with the updated image.
  7. Security credentials in Docker image – Do not store security credentials in the Dockerfile. They are in clear text and checked into a repository. This makes them completely vulnerable. Use -e to specify passwords as runtime environment variable. Alternatively --env-file can be used to read environment variables from a file. Another approach is to used CMD or ENTRYPOINT to specify a script. This script will pull the credentials from a third party and then configure your application.
  8. latest tag: Starting with an image like couchbase is tempting. If no tags are specified then a container is started using the image couchbase:latest.  This image may not actually be latest and instead refer to an older version. Taking an application into production requires a fully controller environment with exact version of the image. Read this Docker: The latest confusion post by fellow Docker Captain @adrianmouat.  Make sure to always use tag when running a container. For example, use couchbase:enterprise-4.5.1 instead of just couchbase.
  9. Impedance mismatch – Don’t use different images, or even different tags in dev, test, staging and production environment. The image that is the “source of truth” should be created once and pushed to a repo. That image should be used for different environments going forward. In some cases, you may consider running your unit tests on the WAR file as part of maven build and then create the image. But any system integration testing should be done on the image that will be pushed in production.
  10. Publishing ports – Don’t use -P to publish all the exposed ports. This will allow you to run multiple containers and publish their exposed ports. But this also means that all the ports will be published. Instead use -p to publish specific ports.

Adding more based upon discussion on twitter …

  1. Root user – Don’t run containers as root user. The host and the container share the same kernel. If the container is compromised, a root user can do more damage to the underlying hosts. Use RUN groupadd -r couchbase && useradd -r -g couchbase couchbase to create a group and a user in it. Use the USER instruction to switch to that user. Each USER creates a new layer in the image. Avoid switching the user back and forth to reduce the number of layers. Thanks to @Aleksandar_78 for this tip!
  2. Dependency between containers – Often applications rely upon containers to be started in a certain order. For example, a database container must be up before an application can connect to it. The application should be resilient to such changes as the containers may be terminated or started at any time. In this case, have the application container wait for the database connection to succeed before proceeding further. Do not use wait-for scripts in Dockerfile for the containers to startup in a specific order. Particularly waiting for a certain number of seconds for a particular container to start is very fragile. Thanks to @ratnopam for this tip!

What other anti-patterns do you follow?

Docker for Java developers is a self-paced hands-on workshop that explains how to get started with Docker for Java developers.

Interested in a more deep dive tutorial? Watch this 2-hours tutorial from JavaOne!

couchbase.com/containers shows how to run Couchbase in a variety of frameworks.

Source: blog.couchbase.com/2016/october/docker-container-anti-patterns

Docker on Windows 2016 Server

This blog is the first part of a multi-part series. The first part showed how to set up Windows Server 2016 as a VirtualBox VM. This second part will show how to configure Docker on Windows 2016 VM.

  1. Start an elevated PowerShell session:docker-windows-2016-22
  2. Run the script to install Docker:
    This will install the PowerShell module, enable containers feature and install Docker.

    docker-windows-2016-23

    The VM needs to be restarted in order for the containers to be enabled. Refer to Container Host Deployment – Windows Server for more detailed instructions.

  3. The VM reboots. Start a PowerShell and check Docker version using docker version command:docker-windows-2016-24More details about Docker can be found using the docker info command:docker-windows-2016-25
  4. Run your first Docker container using the docker run -it -p 80:80 microsoft/iis command:docker-windows-2016-26This will download the Microsoft IIS server Docker image. This is going to take a while so please be patient!
  5. Once the 8.9 GB image is downloaded (after a while), the IIS server is started for you. Check the list of images using the docker images command and the list of running containers using the docker ps command:docker-windows-2016-27More details about the container can be found using the docker inspect command:

  6. The exact IP address of the container can be found using the command:

    IIS main page is accessible at http://<container-ip>, as shown below:

    docker-windows-2016-28

The next part will show how to create your own Docker image on Windows Server 2016.

Source: blog.couchbase.com/2016/october/docker-on-windows-2016-server

Minikube – Rapid Dev & Testing for Kubernetes

One of the attendees from Kubernetes for Java Developers training suggested to try minikube for simplified Kubernetes dev and testing. This blog will show how to get started with minikube using a simple Java application.

minikube-logo

Minikube starts a single node Kubernetes cluster on your local machine for rapid development and testing. Requirements lists the exact set of requirements for different operating systems.

This blog will show:

  • Start one node Kubernetes cluster
  • Run Couchbase service
  • Run Java application
  • View Kubernetes Dashboard

All Kubernetes resource description files used in this blog are at github.com/arun-gupta/kubernetes-java-sample/tree/master/maven.

Start Kubernetes Cluster using Minikube

Create a new directory with the name minikube.

In that directory, download kubectl CLI:

Download minikube CLI:

Start the cluster:

The list of nodes can be seen:

More details about the cluster can be obtained using the kubectl cluster-info command:

Behind the scenes, a Virtual Box VM is started.

Complete set of commands supported can be seen by using --help:

Run Couchbase Service

Create a Couchbase service:

This will start a Couchbase service. The service is using the pods created by the replication controller. The replication controller creates a single node Couchbase server.

The configuration file is at github.com/arun-gupta/kubernetes-java-sample/blob/master/maven/couchbase-service.yml and looks like:

Run Java Application

Run the application:

The configuration file is at github.com/arun-gupta/kubernetes-java-sample/blob/master/maven/bootiful-couchbase.yml and looks like:

This is run-once job which runs a Java (Spring Boot) application and upserts (insert or update) a JSON document in Couchbase.

In this job, COUCHBASE_URI environment variable value is set to couchbase-service. This is the service name created earlier. Docker image used for this service is arungupta/bootiful-couchbase and is created using fabric8-maven-plugin as shown at github.com/arun-gupta/kubernetes-java-sample/blob/master/maven/webapp/pom.xml#L57-L68. Specifically, the command for the Docker image is:

This ensures that COUCHBASE_URI environment variable is overriding spring.couchbase.bootstrap-hosts property as defined in application.properties of the Spring Boot application.

Kubernetes Dashboard

Kubernetes 1.4 included an updated dashboard. For minikube, this can be opened using the following command:

 

The default view is shown below:
minikube-dashboard-1-4

But in our case, a few resources have already been created and so this will look like as shown:

minikube-dashboard-couchbase

Notice, our Jobs, Replication Controllers and Pods are shown here.

Shutdown Kubernetes Cluster

The cluster can be easily shutdown:

couchbase.com/containers provide more details about running Couchbase using different orchestration frameworks. Further references:

  • Couchbase Forums or StackOverflow
  • Follow us at @couchbasedev or @couchbase
  • Read more about Couchbase Server

Source: blog.couchbase.com/2016/september/minikube-rapid-dev–testing-kubernetes

Getting Started with Kubernetes 1.4 using Spring Boot and Couchbase

Kubernetes 1.4 was released earlier this week. Read the blog announcement and CHANGELOG. There are quite a few new features in this release but the key ones that I’m excited about are:

  • Install Kubernetes using kubeadm command. This is in addition to the usual mechanism of downloading from https://github.com/kubernetes/kubernetes/releases. The kubeadm init and kubeadm join commands looks very similar to docker swarm init and docker swarm join for Docker Swarm Mode.
  • Federated Replica Sets
  • ScheduledJob allows to run batch jobs at regular intervals.
  • Constraining pods to a node and affinity and anti-affinity of pods
  • Priority scheduling of pods
  • Nice looking Kubernetes Dashboard (more on this later)

This blog will show:

  • Create a Kubernetes cluster using Amazon Web Services
  • Create a Couchbase service
  • Run a Spring Boot application that stores a JSON document in Couchbase

All the resource description files in this blog are at github.com/arun-gupta/kubernetes-java-sample/tree/master/maven.

Start Kubernetes Cluster

Download binary github.com/kubernetes/kubernetes/releases/download/v1.4.0/kubernetes.tar.gz and extract

Include kubernetes/cluster in PATH

Start a 2-node Kubernetes cluster:

The log will be shown as:

This shows that the Kubernetes cluster has started successfully.

Deploy Couchbase Service

Create Couchbase service and replication controller:

The configuration file is at github.com/arun-gupta/kubernetes-java-sample/blob/master/maven/couchbase-service.yml.

This creates a Couchbase service and the backing replication controller. Name of the service is couchbase-service. This will be used later by the Spring Boot application to communicate with the database.

Check the status of pods:

Note, how the pod status changes from ContainerCreating to Running. The image is downloaded and started in the meanwhile.

Run Spring Boot Application

Run the application:

The configuration file is at github.com/arun-gupta/kubernetes-java-sample/blob/master/maven/bootiful-couchbase.yml. In this service, COUCHBASE_URI environment variable value is set to couchbase-service. This is the service name created earlier.

Docker image used for this service is arungupta/bootiful-couchbase and is created using fabric8-maven-plugin as shown at github.com/arun-gupta/kubernetes-java-sample/blob/master/maven/webapp/pom.xml#L57-L68. Specifically, the command for the Docker image is:

This ensures that COUCHBASE_URI environment variable is overriding spring.couchbase.bootstrap-hosts property as defined in application.properties of the Spring Boot application.

Get the logs:

The main output statement to look in this is

This indicates that the JSON document is upserted (either inserted or updated) in the Couchbase database.

Kubernetes Dashboard

Kubernetes Dashboard is look more comprehensive and claimed to have 90% parity with the CLI. Use kubectl.sh config view command to view the configuration information about the cluster. It looks like:

The clusters.cluster.server property value shows the location of Kubernetes master. The users property show two users that can be used to access the dashboard. Second one uses basic authentication and so copy the username and password property value. In our case, Dashboard UI is accessible at https://52.40.9.27/ui.

kubernetes-dashboard-1-4

All the Kubernetes resources can be easily seen in this fancy dashboard.

Shutdown Kubernetes Cluster

Finally, shutdown the Kubernetes cluster:

couchbase.com/containers provide more details about running Couchbase using different orchestration frameworks.

Further references:

  • Couchbase Forums or StackOverflow
  • Follow us at @couchbasedev or @couchbase
  • Read more about Couchbase Server

Source: blog.couchbase.com/2016/september/kubernetes-1.4-spring-boot-couchbase

Deployment Pipeline using Docker, Jenkins, Java and Couchbase

This blog explains how to create a Deployment Pipeline using Jenkins and Docker for a Java application talking to a database.

Jenkins support the creation of pipelines. They are built with simple text scripts that use a Pipeline DSL (domain-specific language) based on the Groovy programming language.

The script, typically called Jenkinsfile, defines multiple steps to execute both simple and complex tasks according to the parameters that you establish. Once created, pipelines can build code and orchestrate the work required to drive applications from commit to delivery.

A pipeline consists of steps, node and stage. A pipeline is executed on a node – a computer that is part of Jenkins installation. A pipeline often consists of multiple stages. A stage consists of multiple steps. Read Getting Started with Pipeline for more details.

For our application, here is the basic flow:

docker-pipeline-jenkins

Complete source code for the application used is at github.com/arun-gupta/docker-jenkins-pipeline.

The application is defined in the webapp directory. It opens a connection to the Couchbase database and stores a simple JSON document using Couchbase Java SDK. The application also has a test that verifies that the database indeed contains the document that was persisted.

Many thanks to @alexsotob for helping me with Jenkins configuration.

Let’s get started!

Download and Install Jenkins

  • Download Jenkins from jenkins.io. This was tested with Jenkins 2.21.
  • Start Jenkins:
    This command starts Jenkins by specifying the home directory where all the configuration information is stored. It also defines the port on which Jenkins is listening, 9090 in this case.
  • First start of Jenkins shows the following message in the console:
    Copy the password shown here. This will be used to unlock Jenkins.
  • Access the Jenkins console at localhost:9090 and paste the password:docker-pipeline-jenkins-unlockClick on Next.
  • Create the first admin user as shown:
    docker-pipeline-jenkins-create-admin-user
    Click on Save and Finish.
  • Click on Install suggested plugins:docker-pipeline-jenkins-install-suggested-plugins
    A bunch of default plugins are installed:docker-pipeline-jenkins-installing-suggested-plugins
    Found it surprising that Ant and Subversion are the default plugins.
  • Login screen is prompted.
    docker-pipeline-jenkins-login
    Enter the username and password specified earlier.
  • Finally, Jenkins is ready to use:
    docker-pipeline-jenkins-start-using

That’s quite a bit of steps to get started with basic Jenkins. Do I really have to jump through all these hoops to get started with Jenkins? Is there an easier, simpler, dumber, lazier way to start Jenkins? Follow Convention-over-Configuration and give me one-click pre-configured installation.

Install Jenkins Plugins

Install the required plugins in Jenkins.

  1. If your Java project is built using Maven, then you need to configure Maven in Jenkins. Click on Manage Jenkins, Global Tool Configuration, Maven installations, and specify the location of Maven.docker-pipeline-jenkins-configure-maven
    Name the tool as Maven3 as that is the name used in the configuration later.Again a bit lame, why can’t Jenkins pick up the default location of Maven instead of expecting the user to specify a location.
  2. Click on Manage Jenkins, Manage Plugins, Available tab, search for docker pipe. Select CloudBees Docker Pipeline, click on Install without restart.
    docker-pipeline-jenkins-pipeline-plugin
    Click on Install without restart.Docker Pipeline Plugin plugin understands the Jenkinsfile and executes the commands listed there.
  3. Next screen shows the list of plugins that are installed:docker-pipeline-jenkins-pipeline-plugin-restart-jenkins
    The last line shows that CloudBees Docker Pipeline plugin is installed successfully. Select Restart Jenkins checkbox. This will install restart Jenkins as well.

Create Jenkins Job

Let’s create a job in Jenkins that will run the pipeline.

  1. After Jenkins restarts, it shows the login screen. Enter the username and password created earlier. This brings you back to Installing Plugins/Upgrades page. Click on the Jenkins icon in the top left corner to see the main dashboard:docker-pipeline-jenkins-dashboard
  2. Click on create new jobs, give the name as docker-jenkins-pipeline and choose the type as Pipeline:docker-pipeline-jenkins-create-projectClick on OK.
  3. Configure Pipeline as shown:
    docker-pipeline-jenkins-configure-pipelineLocal git repo is used in this case. You can certainly choose a repo hosted on github. Further, this repo can be configured with a git hook or poll at a constant interval to trigger the pipeline.Click on Save to save the configuration.

Run Jenkins Build

Before you start the job, Couchbase database need to be explicitly started as:

This will be resolved after #9 is fixed.  Make sure you can access Couchbase at http://localhost:8091, use Administrator as the login and password as the password. Click on Data Buckets tab and see the books bucket created.

docker-pipeline-couchbase-books

Click on Build Now and you should see an output similar to:

docker-pipeline-jenkins-build-run

All green is good!

Let’s try to understand what happened behind the scene.

Jenkinsfile describes how the pipeline is built. At the top level, it has four stages – Package, Create Docker Image, Run Application and Run Tests. Each stage is shown as a box in Jenkins dashboard. Total time taken for each stage is shown in the box.

Let’s understand what happens in each stage.

  • Package – Application source code lives in the webapp directory. Maven command mvn clean package -DskipTests is used to create a JAR file of the application. Note that the maven project also includes the tests and are explicitly skipped using -DskipTests. Typically, tests would be in a separate downstream project.Maven project creates a far JAR file of the application and includes all the dependencies.
  • Create Docker Image – Docker image of the application is built using the Dockerfile in the webapp directory. The image simply includes the fat JAR and runs it using java -jar.Each image is tagged with the build number using ${env.BUILD_NUMBER}.
  • Run Application – Running the application involves running the application Docker container.IP address of the database container is identified using the docker inspect command.The database container and the application container are both running in the default bridge network. This allows the two containers to communicate with each other. Another enhancement would be to run the pipeline in a swarm mode cluster. This would require to create and use an overlay network.
  • Run Tests – Tests are run against the container using the mvn test command. If the tests pass the image is pushed to Docker Hub. The test results are captured either way.This stage also shows the usage of try/catch/finally block in Jenkinsfile.If the tests pass then the image is pushed to Docker Hub. In this case, it is available at hub.docker.com/r/arungupta/docker-jenkins-pipeline/tags/.

Some TODOs …

  • Move the tests to a downstream project (#7)
  • Use Git hook or poll to trigger pipeline (#8)
  • Automate database startup/shutdown (#9)
  • Run pipeline in a cluster of Docker Engines with Swarm mode (#10)
  • Show alternate configuration to push image to bintray (#11)

Another pain point is that global variables syntax does not seem to be documented anywhere. It is only available at <JENKINS-HOST>:<JENKINS-PORT>/job/docker-jenkins-pipeline/pipeline-syntax/globals. This is again slightly lame!

“not impossible, just not implemented yet” #sadpanda

Some further references to read:

  • Getting Started with the Jenkinsfile
  • CloudBees Docker Pipeline Plugin
  • CloudBees Docker Pipeline Plugin User Guide
  • Jenkinsfile DSL Reference
  • Jenkins Pipeline Talk from JavaZone 2016

More information about Couchbase:

  • Couchbase Developer Portal
  • Couchbase Forums
  • @couchbasedev or @couchbase

Feel free to file bugs at github.com/arun-gupta/docker-jenkins-pipeline/issues or send PR.

Source: blog.couchbase.com/2016/september/deployment-pipeline-docker-jenkins-java-couchbase

Docker Service and Swarm Mode to Create Couchbase Cluster

Docker 1.12 introduced Services. A replicated, distributed and load balanced service can be easily created using docker service create command. A “desired state” of the application, such as run 3 containers of Couchbase, is provided and the self-healing Docker engine ensures that that many containers are running in the cluster. If a container goes down, another container is started. If a node goes down, containers on that node are started on a different node.

This blog will show how to setup a Couchbase cluster using Docker Services.

Many thanks to @marcosnils, another fellow Docker Captain, to help me debug the networking!

Couchbase Cluster

A cluster of Couchbase Servers is typically deployed on commodity servers. Couchbase Server has a peer-to-peer topology where all the nodes are equal and communicate to each other on demand. There is no concept of master nodes, slave nodes, config nodes, name nodes, head nodes, etc, and all the software loaded on each node is identical. It allows the nodes to be added or removed without considering their “type”. This model works particularly well with cloud infrastructure in general.

A typical Couchbase cluster creation process looks like:
  • Start Couchbase: Start n Couchbase servers
  • Create cluster: Pick any server, and add all other servers to it to create the cluster
  • Rebalance cluster: Rebalance the cluster so that data is distributed across the cluster
In order to automate using Docker Services, the cluster creation is split into a “master” and “worker” service.
docker-service-couchbase-cluster
The master service has only one replica. This provides a single reference point to start the cluster creation. This service also exposes port 8091. It allows the Couchbase Web Console to be accessible from outside the cluster.
The worker service uses the exact same image as master service. This keeps the cluster homogenous which allows to scale the cluster easily.
Let’s get started!

Setup Swarm Mode on Ubuntu

  1. Launch an Ubuntu instance on Amazon. This blog used mx4.large size for the AMI.
  2. Install Docker:
  3. Docker Swarm mode is an optional feature and need to be explicitly enabled. Initialize Swarm mode:

Create Couchbase “master” Service

  1. Create an overlay network:
    This is required so that multiple Couchbase Docker containers in the cluster can talk to each other.
  2. Create a “master” service:
    This image is created using the Dockerfile here. This Dockerfile uses a configuration script to configure the base Couchbase Docker image. First, it uses Couchbase REST API to setup memory quota, setup index, data and query services, security credentials, and loads a sample data bucket. Then, it invokes the appropriate Couchbase CLI commands to add the Couchbase node to the cluster or add the node and rebalance the cluster. This is based upon three environment variables:
    • TYPE: Defines whether the joining pod is worker or master
    • COUCHBASE_MASTER: Name of the master service
    • AUTO_REBALANCE: Defines whether the cluster needs to be rebalanced
    For this first configuration file, the TYPE environment variable is set to MASTER and so no additional configuration is done on the Couchbase image.

    This service also uses the previously created overlay network named couchbase. It exposes the port 8091 that makes the Couchbase Web Console accessible outside the cluster. This service contains only one replica of the container.

  3. Check status of the Docker service:

    It shows that the service is running. The “desired” and “expected” number of replicas are 1, and thus are matching.

  4. Check the tasks in the service:

    This shows that the container is running.

  5. Access Couchbase Web Console using public IP address and it should look like:docker-service-couchbase-login
    The image used in the configuration file is configured with the Administrator username and password password. Enter the credentials to see the console:
    docker-service-couchbase-web-console
  6. Click on Server Nodes to see how many Couchbase nodes are part of the cluster. As expected, it shows only one node:docker-service-couchbase-one-active-server

Create Couchbase “worker” Service

  1. Create “worker” service:

    This RC also creates a single replica of Couchbase using the same arungupta/couchbase:swarm image. The key differences here are:

    • TYPE environment variable is set to WORKER. This adds a worker Couchbase node to be added to the cluster.
    • COUCHBASE_MASTER environment variable is passed the name of the master service,  couchbase-master.couchbase in our case. This uses the service discovery mechanism built into Docker for the worker and the master to communicate.
  2. Check service:
  3. Checking the Couchbase Web Console shows the updated output:
    docker-service-couchbase-one-pending-server
    It shows that one server is pending to be rebalanced.During the worker service creation, AUTO_REBALANCE environment variable could have been set to true or false to enable rebalance. This ensures that the node is only added to the cluster but the cluster itself is not rebalanced. Rebalancing the cluster requires to re-distribute the data across multiple nodes of the cluster. The recommended way is to add multiple nodes, and then manually rebalance the cluster using the Web Console.

Add Couchbase Nodes by Scaling Docker Service

  1. Scale the service: 

  2. Check the service:
    This shows that 2 replicas of worker are running.
  3. Check the Couchbase Web Console:
    docker-service-couchbase-two-pending-serversAs expected, two servers are now added in the cluster and pending rebalance.
  4. Optionally, you can rebalance the cluster by clicking on the Rebalance button. and it will show like:docker-service-couchbase-rebalancingAfter the rebalancing is complete, the Couchbase Web Console is updated to as as shown:
    docker-service-couchbase-rebalanced
  5. See all the running containers using docker ps:

In addition to creating a cluster, Couchbase Server supports a range of high availability and disaster recovery (HA/DR) strategies. Most HA/DR strategies rely on a multi-pronged approach of maximizing availability, increasing redundancy within and across data centers, and performing regular backups.

Now that your Couchbase cluster is ready, you can run your first sample application.

Learn more about Couchbase and Containers:

  • Couchbase on Containers
  • Follow us on @couchbasedev or @couchbase
  • Ask questions on Couchbase Forums

Source: http://blog.couchbase.com/2016/september/docker-service-swarm-mode-couchbase-cluster

Stateful Containers on Kubernetes using Persistent Volume and Amazon EBS

This blog will show how to create stateful containers in Kubernetes using Amazon EBS.

Couchbase is a stateful container. This means that state of the container needs to be carried with it. In Kubernetes, the smallest atomic unit of running a container is a pod. So a Couchbase container will run as a pod. And by default, all data stored in Couchbase is stored on the same host.

stateful containers

This figure is originally explained in Kubernetes Cluster on Amazon and Expose Couchbase Service. In addition, this figure shows storage local to the host.

Pods are ephemeral and may be restarted on a different host. A Kubernetes Volume outlives any containers that run within the pod, and data is preserved across container restarts. However the volume will cease to exist when a pod ceases to exist. This is solved by Persistent Volumes that provide persistent, cluster-scoped storage for applications that require long lived data.

Creating and using a persistent volume is a three step process:

  1. Provision: Administrator provision a networked storage in the cluster, such as AWS ElasticBlockStore volumes. This is called as PersistentVolume.
  2. Request storage: User requests storage for pods by using claims. Claims can specify levels of resources (CPU and memory), specific sizes and access modes (e.g. can be mounted once read/write or many times write only). This is called as PersistentVolumeClaim.
  3. Use claim: Claims are mounted as volumes and used in pods for storage.

Specifically, this blog will show how to use an AWS ElasticBlockStore as PersistentVolume, create a PersistentVolumeClaim, and then claim it in a pod.

stateful containers

Complete source code for this blog is at: github.com/arun-gupta/couchbase-kubernetes.

Provision AWS Elastic Block Storage

Following restrictions need to be met if Amazon ElasticBlockStorage is used as a PersistentVolume with Kubernetes:

  • the nodes on which pods are running must be AWS EC2 instances
  • those instances need to be in the same region and availability-zone as the EBS volume
  • EBS only supports a single EC2 instance mounting a volume

Create an AWS Elastic Block Storage:

The region us-west-2 region and us-west-2a availability zone is used here. And so Kubernetes cluster need to start in the same region and availability zone as well.

This shows the output as:

Check if the volume is available as:

It shows the output as:

Note the unique identifier for the volume in VolumeId attribute. You can also verify the EBS block in AWS Console:

kubernetes-pv-couchbase-amazon-ebs

Start Kubernetes Cluster

Download Kubernetes 1.3.3, untar it and start the cluster on Amazon:

Three points to note here:

  • Zone in which the cluster is started is explicitly set to us-west-1a. This matches the zone where EBS storage volume was created.
  • By default, each node size is m3.medium. Here is is set to m3.large.
  • By default, 1 master and 4 worker nodes are created. Here only 3 worker nodes are created.

This will show the output as:

Read more details about starting a Kubernetes cluster on Amazon.

Couchbase Pod w/o Persistent Storage

Let’s create a Couchbase pod without persistent storage. This means that if the pod is rescheduled on a different host then it will not have access to the data created on it.

Here are quick steps to run a Couchbase pod and expose it outside the cluster:

Read more details at Kubernetes cluster at Amazon.

The last command shows the ingress load balancer address. Access Couchbase Web Console at <ip>:8091.

kubernetes-pv-couchbase-amazon-elb

Login to the console using Administrator login and password password.

The main page of Couchbase Web Console shows up:

kubernetes-pv-couchbase-amazon-web-console

A default travel-sample bucket is already created by arungupta/couchbase image. This bucket is shown in the Data Buckets tab:

kubernetes-pv-couchbase-amazon-databucket

Click on Create New Data Bucket button to create a new data bucket. Give it a name k8s, take all the defaults, and click on Create button to create the bucket:

kubernetes-pv-couchbase-amazon-k8s-bucket

Created bucket is shown in the Data Buckets tab:

kubernetes-pv-couchbase-amazon-k8s-bucket-created

Check status of the pod:

Delete the pod:

Watch the new pod being created:

Access the Web Console again and see that the bucket does not exist:

kubernetes-pv-couchbase-amazon-k8s-bucket-gone

Let’s clean up the resources created:

Couchbase Pod with Persistent Storage

Now, lets expose a Couchbase pod with persistent storage. As discussed above, lets create a PersistentVolume and claim the volume.

Request storage

Like any other Kubernetes resources, a persistent volume is created by using a resource description file:

The important pieces of information here are:

  • Creating a storage of 5 GB
  • Storage can be mounted by only one node for reading/writing
  • specifies the volume id created earlier

Read more details about definition of this file at kubernetes.io/docs/user-guide/persistent-volumes/.

This file is available at: github.com/arun-gupta/couchbase-kubernetes/blob/master/pv/couchbase-pv.yml.

The volume itself can be created as:

and shows the output:

Use claim

A PersistentVolumeClaim can be created using this resource file:

In our case, both PersistentVolume and PersistentVolumeClaim are 5 GB but they don’t have to be.

Read more details about definition of this file at kubernetes.io/docs/user-guide/persistent-volumes/#persistentvolumeclaims.

This file is at github.com/arun-gupta/couchbase-kubernetes/blob/master/pv/couchbase-pvc.yml.

The claim can be created as:

and shows the output:

Create RC with Persistent Volume Claim

Create a Couchbase Replication Controller using this resource file:

Key parts here are:

  • Resource defines a Replication Controller using arungupta/couchbase Docker image
  • volumeMounts define which volumes are going to be mounted. /opt/couchbase/var is the directory where Couchbase stores all the data.
  • volumes define different volumes that can be used in this RC definition

Cr