Tag Archives: docker

Deploy Docker Compose Services to Swarm

Docker 1.13 introduced a new version of Docker Compose. The main feature of this release is that it allow services defined using Docker Compose files to be directly deployed to Docker Engine enabled with Swarm mode. This enables simplified deployment of multi-container application on multi-host.

Docker 1.13

This blog will show use a simple Docker Compose file to show how services are created and deployed in Docker 1.13.

Here is a Docker Compose v2 definition for starting a Couchbase database node:

This definition can be started on a Docker Engine without Swarm mode as:

This will start a single replica of the service define in the Compose file. This service can be scaled as:

If the ports are not exposed then this would work fine on a single host. If swarm mode is enabled on on Docker Engine, then it shows the message:

Docker Compose gives us multi-container applications but the applications are still restricted to a single host. And that is a single point of failure.

Swarm mode allows to create a cluster of Docker Engines. With 1.13, docker stack deploy command can be used to deploy a Compose file to Swarm mode.

Here is a Docker Compose v3 definition:

As you can see, the only change is the value of version attribute. There are other changes in Docker Compose v3. Also, read about different Docker Compose versions and how to upgrade from v2 to v3.

Enable swarm mode:

Other nodes can join this swarm cluster and this would easily allow to deploy the multi-container application to a multi-host as well.

Deploy the services defined in Compose file as:

A default value of Compose file here would make the command a bit shorter. #30352 should take care of that.

List of services running can be verified using docker service ls command:

The list of containers running within the service can be seen using docker service ps command:

In this case, a single container is running as part of the service. The node is listed as moby which is the default name of Docker Engine running using Docker for Mac.

The service can now be scaled as:

The list of container can then be seen again as:

Note that the containers are given the name using the format <service-name>_n. Both the containers are running on the same host.

Also note, the two containers are independent Couchbase nodes and are not configured in a cluster yet. This has already been explained at Couchbase Cluster using Docker and a refresh of the steps is coming soon.

A service will typically have multiple containers running spread across multiple hosts. Docker 1.13 introduces a new command docker service logs <service-name> to stream the log of service across all the containers on all hosts to your console. In our case, this can be seen using the command docker service logs couchbase_db and looks like:

The preamble of the log statement uses the format <container-name>.<container-id>@<host>. And then actual log message from your container shows up.

At first instance, attaching container id may seem redundant. But Docker services are self-healing. This means that if a container dies then the Docker Engine will start another container to ensure the specified number of replicas at a given time. This new container will have a new id. And thus it allows  to attach the log message from the right container.

So a quick comparison of commands:

 Docker Compose v2  Docker compose v3
 Start services docker-compose up -d docker stack deploy --compose-file=docker-compose.yml <stack-name> 
 Scale service docker-compose scale <service>=<replicas> docker service scale <service>=<replicas>
 Shutdown docker-compose down docker stack rm <stack-name>
 Multi-host No Yes

Want to get started with Couchbase? Look at Couchbase Starter Kits.

Want to learn more about running Couchbase in containers?

Source: https://blog.couchbase.com/2017/deploy-docker-compose-services-swarm

Docker 1.13 Management Commands

Docker 1.13 was released yesterday, congratulations!

Docker 1.13

A quick summary of the key features:

  • Compose file to deploy Swarm mode services
  • Improved CLI backwards compatibility
  • Clean-up commands
  • CLI restructured
  • Monitoring and Build improvements

Learn more details about these features in this video by @manomarks:

Getting Started with Docker 1.13

Use Docker for Mac or Windows to get started. Once installed, the version information looks like:

Problems with Docker CLI

Docker 1.12 CLI has about ~40 top-level solo commands. While these commands wokred very well but they had a few issues:

  1. The commands are listed in one list without any organization. That makes it difficult for newbies to get started and learn the commands. (#8756)
  2. The command, such as docker inspect, also does not provide enough context whether they are operating on image or container. This mixing of image and container commands can cause confusion. (#13509)
  3. There is no consistency of command names. For example docker images is a plural and gives the list of images where as docker ps is singular and gives the list of containers. And they of course have the naming inconsistency issue. (#8829)
  4. Some of the commands like build and run are used heavily and then some arcane ones like pause and wait not so often. It does not seem fair to keep all the commands at the same level.

Docker 1.13 fixes this problem!

Docker Management Commands

Docker 1.13 groups the commands logically into management commands.

Here are the top-level solo commands now:

Now a list of images is obtained using docker image ls command instead of docker images command. Similar docker container ls shows the list of containers instead of docker ls. This brings a lot of consistency across the commands and that would make it intuitive and easier for newbie and pros to remember the commands.

Each management command has some similar set of sub-commands where they perform the operation on the command category:

Sub-command Purpose
ls List <category> (image, container, volume, secret, etc)
rm Remove <category>
inspect Inspect <category>

And there are other sub-commands based upon the management category.

Some of the heavily used commands are still at the top level.

By default, all the top-level commands are also shown. But you can set the DOCKER_HIDE_LEGACY_COMMANDS environment variable to show only the management commands. So even though docker --help will show all the solo and management commands. But the following commands will only show the new management commands:

The old syntax is still supported but it recommended to start moving to new commands.

A new Couchbase container can be started as:

The list of images can be seen as:

Mapping Docker Solo to Management Commands

Let’s look at how the existing top-level commands match to the management commands:

1.12 1.13 Purpose
attach container attach Attach to a running container
build image build Build an image from a Dockerfile
commit container commit Create a new image from a container’s changes
cp container cp Copy files/folders between a container and the local filesystem
create container create Create a new container
diff container diff Inspect changes on a container’s filesystem
events system events Get real time events from the server
exec container exec Run a command in a running container
export container export Export a container’s filesystem as a tar archive
history image history Show the history of an image
images image ls List images
import image import Import the contents from a tarball to create a filesystem image
info system info Display system-wide information
inspect container inspect Return low-level information on a container, image or task
kill container kill Kill one or more running containers
load image load Load an image from a tar archive or STDIN
login login Log in to a Docker registry.
logout logout Log out from a Docker registry.
logs container logs Fetch the logs of a container
network network Manage Docker networks
node node Manage Docker Swarm nodes
pause container pause Pause all processes within one or more containers
port container port List port mappings or a specific mapping for the container
ps container ls List containers
pull image pull Pull an image or a repository from a registry
push image push Push an image or a repository to a registry
rename container rename Rename a container
restart container restart Restart a container
rm container rm Remove one or more containers
rmi image rm Remove one or more images
run container run Run a command in a new container
save image save Save one or more images to a tar archive (streamed to STDOUT by default)
search search Search the Docker Hub for images
service service Manage Docker services
start container start Start one or more stopped containers
stats container stats Display a live stream of container(s) resource usage statistics
stop container stop Stop one or more running containers
swarm swarm Manage Docker Swarm
tag image tag Tag an image into a repository
top container top Display the running processes of a container
unpause container unpause Unpause all processes within one or more containers
update container update Update configuration of one or more containers
version version Show the Docker version information
volume volume Manage Docker volumes
wait container wait Block until a container stops, then print its exit code

Sign up for Docker Online Meetup on 1/25 at 10am PST for more details on Docker 1.13.

Use Docker for Mac or Windows to get started with Docker 1.13.

And of course, you can learn more about how to run Couchbase on Containers.

Source: https://blog.couchbase.com/2017/docker-1-13-management-commands

Health Check of Docker Containers

One of the new features in Docker 1.12 is how health check for a container can be baked into the image definition. And this can be overridden at the command line.

Just like the CMD instruction, there can be multiple HEALTHCHECK instructions in Dockerfile but only the last one is effective.

This is a great addition because a container reporting status as Up 1 hour may return errors. The container may be up but there is no way for the application inside the container to provide a status. This instruction fixes that.

The Dockerfile that builds arungupta/couchbase image is:

It uses configure-node.sh script to configure the server using Couchbase REST API. The new instruction to notice here is HEALTHCHECK.

This instruction can be specified as:

The <options> can be:

  • --interval=DURATION (default 30s)
  • --timeout=DURATION (default 30s)
  • --retries=N (default 3)

The <command> is the command that runs inside the container to check the health.

If health check is enabled, then the container can have three states:

  • starting – Initial status when the container is still starting
  • healthy – If the command succeeds then the container is healthy
  • unhealthy – If a single run of the <command> takes longer than the specified timeout then it is considered unhealthy. If a health check fails then the <command> will run retries number of times and will be declared unhealthy if the <command> still fails.

The commands exit status indicates the health status of the container. The following values are allowed:

  • 0 – container is healthy
  • 1 – container is not healthy

In our instruction, /pools REST API is invoked using curl. If the command fails then an exit status of 1 is returned, and this marks the container unhealthy for that attempt. This command is invoked every 5 seconds. The container is marked unhealthy if the command does not return successfully within 3 seconds.

Run the container as:

Check the status:

Notice how health: starting status is reported in the STATUS column. Checking after a few seconds shows the status:

And now its reported healthy.

More details about this HEALTHCHECK instruction can be found on docs.docker.com.

Now, if you are running an image that does not have HEALTHCHECK instruction then the docker run command can be used to specify similar values. An equivalent runtime command would be:

Last 5 health checks for a container can be obtained using the docker inspect command:

The output is shown as:

 

Source: http://blog.couchbase.com/2016/november/docker-health-check-keeping-containers-healthy

 

Docker for AWS – Getting Started Video

Want to create a highly-available Docker cluster on Amazon Web Services? Run multi-container applications on it using Docker Services?

Docker Logo
amazon-web-services-logo
Couchbase Logo

Docker for AWS allows you to exactly do that! This video shows:

  • Create a highly-available Docker cluster on Amazon Web Services (0:00)
  • Check configuration (5:43)
  • Use Docker services to create a Couchbase cluster (8:23)

Enjoy!

couchbase.com/containers provide more details about how to run Couchbase in different container frameworks. More information about Couchbase:

Source: blog.couchbase.com/2016/november/docker-for-aws-getting-started-video

Docker Container Anti Patterns

This blog will explain 10 containers anti-patterns that I’ve seen over the past few months:

  1. Data or logs in containers – Containers are ideal for stateless applications and are meant to be ephemeral. This means no data or logs should be stored in the container otherwise they’ll be lost when the container terminates. Instead use volume mapping to persist them outside the containers. ELK stack could be used to store and process logs. If managed volumes are used during early testing process, then remove them using -v switch with the docker rm command.
  2.  IP addresses of container – Each container is assigned an IP address. Multiple containers communicate with each other to create an application, for example an application deployed on an application server will need to talk with a database. Existing containers are terminated and new containers are started all the time. Relying upon the IP address of the container will require constantly updating the application configuration. This will make the application fragile. Instead create services. This will provide a logical name that can be referred independent of the growing and shrinking number of containers. And it also provides a basic load balancing as well.
  3. Run a single process in a container – A Dockerfile has use one CMD and ENTRYPOINT. Often, CMD will use a script that will perform some configuration of the image and then start the container. Don’t try to start multiple processes using that script. Its important to follow separation of concerns pattern when creating Docker images. This will make managing your containers, collecting logs, updating each individual process that much harder. You may consider breaking up application into multiple containers and manage them independently.
  4. Don’t use docker exec – The docker exec command starts a new command in a running container. This is useful for attaching a shell using the docker exec -it {cid} bash. But other than that the container is already running the process that its supposed to be running.
  5. Keep your image lean – Create a new directory and include Dockerfile and other relevant files in that directory. Also consider using .dockerignore to remove any logs, source code, logs etc before creating the image. Make sure to remove any downloaded artifacts after they are unzipped.
  6. Create image from a running container – A new image can be created using the docker commit command. This is useful when any changes in the container have been made. But images created using this are non-reproducible. Instead make changes in the Dockerfile, terminate existing containers and start a new container with the updated image.
  7. Security credentials in Docker image – Do not store security credentials in the Dockerfile. They are in clear text and checked into a repository. This makes them completely vulnerable. Use -e to specify passwords as runtime environment variable. Alternatively --env-file can be used to read environment variables from a file. Another approach is to used CMD or ENTRYPOINT to specify a script. This script will pull the credentials from a third party and then configure your application.
  8. latest tag: Starting with an image like couchbase is tempting. If no tags are specified then a container is started using the image couchbase:latest.  This image may not actually be latest and instead refer to an older version. Taking an application into production requires a fully controller environment with exact version of the image. Read this Docker: The latest confusion post by fellow Docker Captain @adrianmouat.  Make sure to always use tag when running a container. For example, use couchbase:enterprise-4.5.1 instead of just couchbase.
  9. Impedance mismatch – Don’t use different images, or even different tags in dev, test, staging and production environment. The image that is the “source of truth” should be created once and pushed to a repo. That image should be used for different environments going forward. In some cases, you may consider running your unit tests on the WAR file as part of maven build and then create the image. But any system integration testing should be done on the image that will be pushed in production.
  10. Publishing ports – Don’t use -P to publish all the exposed ports. This will allow you to run multiple containers and publish their exposed ports. But this also means that all the ports will be published. Instead use -p to publish specific ports.

Adding more based upon discussion on twitter …

  1. Root user – Don’t run containers as root user. The host and the container share the same kernel. If the container is compromised, a root user can do more damage to the underlying hosts. Use RUN groupadd -r couchbase && useradd -r -g couchbase couchbase to create a group and a user in it. Use the USER instruction to switch to that user. Each USER creates a new layer in the image. Avoid switching the user back and forth to reduce the number of layers. Thanks to @Aleksandar_78 for this tip!
  2. Dependency between containers – Often applications rely upon containers to be started in a certain order. For example, a database container must be up before an application can connect to it. The application should be resilient to such changes as the containers may be terminated or started at any time. In this case, have the application container wait for the database connection to succeed before proceeding further. Do not use wait-for scripts in Dockerfile for the containers to startup in a specific order. Particularly waiting for a certain number of seconds for a particular container to start is very fragile. Thanks to @ratnopam for this tip!

What other anti-patterns do you follow?

Docker for Java developers is a self-paced hands-on workshop that explains how to get started with Docker for Java developers.

Interested in a more deep dive tutorial? Watch this 2-hours tutorial from JavaOne!

couchbase.com/containers shows how to run Couchbase in a variety of frameworks.

Source: blog.couchbase.com/2016/october/docker-container-anti-patterns

Docker on Windows 2016 Server

This blog is the first part of a multi-part series. The first part showed how to set up Windows Server 2016 as a VirtualBox VM. This second part will show how to configure Docker on Windows 2016 VM.

  1. Start an elevated PowerShell session:docker-windows-2016-22
  2. Run the script to install Docker:
    This will install the PowerShell module, enable containers feature and install Docker.

    docker-windows-2016-23

    The VM needs to be restarted in order for the containers to be enabled. Refer to Container Host Deployment – Windows Server for more detailed instructions.

  3. The VM reboots. Start a PowerShell and check Docker version using docker version command:docker-windows-2016-24More details about Docker can be found using the docker info command:docker-windows-2016-25
  4. Run your first Docker container using the docker run -it -p 80:80 microsoft/iis command:docker-windows-2016-26This will download the Microsoft IIS server Docker image. This is going to take a while so please be patient!
  5. Once the 8.9 GB image is downloaded (after a while), the IIS server is started for you. Check the list of images using the docker images command and the list of running containers using the docker ps command:docker-windows-2016-27More details about the container can be found using the docker inspect command:

  6. The exact IP address of the container can be found using the command:

    IIS main page is accessible at http://<container-ip>, as shown below:

    docker-windows-2016-28

The next part will show how to create your own Docker image on Windows Server 2016.

Source: blog.couchbase.com/2016/october/docker-on-windows-2016-server

Persisting Couchbase Data Across Container Restarts

Best Practices for Virtualized Platforms provide best practices for running Couchbase on a virtualized platform like Amazon Web Services and Azure. In addition, it also provide some recommendations for running it as Docker container.

One of the recommendations is to map Couchbase node specific data to a local folder. Let’s understand that in more detail.

Implicit Per-Container Storage

If a Couchbase container is started as:

This container:

  • Starts in a detached mode using -d
  • Different query, caching and administration ports are mapped using -p
  • A name is provided using --name
  • Image is couchbase/server:sandbox

By default, the data for the container is stored in a managed volume. Checking volume mounts using the docker inspect command shows:

The data for Couchbase is stored in the container filesystem defined by the value of Source attribute. This can be verified by logging into the root filesystem:

Now you can see the data directory:

A new directory is created for a new run of the container. This directory is still around when the container is stopped and removed but no longer easily accessible. Thus no data is preserved across container restarts.

The volume can be explicitly removed, along with container, using the command:

If the container terminates then the entire state of the application is lost.

Explicit Host Directory Mapping

Now, let’s start a Couchbase container with explicit volume mapping:

This container is very similar to the container started earlier. The main difference is that a directory from host ~/couchbase is mapped to a directory in the container /opt/couchbase/var.

Couchbase container persists any data in /opt/couchbase/var directory in the container filesystem. Now that directory is mapped to a directory on the host filesystem. This allows to persist state of the container outside on the host filesystem. The bypasses the union filesystem used by Docker and exposes the host filesystem to the container. This allows the state to persist across container restarts. The new container only needs to start with the exact same volume mapping.

More details about the container can be seen as:

jq is a JSON processor that needs to be installed separately. And the output is shown as:

This shows the source and destination directory. RW shows that the volume is read/write.

If the container is started using Docker for Mac, then Couchbase Web Console is accessible at http://localhost:8091. The Data Buckets tab shows the default travel-sample bucket:

docker-volume-couchbase-01

Click on Create New Data Bucket to create a new data bucket. Give it the name sample:

docker-volume-couchbase-02

The Data Buckets tab is updated with this newly created bucket:

docker-volume-couchbase-03

Now stop and remove the container:

Start the container again using the same command:

Data Buckets tab will show the same two buckets in the Couchbase Web Console.

 

In this case, if the container is started on a different host then the state would not be available. Or if the host dies then the state is lost.

An alternative and a more robust and foolproof way to manage persistence in containers is using a shared network filesystem such as Ceph, GlusterFS or Network Filesystem. Some other common approaches are to use Docker Volume Plugins like Flocker from ClusterHQ or Software Defined Storage such as PortWorx. All of these storage technique simplify how state of a container can be saved in a multi-container multi-host environment. A future blog will cover these techniques in detail.

Read more details in Managing data in containers.

couchbase.com/containers provide more details about how to run Couchbase in different container frameworks.

More information about Couchbase:

Source: blog.couchbase.com/2016/october/persisting-couchbase-data-across-container-restarts

Windows Server 2016 Using VirtualBox – Getting Ready for Docker

Windows  Server 2016 was announced a few weeks ago.  Download Windows 2016 Server Evaluation version. This blog is the first part of a multi-part series. The first part will show how to set up Windows Server 2016 as a VirtualBox VM. This is an update to Windows Server 2016 using VirtualBox for Docker Containers.

  1. Download Windows 2016 Server Evaluation version.  Its ~5GB download so try on a reliable and fast Internet connection.
  2. Create a new VM using Virtual Box:

    docker-windows-2016-1

  3. Set up a dynamically allocated HDD, make sure to choose 30GB (as shown):

    docker-windows-2016-2
    The installed operating system is ~17GB. The base windowsservercore image is 3.8GB. So if you chose the default 20GB HDD, then no Docker containers can run on it.

  4. Point to the downloaded ISO:

    docker-windows-2016-3

  5. Start the VM, take default settings:

    docker-windows-2016-4

  6. Click on Next:

    docker-windows-2016-5

    Click on Install now.

  7. Select the Desktop Experience version otherwise the standard Windows desktop will not be available after Windows is booted:
    docker-windows-2016-6

    Click on Next.

  8. Accept the license terms:

    docker-windows-2016-7

  9. Select Custom Install as we are installing as a VM:

    docker-windows-2016-8

  10. Take the default for the allocated space:

    docker-windows-2016-9

  11. Click on Next to start the installation:

    docker-windows-2016-10

    Wait for a few minutes for the installation to complete:

    docker-windows-2016-11

    It will take a few minutes for the install to complete.

  12. Enter the administrator’s password:

    docker-windows-2016-13

    Seems like it requires alphabets, numbers and special characters. The dialog expects to meet the password criteria without showing the criteria, this is weird!

  13. Entering the credentials shows the Windows login screen:

    docker-windows-2016-14

  14. In a classical Windows, you do a three-finger salute of Ctrl+Alt+Del to view Desktop. This key combination needs to be sent to Windows VM using  Virtual Box menu:

    docker-windows-2016-15

  15. Server Manager Dashboard shows up:

    docker-windows-2016-16

  16. Latest updates need to be installed. Click on the Start icon, and search for update:

    docker-windows-2016-17Select Check for Updates.

  17. Update window looks like:

    docker-windows-2016-18Select Check for updates again.

  18. Latest updates are downloaded and installed:

    docker-windows-2016-19

  19. Once the updates are downloaded and installed, then the VM needs to be restarted:

    docker-windows-2016-20Click on Restart now to restart the VM.

  20. Restarted VM looks like as shown:

    docker-windows-2016-21

 

This shows that Windows Server 2016 VM is now ready. Subsequent parts of this multi-part blog will show how to configure Docker containers and run a multi-container application using Docker Compose.

Source: blog.couchbase.com/2016/october/windows-server-2016-using-virtualbox-getting-ready-docker

Deployment Pipeline using Docker, Jenkins, Java and Couchbase

This blog explains how to create a Deployment Pipeline using Jenkins and Docker for a Java application talking to a database.

Jenkins support the creation of pipelines. They are built with simple text scripts that use a Pipeline DSL (domain-specific language) based on the Groovy programming language.

The script, typically called Jenkinsfile, defines multiple steps to execute both simple and complex tasks according to the parameters that you establish. Once created, pipelines can build code and orchestrate the work required to drive applications from commit to delivery.

A pipeline consists of steps, node and stage. A pipeline is executed on a node – a computer that is part of Jenkins installation. A pipeline often consists of multiple stages. A stage consists of multiple steps. Read Getting Started with Pipeline for more details.

For our application, here is the basic flow:

docker-pipeline-jenkins

Complete source code for the application used is at github.com/arun-gupta/docker-jenkins-pipeline.

The application is defined in the webapp directory. It opens a connection to the Couchbase database and stores a simple JSON document using Couchbase Java SDK. The application also has a test that verifies that the database indeed contains the document that was persisted.

Many thanks to @alexsotob for helping me with Jenkins configuration.

Let’s get started!

Download and Install Jenkins

  • Download Jenkins from jenkins.io. This was tested with Jenkins 2.21.
  • Start Jenkins:
    This command starts Jenkins by specifying the home directory where all the configuration information is stored. It also defines the port on which Jenkins is listening, 9090 in this case.
  • First start of Jenkins shows the following message in the console:
    Copy the password shown here. This will be used to unlock Jenkins.
  • Access the Jenkins console at localhost:9090 and paste the password:docker-pipeline-jenkins-unlockClick on Next.
  • Create the first admin user as shown:
    docker-pipeline-jenkins-create-admin-user
    Click on Save and Finish.
  • Click on Install suggested plugins:docker-pipeline-jenkins-install-suggested-plugins
    A bunch of default plugins are installed:docker-pipeline-jenkins-installing-suggested-plugins
    Found it surprising that Ant and Subversion are the default plugins.
  • Login screen is prompted.
    docker-pipeline-jenkins-login
    Enter the username and password specified earlier.
  • Finally, Jenkins is ready to use:
    docker-pipeline-jenkins-start-using

That’s quite a bit of steps to get started with basic Jenkins. Do I really have to jump through all these hoops to get started with Jenkins? Is there an easier, simpler, dumber, lazier way to start Jenkins? Follow Convention-over-Configuration and give me one-click pre-configured installation.

Install Jenkins Plugins

Install the required plugins in Jenkins.

  1. If your Java project is built using Maven, then you need to configure Maven in Jenkins. Click on Manage Jenkins, Global Tool Configuration, Maven installations, and specify the location of Maven.docker-pipeline-jenkins-configure-maven
    Name the tool as Maven3 as that is the name used in the configuration later.Again a bit lame, why can’t Jenkins pick up the default location of Maven instead of expecting the user to specify a location.
  2. Click on Manage Jenkins, Manage Plugins, Available tab, search for docker pipe. Select CloudBees Docker Pipeline, click on Install without restart.
    docker-pipeline-jenkins-pipeline-plugin
    Click on Install without restart.Docker Pipeline Plugin plugin understands the Jenkinsfile and executes the commands listed there.
  3. Next screen shows the list of plugins that are installed:docker-pipeline-jenkins-pipeline-plugin-restart-jenkins
    The last line shows that CloudBees Docker Pipeline plugin is installed successfully. Select Restart Jenkins checkbox. This will install restart Jenkins as well.

Create Jenkins Job

Let’s create a job in Jenkins that will run the pipeline.

  1. After Jenkins restarts, it shows the login screen. Enter the username and password created earlier. This brings you back to Installing Plugins/Upgrades page. Click on the Jenkins icon in the top left corner to see the main dashboard:docker-pipeline-jenkins-dashboard
  2. Click on create new jobs, give the name as docker-jenkins-pipeline and choose the type as Pipeline:docker-pipeline-jenkins-create-projectClick on OK.
  3. Configure Pipeline as shown:
    docker-pipeline-jenkins-configure-pipelineLocal git repo is used in this case. You can certainly choose a repo hosted on github. Further, this repo can be configured with a git hook or poll at a constant interval to trigger the pipeline.Click on Save to save the configuration.

Run Jenkins Build

Before you start the job, Couchbase database need to be explicitly started as:

This will be resolved after #9 is fixed.  Make sure you can access Couchbase at http://localhost:8091, use Administrator as the login and password as the password. Click on Data Buckets tab and see the books bucket created.

docker-pipeline-couchbase-books

Click on Build Now and you should see an output similar to:

docker-pipeline-jenkins-build-run

All green is good!

Let’s try to understand what happened behind the scene.

Jenkinsfile describes how the pipeline is built. At the top level, it has four stages – Package, Create Docker Image, Run Application and Run Tests. Each stage is shown as a box in Jenkins dashboard. Total time taken for each stage is shown in the box.

Let’s understand what happens in each stage.

  • Package – Application source code lives in the webapp directory. Maven command mvn clean package -DskipTests is used to create a JAR file of the application. Note that the maven project also includes the tests and are explicitly skipped using -DskipTests. Typically, tests would be in a separate downstream project.Maven project creates a far JAR file of the application and includes all the dependencies.
  • Create Docker Image – Docker image of the application is built using the Dockerfile in the webapp directory. The image simply includes the fat JAR and runs it using java -jar.Each image is tagged with the build number using ${env.BUILD_NUMBER}.
  • Run Application – Running the application involves running the application Docker container.IP address of the database container is identified using the docker inspect command.The database container and the application container are both running in the default bridge network. This allows the two containers to communicate with each other. Another enhancement would be to run the pipeline in a swarm mode cluster. This would require to create and use an overlay network.
  • Run Tests – Tests are run against the container using the mvn test command. If the tests pass the image is pushed to Docker Hub. The test results are captured either way.This stage also shows the usage of try/catch/finally block in Jenkinsfile.If the tests pass then the image is pushed to Docker Hub. In this case, it is available at hub.docker.com/r/arungupta/docker-jenkins-pipeline/tags/.

Some TODOs …

  • Move the tests to a downstream project (#7)
  • Use Git hook or poll to trigger pipeline (#8)
  • Automate database startup/shutdown (#9)
  • Run pipeline in a cluster of Docker Engines with Swarm mode (#10)
  • Show alternate configuration to push image to bintray (#11)

Another pain point is that global variables syntax does not seem to be documented anywhere. It is only available at <JENKINS-HOST>:<JENKINS-PORT>/job/docker-jenkins-pipeline/pipeline-syntax/globals. This is again slightly lame!

not impossible, just not implemented yet” #sadpanda

Some further references to read:

More information about Couchbase:

Feel free to file bugs at github.com/arun-gupta/docker-jenkins-pipeline/issues or send PR.

Source: blog.couchbase.com/2016/september/deployment-pipeline-docker-jenkins-java-couchbase

Docker Service and Swarm Mode to Create Couchbase Cluster

Docker 1.12 introduced Services. A replicated, distributed and load balanced service can be easily created using docker service create command. A “desired state” of the application, such as run 3 containers of Couchbase, is provided and the self-healing Docker engine ensures that that many containers are running in the cluster. If a container goes down, another container is started. If a node goes down, containers on that node are started on a different node.

This blog will show how to setup a Couchbase cluster using Docker Services.

Many thanks to @marcosnils, another fellow Docker Captain, to help me debug the networking!

Couchbase Cluster

A cluster of Couchbase Servers is typically deployed on commodity servers. Couchbase Server has a peer-to-peer topology where all the nodes are equal and communicate to each other on demand. There is no concept of master nodes, slave nodes, config nodes, name nodes, head nodes, etc, and all the software loaded on each node is identical. It allows the nodes to be added or removed without considering their “type”. This model works particularly well with cloud infrastructure in general.

A typical Couchbase cluster creation process looks like:
  • Start Couchbase: Start n Couchbase servers
  • Create cluster: Pick any server, and add all other servers to it to create the cluster
  • Rebalance cluster: Rebalance the cluster so that data is distributed across the cluster
In order to automate using Docker Services, the cluster creation is split into a “master” and “worker” service.
docker-service-couchbase-cluster
The master service has only one replica. This provides a single reference point to start the cluster creation. This service also exposes port 8091. It allows the Couchbase Web Console to be accessible from outside the cluster.
The worker service uses the exact same image as master service. This keeps the cluster homogenous which allows to scale the cluster easily.
Let’s get started!

Setup Swarm Mode on Ubuntu

  1. Launch an Ubuntu instance on Amazon. This blog used mx4.large size for the AMI.
  2. Install Docker:
  3. Docker Swarm mode is an optional feature and need to be explicitly enabled. Initialize Swarm mode:

Create Couchbase “master” Service

  1. Create an overlay network:
    This is required so that multiple Couchbase Docker containers in the cluster can talk to each other.
  2. Create a “master” service:
    This image is created using the Dockerfile here. This Dockerfile uses a configuration script to configure the base Couchbase Docker image. First, it uses Couchbase REST API to setup memory quota, setup index, data and query services, security credentials, and loads a sample data bucket. Then, it invokes the appropriate Couchbase CLI commands to add the Couchbase node to the cluster or add the node and rebalance the cluster. This is based upon three environment variables:
    • TYPE: Defines whether the joining pod is worker or master
    • COUCHBASE_MASTER: Name of the master service
    • AUTO_REBALANCE: Defines whether the cluster needs to be rebalanced
    For this first configuration file, the TYPE environment variable is set to MASTER and so no additional configuration is done on the Couchbase image.

    This service also uses the previously created overlay network named couchbase. It exposes the port 8091 that makes the Couchbase Web Console accessible outside the cluster. This service contains only one replica of the container.

  3. Check status of the Docker service:

    It shows that the service is running. The “desired” and “expected” number of replicas are 1, and thus are matching.

  4. Check the tasks in the service:

    This shows that the container is running.

  5. Access Couchbase Web Console using public IP address and it should look like:docker-service-couchbase-login
    The image used in the configuration file is configured with the Administrator username and password password. Enter the credentials to see the console:
    docker-service-couchbase-web-console
  6. Click on Server Nodes to see how many Couchbase nodes are part of the cluster. As expected, it shows only one node:docker-service-couchbase-one-active-server

Create Couchbase “worker” Service

  1. Create “worker” service:

    This RC also creates a single replica of Couchbase using the same arungupta/couchbase:swarm image. The key differences here are:

    • TYPE environment variable is set to WORKER. This adds a worker Couchbase node to be added to the cluster.
    • COUCHBASE_MASTER environment variable is passed the name of the master service,  couchbase-master.couchbase in our case. This uses the service discovery mechanism built into Docker for the worker and the master to communicate.
  2. Check service:
  3. Checking the Couchbase Web Console shows the updated output:
    docker-service-couchbase-one-pending-server
    It shows that one server is pending to be rebalanced.During the worker service creation, AUTO_REBALANCE environment variable could have been set to true or false to enable rebalance. This ensures that the node is only added to the cluster but the cluster itself is not rebalanced. Rebalancing the cluster requires to re-distribute the data across multiple nodes of the cluster. The recommended way is to add multiple nodes, and then manually rebalance the cluster using the Web Console.

Add Couchbase Nodes by Scaling Docker Service

  1. Scale the service: 

  2. Check the service:
    This shows that 2 replicas of worker are running.
  3. Check the Couchbase Web Console:
    docker-service-couchbase-two-pending-serversAs expected, two servers are now added in the cluster and pending rebalance.
  4. Optionally, you can rebalance the cluster by clicking on the Rebalance button. and it will show like:docker-service-couchbase-rebalancingAfter the rebalancing is complete, the Couchbase Web Console is updated to as as shown:
    docker-service-couchbase-rebalanced
  5. See all the running containers using docker ps:

In addition to creating a cluster, Couchbase Server supports a range of high availability and disaster recovery (HA/DR) strategies. Most HA/DR strategies rely on a multi-pronged approach of maximizing availability, increasing redundancy within and across data centers, and performing regular backups.

Now that your Couchbase cluster is ready, you can run your first sample application.

Learn more about Couchbase and Containers:

Source: http://blog.couchbase.com/2016/september/docker-service-swarm-mode-couchbase-cluster

Docker, Minecraft, Rx Java, CI/CD, Couchbase at JavaOne 2016

javaone2016-session-catalog

JavaOne 2016 Session Catalog is now live!

There are 400+ sessions from Sep 17 (Saturday) through Sep 22 (Thursday). Here are some of the places where you’ll find me:

Day/Date Time Title Co-speaker
Saturday, Sep 17 8am – 5pm JavaOne4Kids Several
Sunday, Sep 18 10:30am – 11:15am Dive into the Newest, Latest, and Cutting-Edgiest Technologies in the NetBeans IDE [UGF6434]  @mr__m
 Monday, Sep 19 8:30am – 10:30am Docker for Java Developers [TUT1887]  –
Monday, Sep 19 12:30pm – 1:30pm Docker Support in NetBeans, Eclipse, and IntelliJ [CON4236]  –
Monday, Sep 19 TBD O’Reilly Media Book Signing
Monday, Sep 19 5:30pm – 6:30pm Mobycraft: Manage Docker Containers by Using Minecraft [CON1931]  @aditya_g
Wednesday, Sep 21 1pm – 2pm Full-Stack Reactive Java Applications with Docker [CON3987]  @nraboy
Thursday, Sep 22 2:30pm – 3:30pm Building a Private CI/CD Pipeline with Java and Docker on Oracle Cloud [CON7609]  @jbaruch

And then there is “hallway track”, parties, Couchbase booth (#5002), O’Reilly Media book signing, Java Hub, and other places.

Where will I see you?

j1-125x125-i-like-this-3089434j1-125x125-im-attending-3089435j1-125x125-im-speaking-3089437j1-125x125-im-demoing-3089436j1-125x125-see-me-here-3089441j1-125x125-join-me-3089439j1-125x125-we-exhibiting-3089443  j1-125x125-register-now-3089440

Complete set of these images are available at JavaOne Speaker Toolkit.

Source: blog.couchbase.com/2016/july/docker-minecraft-rxjava-cicd-couchbase-javaone-2016

Stateful Containers on Kubernetes using Persistent Volume and Amazon EBS

This blog will show how to create stateful containers in Kubernetes using Amazon EBS.

Couchbase is a stateful container. This means that state of the container needs to be carried with it. In Kubernetes, the smallest atomic unit of running a container is a pod. So a Couchbase container will run as a pod. And by default, all data stored in Couchbase is stored on the same host.

stateful containers

This figure is originally explained in Kubernetes Cluster on Amazon and Expose Couchbase Service. In addition, this figure shows storage local to the host.

Pods are ephemeral and may be restarted on a different host. A Kubernetes Volume outlives any containers that run within the pod, and data is preserved across container restarts. However the volume will cease to exist when a pod ceases to exist. This is solved by Persistent Volumes that provide persistent, cluster-scoped storage for applications that require long lived data.

Creating and using a persistent volume is a three step process:

  1. Provision: Administrator provision a networked storage in the cluster, such as AWS ElasticBlockStore volumes. This is called as PersistentVolume.
  2. Request storage: User requests storage for pods by using claims. Claims can specify levels of resources (CPU and memory), specific sizes and access modes (e.g. can be mounted once read/write or many times write only). This is called as PersistentVolumeClaim.
  3. Use claim: Claims are mounted as volumes and used in pods for storage.

Specifically, this blog will show how to use an AWS ElasticBlockStore as PersistentVolume, create a PersistentVolumeClaim, and then claim it in a pod.

stateful containers

Complete source code for this blog is at: github.com/arun-gupta/couchbase-kubernetes.

Provision AWS Elastic Block Storage

Following restrictions need to be met if Amazon ElasticBlockStorage is used as a PersistentVolume with Kubernetes:

  • the nodes on which pods are running must be AWS EC2 instances
  • those instances need to be in the same region and availability-zone as the EBS volume
  • EBS only supports a single EC2 instance mounting a volume

Create an AWS Elastic Block Storage:

The region us-west-2 region and us-west-2a availability zone is used here. And so Kubernetes cluster need to start in the same region and availability zone as well.

This shows the output as:

Check if the volume is available as:

It shows the output as:

Note the unique identifier for the volume in VolumeId attribute. You can also verify the EBS block in AWS Console:

kubernetes-pv-couchbase-amazon-ebs

Start Kubernetes Cluster

Download Kubernetes 1.3.3, untar it and start the cluster on Amazon:

Three points to note here:

  • Zone in which the cluster is started is explicitly set to us-west-1a. This matches the zone where EBS storage volume was created.
  • By default, each node size is m3.medium. Here is is set to m3.large.
  • By default, 1 master and 4 worker nodes are created. Here only 3 worker nodes are created.

This will show the output as:

Read more details about starting a Kubernetes cluster on Amazon.

Couchbase Pod w/o Persistent Storage

Let’s create a Couchbase pod without persistent storage. This means that if the pod is rescheduled on a different host then it will not have access to the data created on it.

Here are quick steps to run a Couchbase pod and expose it outside the cluster:

Read more details at Kubernetes cluster at Amazon.

The last command shows the ingress load balancer address. Access Couchbase Web Console at <ip>:8091.

kubernetes-pv-couchbase-amazon-elb

Login to the console using Administrator login and password password.

The main page of Couchbase Web Console shows up:

kubernetes-pv-couchbase-amazon-web-console

A default travel-sample bucket is already created by arungupta/couchbase image. This bucket is shown in the Data Buckets tab:

kubernetes-pv-couchbase-amazon-databucket

Click on Create New Data Bucket button to create a new data bucket. Give it a name k8s, take all the defaults, and click on Create button to create the bucket:

kubernetes-pv-couchbase-amazon-k8s-bucket

Created bucket is shown in the Data Buckets tab:

kubernetes-pv-couchbase-amazon-k8s-bucket-created

Check status of the pod:

Delete the pod:

Watch the new pod being created:

Access the Web Console again and see that the bucket does not exist:

kubernetes-pv-couchbase-amazon-k8s-bucket-gone

Let’s clean up the resources created:

Couchbase Pod with Persistent Storage

Now, lets expose a Couchbase pod with persistent storage. As discussed above, lets create a PersistentVolume and claim the volume.

Request storage

Like any other Kubernetes resources, a persistent volume is created by using a resource description file:

The important pieces of information here are:

  • Creating a storage of 5 GB
  • Storage can be mounted by only one node for reading/writing
  • specifies the volume id created earlier

Read more details about definition of this file at kubernetes.io/docs/user-guide/persistent-volumes/.

This file is available at: github.com/arun-gupta/couchbase-kubernetes/blob/master/pv/couchbase-pv.yml.

The volume itself can be created as:

and shows the output:

Use claim

A PersistentVolumeClaim can be created using this resource file:

In our case, both PersistentVolume and PersistentVolumeClaim are 5 GB but they don’t have to be.

Read more details about definition of this file at kubernetes.io/docs/user-guide/persistent-volumes/#persistentvolumeclaims.

This file is at github.com/arun-gupta/couchbase-kubernetes/blob/master/pv/couchbase-pvc.yml.

The claim can be created as:

and shows the output:

Create RC with Persistent Volume Claim

Create a Couchbase Replication Controller using this resource file:

Key parts here are:

  • Resource defines a Replication Controller using arungupta/couchbase Docker image
  • volumeMounts define which volumes are going to be mounted. /opt/couchbase/var is the directory where Couchbase stores all the data.
  • volumes define different volumes that can be used in this RC definition

Create the RC as:

and shows the output:

Check for pod as kubectl.sh get -w po to see:

Expose RC as a service:

Get all the services:

Describe the service as kubectl.sh describe svc couchbase to see:

Wait for ~3 mins for the load balancer to settle. Access the Couchbase Web Console at <ingress-lb>:8091. Once again, only travel-sample bucket exists. This is created by arungupta/couchbase image used in the RC definition.

Show Stateful Containers

Lets create a new bucket. Give it a name kubernetes-pv, take all defaults and click on Create button to create the bucket.

kubernetes-pv-couchbase-amazon-kubernetes-pv-bucket

The bucket now shows up in the console:

kubernetes-pv-couchbase-amazon-kubernetes-pv-bucket-created

Terminate Couchbase pod and see the state getting restored.

Get the pods again:

Delete the pod:

Pod gets recreated:

And now when you access the Couchbase Web Console, the earlier created bucket still exists:

kubernetes-pv-couchbase-amazon-kubernetes-pv-bucket-still-thereThat’s because the data was stored in the backing EBS storage.

Cleanup Kubernetes Cluster

Shutdown Kubernetes cluster:

And detach the volume:

Complete source code for this blog is at: github.com/arun-gupta/couchbase-kubernetes.

Enjoy!

Source: blog.couchbase.com/2016/july/stateful-containers-kubernetes-amazon-ebs

Labels and Constraints with Docker Daemon and Service

Metadata, such as labels, can be attached to Docker daemon. A label is a key/value pair and allows the Docker host to be a target of containers. The semantics of labels is completely defined by the application. A new constraint can be specified during service creation targeting the tasks on a particular host.

Let’s see how we can use labels and constraints in Docker for a real-world application.

Couchbase using Multidimensional Scaling (or MDS) allows to split Index, Data, Query and Full-text search service on multiple nodes. The needs for each service is different. For example, Query is CPU heavy, Index is disk intensive and Data is a mix of memory and fast read/write, such as SSD.

MDS allows the hardware resources to be independently assigned and optimized on a per node basis, as application requirements change.

couchbase-mds

Read more about Multidimensional Scaling.

Let’s see how this can be easily accomplished in a three-node cluster using Docker swarm mode.

 

Start Ubuntu Instances

Start three instances on EC2 of Ubuntu Server 14.04 LTS (HVM) (AMI ID: ami-06116566). Take defaults in all cases except for the security group. Swarm mode requires the following three ports open between hosts:

  • TCP port 2377 for cluster management communications
  • TCP and UDP port 7946 for communication among nodes
  • TCP and UDP port 4789 for overlay network traffic

Make sure to create a new security group with these rules:

ec2-swarmmode-security-group

Wait for a few minutes for the instances to be provisioned.

Set up Docker on Ubuntu

Swarm mode is introduced in Docker 1.12. At the time of this writing, 1.12 RC4 is the latest candidate. Use the following script to install the RC4 release with experimental features:

This script assumes that AWS CLI is already setup and performs the following configuration for all running instances in your configured EC2 account:

  • Get public IP address of each instance
  • For each instance
    • Install latest Docker release with experimental features
    • Adds ubuntu user to the docker group. This allows Docker to be used as a non-root user.
    • Prints the Docker version

This simple script will setup Docker host on all three instances.

Assign Labels to Docker Daemon

Labels can be defined using DOCKER_OPTS. For Ubuntu, this is defined in the /etc/default/docker file.

Distinct labels need to be assigned to each node. For example, use couchbase.mds key and index value.

You also need to restart Docker daemon. Finally, docker info displays system-wide information:

As you can see, labels are visible in this information.

For the second node, assign a different label:

Make sure to use the IP address of the second EC2 instance. The updated information about the Docker daemon in this case will be:

And finally, the last node:

The updated information about the Docker daemon for this host will show:

In our case, a homogenous cluster is created where machines are exactly alike, including their operating system, CPU, disk and memory capacity. In real world, you’ll typically have the same operating system but the instance capacity, such as disk, CPU and memory, would differ based upon what Couchbase services you want to run on them. These labels would make perfect sense in that case but they do show the point here.

Enable Swarm Mode and Create Cluster

Let’s enable Swarm Mode and create a cluster of 1 Manager and 2 Worker nodes. By default, manager are worker nodes as well.

Initialize Swarm on the first node:

This will show the output:

Add other two nodes as worker:

The exact commands, and output, in this case are:

Complete details about the cluster can now be obtained:

And this shows the output:

This shows that we’ve created a 3-node cluster with one manager.

Run Docker Service with Constraints

Now, we are going to run three Couchbase services with different constraints. Each service specifies constraint using --constraint engine.labels.<label> format where <label> matches the labels defined earlier for the nodes.

Each service is given a unique name as it allows to scale them individually. All commands are directed towards the Swarm manager:

The exact commands in our case are:

The list of services can be verified as:

This shows the output as:

And the list of tasks (essentially containers within that service) for each service can then be verified as:

And the output in our case:

This shows the services are nicely distributed across different nodes. Feel free to check out if the task is indeed scheduled on the node with the right label.

All Couchbase instances can be configured in a cluster to provide a complete database solution for your web, mobile and IoT applications.

Want to learn more?

Source: blog.couchbase.com/2016/july/labels-constraints-docker-daemon-service

Couchbase Docker Container on Amazon ECS

This blog will explain how to run a Couchbase Docker container using Amazon EC2 Container Service (Amazon ECS).

Many thanks to @moviolone for helping understand the concepts and getting this setup running.

What is Amazon ECS?

Amazon ECS is a container management service that makes it easy to run, stop, and manage Docker containers on a cluster of Amazon EC2 instances. Amazon ECS integrates well with rest of the AWS infrastructure and eliminates the need to operate your own cluster or configuration management systems.

ec2-container-service

One obvious question to wonder is how is this different from other container orchestration frameworks like Docker Swarm, Kubernetes, or Mesos. The first big difference is that each of these frameworks are open source. Amazon uses a proprietary orchestration framework at this time.

A big advantage of ECS is that just like rest of the AWS infrastructure, this is a managed service. And so you only need to worry about deploying your containers without worrying about the infrastructure.

A better comparison of ECS is with Docker for AWS/Azure (backed by newly introduced Swarm Mode in Docker), Google Container Engine (backed by Kubernetes), DC/OS (backed by Mesos) as they are managed services as well.

An advantage point of ECS is that it seamlessly integrates with AWS infrastructure such as deploying container instances using CloudFormation templates, scaling containers using Autoscaling Group, port mapping using Security Groups, manage incoming container traffic using Elastic Load Balancer, viewing logs using CloudWatch and others.

If you are already bought in the Amazon infrastructure, then ECS sounds like a good fit. Docker for AWS, announced at DockerCon, is also a similar offering in this space.

However, there are a couple of cons that you need to be aware of as well:

  • Portability – Application designed Docker Swarm, Kubernetes and Mesos can run on a variety of platforms, such as Amazon, Azure, GCE, OpenStack, on-prem, VMWare, bare metal data centers, etc. But ECS is tied to Amazon only. Do you consider that as a vendor lock-in?
    Amazon may release their orchestration platform or scheduler as a standalone product, but that’s not very typical.
  • Container format – ECS service is focused on Docker containers only. For all practical purposes, at least today, this may be perfectly fine. I’ve not heard or seen any deployments of Rkt or any other container formats. However, this may change once OCI-compliant runtimes start showing up in the future.

One last thing, before we dig in the concepts and code, there is no additional charge for Amazon EC2 Container Service. You pay for AWS resources (e.g. EC2 instances or EBS volumes) you create to store and run your application.

Amazon ECS Concepts

Here is an overview of the key concepts in ECS:

amazon-ecs-concepts

  • Container Instance: An AMI instance that is primed for running containers. By default, each Amazon instance uses Amazon ECS-Optimized Linux AMI. This is the recommended image to run ECS container service. The key components of this base image are:
    • Amazon Linux AMI
    • Amazon ECS Container Agent – It manages containers lifecycle on behalf of ECS and allows them to connect to the cluster.
    • Docker Engine (as of this writing, this is version 1.11.1)

    Other images like CoreOS, Suse or Ubuntu can be configured to meet Container Instance AMI specification. This can be done because ECS Agent code is available in open source.

  • Task: A task is defined as a JSON file and describes an application that contains one or more container definitions. This usually points to Docker images from a registry, port/volume mapping, etc.
  • Service: ECS maintains the “desired state” of your application. This is achieved by creating a service. A service specifies the number of instances of a task definition that needs to run at a given time. If the task in a service becomes unhealthy or stop running, then the service scheduler will bounce the task. It ensures that the desired and actual state are match. This is what provides resilience in ECS.New tasks within a Service are balanced across Availability Zones in your cluster. Service scheduler figures out which container instances can meet the needs of a service and schedules it on a valid container instance in an optimal Availability Zone (one with the fewest number of tasks running).

Getting Started with Amazon EC2 Container Service

Login to your AWS EC2 console and click on the EC2 Container Service:

aws-ec2-container-1

Click on the Get started button to define your application.

Create ECS Task

In ECS, Docker workloads are defined as tasks. A task can contain multiple containers. All containers for a task are co-located on the same machine.

Enter the values as shown:

aws-ec2-container-2

Few items specified in this step:

  • Task definition is description of an application that contains one or more container definitions.
  • Container name is the name that will be given to the container started as part of this task.
  • Image allows to specify one or more images that need to be started as containers as part of this application. The image specified here uses couchbase:latest as the base image and uses Couchbase REST API to configure the server. Dockerfile for this image provide more details about how this image is prepared.
  • Maximum memory is the memory that needs to be allocated for the container (equivalent to -m Docker CLI switch). Couchbase needs 1GB for running in dev and so that is specified here.
  • And finally the port mappings (-p on Docker CLI). Port 8091 is needed for Couchbase administration.

More details about these is available in Task Definition Parameters.

Create ECS Service

 

Click on Next step to configure a service.

aws-ec2-container-3

Give a service name. The desired state can be specified here. For now, we’ll keep it simple and launch a single node Couchbase container. And since the desired state is run a single container, no ELB is required.

More details about these is available in Service Definition Parameters.

Create ECS Cluster

Tasks run on a container instance, and these instances need to register in a cluster. This allows us to scale the cluster up/down later to accommodate for running more containers.

Click on Next step to configure the cluster.

aws-ec2-container-4

In this image:

  • Take the default cluster name
  • A homogenous cluster of container instances is created. m3.medium is a good size to run Couchbase node
  • Choose a previously created security key. This will allow to open a ssh connection to the container instance
  • A new IAM role will be created to allow ECS agent to communicate with ECS service

Container instances in a cluster can span multiple availability zones and be balanced with ELB.

Review all the specified options:

aws-ec2-container-5

Click on Launch instance & run service button to start the service.

The following status is shown after the service is created:

aws-ec2-container-6

The output shows that the cluster, service and task definitions are created. It takes a few minutes for the instances to be provisioned and initializedand tasks to run on them.

View ECS Service and Task

Click on View Service button to see the newly created service.

aws-ec2-container-7

Few things in this image:

  • The service shows the task definition couchbase:6. Each service is assigned a task definition and multiple versions are indicated by the trailing number at the end. In this case, a few versions were created earlier but otherwise the version number starts from 1.
  • Desired and Running count is shown as 1.
  • Minimum healthy percent and Maximum percent are used if a new version of task definition needs to be deployed. With 100% and 200% corresponding values, a new version of the task will be deployed first and then the older versions will be terminated. We’ll play with these numbers in a subsequent blog.
  • Running task is shown towards bottom of the screen. Click on the UUID to learn more about the running task.

aws-ec2-container-8

Task definition shows EC2 instance where it is running, current status, port mapping and several other useful information. The critical piece that we need to look at is the External Link. This URL is where our Couchbase Web Console will be accessible.

Couchbase Web Console

Clicking on this link will open a new tab with Couchbase Web Console:

aws-ec2-container-10

 

Enter the login as Administrator and password as password. These are configured in arungupta/couchbase image.

And here you see Couchbase Web Console in full glory!

aws-ec2-container-11

This blog explained how to run a Couchbase Docker container using Amazon ECS.

Future blogs will show …

  • Setup a Couchbase cluster using ECS
  • Deploy a multi-container application using Docker Compose (v2 is now supported)
  • Setup ECS cluster using CLI

Amazon ECS and Couchbase References

Source: blog.couchbase.com/2016/july/couchbase-docker-container-amazon-ecs

Docker Daemon Log with Docker for Mac

Did you know that Docker for Mac is now in general beta?

docker-for-mac

What is Docker for Mac?

Docker for Mac is a native Mac application architected from scratch, with a native user interface and auto-update capability, deeply integrated with OS X native virtualization

If you are using Docker Machine, then you can ssh to the machine using docker-machine ssh <machine-name> command and find the logs at /var/log/docker.

As Docker for Mac provide a native integration with Mac, the logs also can be found using the natural tools.

Mac Console for Docker Daemon Logs

Console is a utility available in Applications -> Utilities. log viewer included with macOS. It allows users to search through all of the system’s logged messages, and can alert the user when certain types of messages are logged. The console allows you to read the system logs, help find certain ones, monitor them, and filter their contents.

File -> New System Log Query…

docker-logs-console1

Give the query a name and set Sender to docker. Click on OK to save the query:

docker-logs-console2

Now the daemon logs can be easily seen here.

Now Console Log Query can be used to search logs, filter the results in various ways, and create reports.

Docker Daemon Log using CLI

You are not a GUI types person, and prefer a CLI approach. Then use syslog CLI. The command to see Docker daemon log is:

syslog -k Sender Docker

And it shows the output as:

Use syslog -help to find all the options for this CLI.

Docker Daemon Log File

If you really want the hard core way, then the log files are available at:

~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/log.

Check complete details at: docs.docker.com/docker-for-mac/troubleshoot/#/checking-the-logs.

What is holding you back from using Docker for Mac?

Enjoy!

Source: blog.couchbase.com/2016/july/docker-daemon-log-mac