Getting Started with ELK Stack on WildFly

Your typical business application would consist of a variety of servers such as WildFly, MySQL, Apache, ActiveMQ, and others. They each have a log format, with minimal to no consistency across them. The log statement typically consist of some sort of timestamp (could be widely varied) and some text information. Logs could be multi-line. If you are running a cluster of servers then these logs are decentralized, in different directories.

How do you aggregate these logs? Provide a consistent visualization over them? Make this data available to business users?

This blog will:

  • Introduce ELK stack
  • Explain how to start it
  • Start a WildFly instance to send log messages to the ELK stack (Logstash)
  • View the messages using ELK stack (Kibana)

What is ELK Stack?

ELK stack provides a powerful platform to index, search and analyze your data. It uses  Logstash for log aggregation, Elasticsearch for searching, and Kibana for visualizing and analyzing data. In short, ELK stack:

  • Collect logs and events data (Logstash)
  • Make it searchable in fast and meaningful ways (Elasticsearch)
  • Use powerful analytics to summarize data across many dimensions (Kibana)

logstash-logo

Logstash is a flexible, open source data collection, enrichment, and transportation pipeline.

elasticsearch-logo

Elasticsearch is a distributed, open source search and analytics engine, designed for horizontal scalability, reliability, and easy management.

kibana-logo

Kibana is an open source data visualization platform that allows you to interact with your data through stunning, powerful graphics.

How does ELK Stack work?

Logstash can collect logs from a variety of sources (using input plugins), process the data into a common format using filters, and stream data to a variety of sources (using output plugins). Multiple filters can be chained to parse the data into a common format. Together, they build a Logstash Processing Pipeline.

Logstash Processing Pipeline

Inputs and outputs support codecs that enable you to encode or decode the data as it enters or exits the pipeline without having to use a separate filter.

Logstash can then store the data in Elasticsearch and Kibana provides a visualization of that data. Here is a sample pipeline that can collect logs from different servers and run it through the ELK stack.

ELK Stack

Start ELK Stack

You can download individual components of ELK stack and start that way. There is plenty of advise on how to configure these components. But I like to start with a KISS, and Docker makes it easy to KISS!

All the source code on this blog is at github.com/arun-gupta/elk.

  1. Clone the repo:
  2. Run the ELK stack:

    This will use the pre-built Elasticsearch, Logstack, and Kibana images. It is built upon the work done in github.com/nathanleclaire/elk.

    docker ps will show the output as:

    It shows all the containers running.

WildFly and ELK

James (@the_jamezp) blogged about Centralized Logging for WildFly with ELK Stack. The blog explains how to configure WildFly to send log messages to Logstash. It uses the highly modular nature of WildFly to install jboss-logmanager-ext library and install it as a module. The configured logmanager includes @timestamp field to the log messages sent to logstash. These log messages are then sent to Elasticsearch.

Instead of following the steps, lets Docker KISS and use a pre-configured image to get you started.

Start the image as:

Make sure to substitute <DOCKER_HOST_IP> with the IP address of the host where your Docker host is running. This can be easily found using docker-machine ip <MACHINE_NAME>.

View Logs using ELK Stack

Kibana runs on an embedded nginx and is configured to run on port 80 in docker-compose.yml. Lets view the logs using that.

  1. Access http://<DOCKER_HOST_IP> in your machine and it should show the default page as:ELK Stack WildFly PatternThe @timestamp field was created by logmanager configured in WildFly.
  2. Click on Create to create an index pattern and select Discover tab to view the logs as:ELK Stack WildFly Output

Try connecting other sources and enjoy the power of distributed consolidated by ELK!

Some more references …

Distributed logging and visualization is a critical component in a microservices world where multiple services would come and go at a given time. A future blog will show how to use ELK stack with a microservices architecture based application.

Enjoy!

Kubernetes Design Patterns

14,000 commits and 400 contributors (including one tiny commit from me!) is what build Kubernetes 1.0. It is now available!

This blog discusses some of the Kubernetes design patterns. All source code for the design patterns discussed below are available at kubernetes-java-sample.

Key Concepts of Kubernetes

At a very high level, there are three key concepts:

  • Pods are the smallest deployable units that can be created, scheduled, and managed. Its a logical collection of containers that belong to an application.
  • Master is the central control point that provides a unified view of the cluster. There is a single master node that control multiple minions.
  • Node is a worker node that run tasks as delegated by the master. Minions can run one or more pods. It provides an application-specific “virtual host” in a containerized environment.

Kubernetes Key Concepts

 

Some other concepts to be aware of:

  • Replication Controller is a resource at Master that ensures that requested number of pods are running on nodes at all times.
  • Service is an object on master that provides load balancing across a replicated group of pods.
  • Label is an arbitrary key/value pair in a distributed watchable storage that the Replication Controller uses for service discovery.

Start Kubernetes Cluster

  1. Easiest way to start a Kubernetes cluster on a Mac OS is using Vagrant:
  2. Alternatively, Kubernetes can be downloaded from github.com/GoogleCloudPlatform/kubernetes/releases/download/v1.0.0/kubernetes.tar.gz, and cluster can be started as:

Kubernetes Cluster Vagrant

A Pod with One Container

This section will explain how to start a Pod with one Container. WildFly base Docker image will be used as the Container.

Kubernetes One Pod

Pod, Replication Controller, Service, etc are all resources in Kubernetes. They can be created using the kubectl by using a configuration file.

The configuration file in this case:

Complete details on how to create a Pod are explained at github.com/arun-gupta/kubernetes-java-sample#a-pod-with-one-container

Java EE Application Deployed in a Pod with One Container

This section will show how to deploy a Java EE application in a Pod with one Container. WildFly, with an in-memory H2 database, will be used as the container.

Kubernetes Java EE 7 Application

Configuration file is:

Complete details at github.com/arun-gupta/kubernetes-java-sample#java-ee-application-deployed-in-a-pod-with-one-container-wildfly–h2-in-memory-database.

A Replication Controller with Two Replicas of a Pod

This section will explain how to start a Replication Controller with two replicas of a Pod. Each Pod will have one WildFly container.

Kubernetes Replication Controller

Configuration file is:

Complete details at github.com/arun-gupta/kubernetes-java-sample#a-replication-controller-with-two-replicas-of-a-pod-wildfly

Rescheduling Pods

Replication Controller ensures that specified number of pod “replicas” are running at any one time. If there are too many, the replication controller kills some pods. If there are too few, it starts more.

Kubernetes Pod Rescheduling

Complete details at github.com/arun-gupta/kubernetes-java-sample#rescheduling-pods.

Scaling Pods

Replication Controller allows dynamic scaling up and down of Pods.

Kubernetes Scaling Pods

Complete details at github.com/arun-gupta/kubernetes-java-sample#scaling-pods.

Kubernetes Service

Pods are ephemeral. IP address assigned to a Pod cannot be relied upon. Kubernetes, Replication Controller in particular, create and destroy Pods dynamically. A consumer Pod cannot rely upon the IP address of a producer Pod.

Kubernetes Service is an abstraction which defines a set of logical Pods. The set of Pods targeted by a Service are determined by labels associated with the Pods.

This section will show how to run a WildFly and MySQL containers in separate Pods. WildFly Pod will talk to the MySQL Pod using a Service.

Kubernetes Service

Complete details at github.com/arun-gupta/kubernetes-java-sample#kubernetes-service.

Here are couple of blogs that will help you get started:

The complete set of Kubernetes blog entries provide more details.

Enjoy!

Scaling Kubernetes Cluster

kubernetes-logo

Automatic Restarting of Pods inside Replication Controller of Kubernetes Cluster shows how Kubernetes reschedule pods in the cluster if one or more of existing Pods disappear for some reason. This is a common usage pattern and one of the key features of Kubernetes.

Another common usage pattern of Replication Controller is scaling:

The replication controller makes it easy to scale the number of replicas up or down, either manually or by an auto-scaling control agent, by simply updating the replicas field.

This blog will show how a Kubernetes cluster can be easily scaled up and down.

All the code used in this blog is available at kubernetes-java-sample.

Start Replication Controller and Verify

  1. Start a Replication Controller as:
  2. Get status of the Pods:

    Make sure to wait for the status to change to Running.

    Note down name of the Pods as wildfly-rc-bgtkg” and wildfly-rc-bgtkg”.

  3. Get status of the Replication Controller:

    If multiple Replication Controllers are running then you can query for this specific one using the label:

Scaling Kubernetes Cluster Up

Replication Controller allows dynamic scaling up and down of Pods.

  1. Scale up the number of Pods:
  2. Status of the Pods can be seen in another shell:

    Notice a new Pod with the name wildfly-rc-aqaqn is created.

Scale Kubernetes Cluster Down

  1. Scale down the number of Pods:
  2. Status of the Pods using -w is not correctly updated (#11338). But status of the Pods can be seen correctly as:

    Notice only one Pod is now running.

Kubernetes dynamically scales the Pods up and down using the scale --replicas command.

All code used in this blog is available at kubernetes-java-sample.

Enjoy!

Automatic Restarting of Pods inside Replication Controller of Kubernetes Cluster

kubernetes-logo

A key feature of Kubernetes is its ability to maintain the “desired state” using declared primitives. Replication Controllers is a key concept that helps achieve this state.

A replication controller ensures that a specified number of pod “replicas” are running at any one time. If there are too many, it will kill some. If there are too few, it will start more.

Lets take a look on how to spin up a Replication Controller with two replicas of a Pod. Then we’ll kill one pod and see how Kubernetes will start another Pod automatically.

Start Kubernetes Cluster

  1. Easiest way to start a Kubernetes cluster on a Mac OS is using Vagrant:
  2. Alternatively, Kubernetes can be downloaded from github.com/GoogleCloudPlatform/kubernetes/releases/download/v1.0.0/kubernetes.tar.gz, and cluster can be started as:

Start and Verify Replication Controller and Pods

  1. All configuration files required by Kubernetes to start Replication Controller are in kubernetes-java-sample project.  Clone the workspace:
  2. Start a Replication Controller that has two replicas of a pod, each with a WildFly container:

    The configuration file used is shown:

    Default WildFly Docker image is used here.
  3. Get status of the Pods:

    Notice -w refreshes the status whenever there is a change. The status changes from Pending to Running and then Ready to receive requests.
  4. Get status of the Replication Controller:

    If multiple Replication Controllers are running then you can query for this specific one using the label:
  5. Get name of the running Pods:
  6. Find IP address of each Pod (using the name):

    And of the other Pod as well:
  7. Pod’s IP address is accessible only inside the cluster. Login to the minion to access WildFly’s main page hosted by the containers:

Automatic Restart of Pods

Lets delete a Pod and see how a new Pod is automatically created.

Notice how the Pod with name wildfly-rc-15xg5 was deleted and a new Pod with the name wildfly-rc-0xoms was created.

Finally, delete the Replication Controller:

The latest configuration files and detailed instructions are at kubernetes-java-sample.

In real world, you’ll typically wrap this Replication Controller in a Service and front-end with a Load Balancer. But that’s a topic for another blog!

Enjoy!

Minecon 2015 Wrapup

minecon2015-logo

From a gathering of ~50 people in 2010, Minecon 2015 with 10,000 attendees created a new world record for the biggest convention for a single game.

Minecon 2015 Experience

Do you want to know what what it feels like to be at Minecon?

Minecraft Modding Workshop

Devoxx4Kids was fortunate to give Minecraft Modding workshops to ~200 kids at Minecon 2015. Feedback from all the parents and kids was quite outstanding. Glad we were able to ignite spark in some kids and get them excited in programming, and open source tools like Java, Eclipse, and Minecraft Forge.

Here are a couple of tweets:

All the instructions for minecraft modding are at minecraftmodding.org.

Many thanks to Mark Little and his son Adam, and my son for helping with a successful workshop. Its very important that kids feel comfortable to play with open source tools, and be willing to hack!

Using Mods for Teaching Panel

I also got the opportunity to lead a panel on Using Mods for Teaching with @DorineFlies, @YouthDigital, and @_moonlapse.

Here are some of the questions we addressed:

  1. How are you involved with modding?
  2. How many students/kids have you reached out so far?
  3. What languages/platforms do you use?
  4. Can modding be the right medium for first introduction to programming?
  5. What is an appropriate age to start modding?
  6. What can be done to fundamentally change STEM education in schools?
  7. What would you like from Mojang to improve the modding experience?

The panel was recorded and should be made available at youtube.com/user/TeamMojang/videos. I’ll update this blog when the exact link is available.

Minecraft Youtubers

One of the big craze, and genuine one, in the Minecraft world is about youtubers who produce video of game plays and most of them have 1m+ subscribers. Several of them were attending Minecon and we were fortunate to meet a few of them:

 

As Lydia walked around the main hall, most of the kids were super excited to meet their favorite youtubers!

Minecon 2015 Cape

Every Minecon attendee also get a cape that their in-game character can wear it and show-off the fact you attended a big celebration! Theme for this year was Iron Golem and it looks like as shown:

Minecon 2015 Cape

Minecraft Characters with Snaak

We also met the team behind Snaak and played around with creating some of the Minecraft characters using it.

 

Minecraft and HoloLens

A re-run of HoloLens and Minecraft video was also shown during one of the keynotes, and a preview is available here:

Here is the complete opening ceremony animation:

Minecon 2015 Photo Album

Check out some pictures from our trip:

 
   
 
 

And the complete photo album:

To me the highlight of the conference was meeting @SeargeDP. If there is one name that is responsible for starting modding in Minecraft, that would be him! Many thanks to him for giving us a chance to deliver minecraft modding workshops at Minecon.

And then, of course, meeting @lexmanos who is the lead developer for Minecraft Forge. We’ve authored an O’Reilly book (targeted at 8+ years old kid) and video on this topic. Several Devoxx4Kids chapters around the world have delivered workshops using the instructions based on Minecraft Forge and explained at minecraftmodding.org.

Check out a nice credential about book from one of the parents we met:

And last, but not the least, many thanks to the Mojang team for keeping the release cadence and supporting different modding communities.

Minecraft is truly a revolutionary game and allows to introduce Java programming to kids at a very early age!

Hopefully we get invited to Minecon 2016 again :)

 

Multi-container Applications using Docker Compose and Swarm

Docker Compose to Orchestrate Containers shows how to run two linked Docker containers using Docker Compose. Clustering Using Docker Swarm shows how to configure a Docker Swarm cluster.

This blog will show how to run a multi-container application created using Docker Compose in a Docker Swarm cluster.

Updated version of Docker Compose and Docker Swarm are released with Docker 1.7.0.

Docker 1.7.0 CLI

Get the latest Docker CLI:

and check the version as:

Docker Machine 0.3.0

Get the latest Docker Machine as:

and check the version as:

Docker Compose 1.3.0

Get the latest Docker Compose as:

and verify the version as:

Docker Swarm 0.3.0

Swarm is run as a Docker container and can be downloaded as:

You can learn about Docker Swarm at docs.docker.com/swarm or Clustering using Docker Swarm.

Create Docker Swarm Cluster

The key components of Docker Swarm are shown below:

and explained in Clustering Using Docker Swarm.

  1. The easiest way of getting started with Swarm is by using the official Docker image:

    This command returns a discovery token, referred as <TOKEN> in this document, and is the unique cluster id. It will be used when creating master and nodes later. This cluster id is returned by the hosted discovery service on Docker Hub.

    It shows the output as:

    The last line is the <TOKEN>.

    Make sure to note this cluster id now as there is no means to list it later. This should be fixed with#661.

  2. Swarm is fully integrated with Docker Machine, and so is the easiest way to get started. Let’s create a Swarm Master next:

    Replace <TOKEN> with the cluster id obtained in the previous step.

    --swarm configures the machine with Swarm, --swarm-master configures the created machine to be Swarm master. Swarm master creation talks to the hosted service on Docker Hub and informs that a master is created in the cluster.

  3. Connect to this newly created master and find some more information about it:

    This will show the output as:

  4. Create a Swarm node

    Replace <TOKEN> with the cluster id obtained in an earlier step.

    Node creation talks to the hosted service at Docker Hub and joins the previously created cluster. This is specified by --swarm-discovery token://... and specifying the cluster id obtained earlier.

  5. To make it a real cluster, let’s create a second node:

    Replace <TOKEN> with the cluster id obtained in the previous step.

  6. List all the nodes created so far:

    This shows the output similar to the one below:

    The machines that are part of the cluster have the cluster’s name in the SWARM column, blank otherwise. For example, “lab” and “summit2015” are standalone machines where as all other machines are part of the “swarm-master” cluster. The Swarm master is also identified by (master) in the SWARM column.

  7. Connect to the Swarm cluster and find some information about it:

    This shows the output as:

    There are 3 nodes – one Swarm master and 2 Swarm nodes. There is a total of 4 containers running in this cluster – one Swarm agent on master and each node, and there is an additional swarm-agent-master running on the master.

  8. List nodes in the cluster with the following command:

    This shows the output as:

Deploy Java EE Application to Docker Swarm Cluster using Docker Compose

Docker Compose to Orchestrate Containers explains how multi container applications can be easily started using Docker Compose.

  1. Use the docker-compose.yml file explained in that blog to start the containers as:

    The docker-compose.yml file looks like:
  2. Check the containers running in the cluster as:

    to see the output as:
  3. “swarm-node-02” is running three containers and so lets look at the list of containers running there:

    and see the list of running containers as:
  4. Application can then be accessed again using:

    and shows the output as:

Latest instructions for this setup are always available at: github.com/javaee-samples/docker-java/blob/master/chapters/docker-swarm.adoc.

Enjoy!

Microservices and DevOps Journey at Wix

Wix.com started their journey on DevOps and Microservices about two years ago and recently switched from a monolithic application to a microservices-based application. Yes, it took them full two years to complete the transition from monolith to microservices!

I got connected with Aviran Mordo (@aviranm), head of backend engineering at Wix on twitter.

They migrated to microservices because the “system could not scale” and the requirements for functional components were varied. The journey took their WAR-based deployment on Tomcat to fat JAR with embedded Jetty. On a side note, take a look at WildFly Swarm if you are interested in a similar approach for Java EE applications.

Video Interview

I discussed some points with him about this journey and you can watch the same too.

In this discussion, you’ll learn:

  • Why Continuous Delivery and DevOps are important requirements for microservices?
  • How they migrated from a big monolith to smaller monoliths and then a full-blown microservices architecture
  • How database referential integrity constraints were moved from database to application?
  • “micro” in microservices refers to the area of responsibility, nothing to do with LOC
  • Neither REST nor messaging was used for communication between different services. Which protocol was used? JSON-RPC
  • How do services register and discover each other? Is that required during early phases?
  • Why YAGNI and KISS are important?
  • Chef for configuration management and how to make it accessible for massive deployments
  • TeamCity for CI
  • Is 100% automation a requirement? Can 100% automation be achieved? Learn about Petri, Wix’s open source framework for A/B testing
  • Relevance of hybrid cloud (Google, Amazon, Private data center) and redundancy
  • Hardest part of migrating from monolith to microservice
  • How much code was repurposed during refactoring?
  • Where was the most effort spent during the two years of migration?
  • Distributed transactions
  • What was the biggest challenge in DevOps journey? Look out for a nice story towards the end that could be motivating for your team as well 😉

Additional Material

Watch the slides from DevoxxUK:

You can also learn more about their architecture in Scaling Wix to 60m Users.

Enjoy!

DevNation and Red Hat Summit 2015 Wrapup

RedHat Summit Logo DevNation Logo

Red Hat Summit and DevNation is a wrap!

It took two full night sleep and a long afternoon nap to fully recover from the excitement, stimulation, and exhaustion that sets in after meeting awesome developers, customers, partners, colleagues, and geeks from around the world. The fact that I gave four talks, one hands-on lab, participated in two panels, ran Devoxx4Kids event, talked to a lots of analysts, book signing, breakfasts/lunches/dinners/receptions, ran every morning by Charles river – all within 6 days added to the exhaustion as well 😉

In the end, it was very rewarding and inspiring to see the work others are doing!

Complete set of slides are available at redhat.com/summit/2015/presentations. Here are links to the slides from my sessions:

Watch the middleware keynote by Craig Muzilla:

Watch Burr Sutter geek show starting at ~19:00.

Learn/understand more about our middleware offerings using Accelerate, Integrate, and Automate.

Some pictures from the event …

 
 
   
   
 

Complete album …

Here are some other photo albums:

Enjoy!

 

Minecon 2015 – Minecraft Modding Workshop and Education Panel

Devoxx4Kidsminecon 2015
Devoxx4Kids Minecraft Modding workshop has been used all around the world by kids as their first introduction to Java programming, even programming in several cases. This workshop has been delivered at our meetups, corporate events such as OSCON, Silicon Valley Code Camp, JavaOne and Red Hat Summit, schools, libraries, workplaces, homes, and many other venues. And now its going to the most coveted and sought after place for all minecrafters around the world – Minecon 2015!

Many thanks to Microsoft (now the parent company of Mojang) for extending the invitation!

Minecon is the biggest gathering of all minecrafters on this planet. About 10,000 attendees are expected to gather on Jul 4th and 5th in London this year. This will be our first time there. We are super excited and can’t wait to experience the phenomenon.

In addition, Microsoft has also asked us to lead a panel on “Using Mods for Teaching”. This panel will discuss the relevance of modding in education. We’ll try to answer some of the questions as:

  • Can modding be the right medium for first introduction to programming?
  • What is an appropriate age to start modding?
  • What would you like from Mojang to improve the modding experience?
  • What can be done to fundamentally change STEM education in schools?

Participate in Minecon 2015 Virtually

You can participate in Minecon virtually as well!

Are there any questions that you’d like to ask the panelists? Leave a comment in the blog and I’ll try to accommodate as many of them as possible.

As always, follow me @arungupta to watch out for comments/pictures from the event!

Devoxx4Kids Ancillary Event at Corporate

Running a Devoxx4Kids event as an ancillary to the main event is a great way to engage local community. Such an event can consist of not just minecraft modding, but several other topics such as Scratch, Greenfoot, and HTML5/CSS. Let us know if your corporate event is interested in running a Devoxx4Kids workshop the weekend before/after the main event!

Devoxx4Kids at Red Hat Summit 2015

RedHat Summit LogoDevNation Logo Devoxx4Kids

~70 kids were sprawling in a part of the Hynes Convention Center in Boston last Sunday. They were invited by Red Hat to participate in a variety of workshops and get them introduced to technology in a fun way. The workshops were facilitated by Devoxx4Kids instructors and Red Hatters. This is right before the start of Red Hat Summit – a premier open source technology conference for the “big kids”.

The kids had an opportunity to learn the following technologies:

  • MIT App Inventor
  • Greenfoot
  • Minecraft Modding
  • Dr Racket
  • Scratch
  • Flappy Birds

The feedback from all the kids and parents have been very encouraging, and everybody seem to have a great time. Here are some pictures from the event:

 
   
 
 
 
 

Complete set of pictures are:

Number of emails exchanged with the team: Hundreds
Support from the team/instructors/volunteers to run the event: Invaluable
Flight got delayed, stayed up until 3:30 ensuring machines work: Tiring
Kids enjoyed learning how to build the games: Priceless

Overall, it has been quite a joyful and rewarding experience to promote STEM to elementary and middle school kids!

Running a Devoxx4Kids event as an ancillary to the main event is a great way to engage local community. Red Hat Summit is the fourth corporate event in the US (after OSCON 2nd year, JavaOne 2nd year, and Silicon Valley Code Camp) to partner with Devoxx4Kids and give back to the community. Let us know if your corporate event is interested in running a Devoxx4Kids workshop the weekend before/after the main event!

 

Docker 1.7.0, Docker Machine 0.3.0, Docker Compose 1.3.0, Docker Swarm 0.3.0

Docker 1.7.0 is released (change log) and so time to update Docker Hosts, CLI, and other tools.

Docker 1.7.0

Docker Host is running inside a Docker Machine and so the machine needs to be upgraded. The machine must be stopped otherwise you get an error as:

So start the machine as:

And then upgrade the machine as:

The machine is anyway stopped to perform an upgrade, and so the need to start the machine seems superfluous (#1399).

Upgrading the host updates .docker/machine/cache/boot2docker.iso. Any previously created machines cache the boot2docker.iso in .docker/machine/machines/<MACHINE-NAME> and so they’ll continue to boot using the same version.

Docker CLI

Update the Docker CLI as:

Now docker version shows the following output:

Note, the client API version (1.7.0) and the server API version (1.7.0) are both shown here.

If you update only the CLI and not the Docker Host, then the following error message is shown:

This error messages shows a version mismatch between the client CLI and the Docker Host running in the machine. The will typically happen if the active machine was created a few days ago using an older boot2docker.iso. There seems to be no way straight forward way to find out the exact version currently being used (#1398).

There seems to be no way for a new client to talk to the old server (#14077), and thus the host needs to be upgraded. There is a proposal to override the API version of client (#11486), but at this time there is no ETA for the fix. So the only option is to upgrade the docker machine, which will then then upgrade to the latest version of Docker.

So upgrading the CLI requires to upgrade the machine as well.

Here are the options supported by Docker CLI :

Docker Machine 0.3.0

This was rather straight forward:

There are a lots of new features, including an experimental provisioner for Red Hat Enterprise Linux 7.0.

The version is shown as:

Complete list of commands are:

Docker Compose 1.3.0

Docker Compose can be updated to 1.3.0 as:

The version is shown as:

Two important points to note:

  • At least Docker 1.6.0 is required
  • There are breaking changes from Compose 1.2 and so you either need to remove and recreate your containers, or migrate them.Fortunately docker-compose migrate-to-labels can be used to migrate pre-Compose 1.3.0 containers to the latest format. This will recreate the containers with labels added.

Learn more in Docker Compose to Orchestrate Containers.

Docker Swarm 0.3.0

As of this blog, Docker Swarm 0.3.0 RC3 is available. Clustering Using Docker Swarm provide a good introduction to Docker Swarm and can be used to get started with the latest Docker Swarm release.

34 issues have been fixed since 0.2.0 but the commit notifications since 0.2.0 for Release Candidates seem to show no significant changes.

More detailed blogs on each Docker component will be shared in subsequent blogs.

Enjoy!

ZooKeeper for Microservice Registration and Discovery

In a microservice world, multiple services are typically distributed in a PaaS environment. Immutable infrastructure is provided by containers or immutable VM images. Services may scale up and down based upon certain pre-defined metrics. Exact address of the service may not be known until the service is deployed and ready to be used.

This dynamic nature of service endpoint address is handled by service registration and discovery. In this, each service registers with a broker and provide more details about itself, such as the endpoint address. Other consumer services then queries the broker to find out the location of a service and invoke it. There are several ways to register and query services such as ZooKeeper, etcd, consul, Kubernetes, Netflix Eureka and others.

Monolithic to Microservice Refactoring showed how to refactor an existing monolith to a microservice-based application. User, Catalog, and Order service URIs were defined statically. This blog will show how to register and discover microservices using ZooKeeper.

Many thanks to Ioannis Canellos (@iocanel) for all the ZooKeeper hacking!

What is ZooKeeper?

Apache ZooKeeperZooKeeper is an Apache project and provides a distributed, eventually consistent hierarchical configuration store.

 

ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. All of these kinds of services are used in some form or another by distributed applications.

So a service can register with ZooKeeper using a logical name, and the configuration information can contain the URI endpoint. It can consists of other details as well, such as QoS.

Apache Curator ZooKeeper has a steep learning curve as explained in Apache ZooKeeper Made Simpler with Curator. So, instead of using ZooKeeper directly, this blog will use Apache Curator.

Curator n ˈkyoor͝ˌātər: a keeper or custodian of a museum or other collection – A ZooKeeper Keeper.

Apache Curator has several components, and this blog will use the Framework:

The Curator Framework is a high-level API that greatly simplifies using ZooKeeper. It adds many features that build on ZooKeeper and handles the complexity of managing connections to the ZooKeeper cluster and retrying operations.

ZooKeeper Concepts

ZooKeeper Overview provides a great overview of the main concepts. Here are some of the relevant ones:

  • Znodes: ZooKeeper stores data in a shared hierarchical namespace that is organized like a standard filesystem. The name space consists of data registers – called znodes, in ZooKeeper parlance – and these are similar to files and directories.
  • Node name: Every node in ZooKeeper’s name space is identified by a path. Exact name of a node is a sequence of path elements separated by a slash (/).
  • Client/Server: Clients connect to a single ZooKeeper server. The client maintains a TCP connection through which it sends requests, gets responses, gets watch events, and sends heart beats. If the TCP connection to the server breaks, the client will connect to a different server.
  • Configuration data: Each node in a ZooKeeper namespace can have data associated with it as well as children. ZooKeeper was originally designed to store coordination data, so the data stored at each node is usually small, in less than KB range).
  • Ensemble: ZooKeeper itself is intended to be replicated over a sets of hosts called an ensemble. The servers that make up the ZooKeeper service must all know about each other.
  • Watches: ZooKeeper supports the concept of watches. Clients can set a watch on a znode. A watch will be triggered and removed when the znode changes.

ZooKeeper is a CP system with regards to CAP theorem. This means if there is a partition failure, it will be consistent but not available. This can lead to problems that are explained in Eureka! Why You Shouldn’t Use ZooKeeper for Service Discovery.

Nevertheless, ZooKeeper is one of the most popular service discovery mechanisms used in microservices world.

Lets get started!

Start ZooKeeper

  1. Start a ZooKeeper instance in a Docker container:
  2. Verify ZooKeeper instance by using telnet as:

    Type the command “ruok” to verify that the server is running in a non-error state.The server will respond with “imok” if it is running:

    Otherwise it will not respond at all. ZooKeeper has other similar four-letter commands.

Service Registration and Discovery

Each service, User, Catalog, and Order in our case, has an eagerly initialized bean that registers and unregisters the service as part of lifecycle initialization methods as. Here is the code from CatalogService:

The code is pretty simple, it injects ServiceRegistry class, with @ZooKeeperRegistry qualifier. This is then used to register and unregister the service. Multiple URIs, one each for a stateless service, can be registered under the same logical name.

At this time, the qualifier comes from another maven module. A cleaner Java EE way would be to move the @ZooKeeperRegistry qualifier to a CDI extension (#20). And when this qualifier when specified on any REST endpoint will register the service with ZooKeeper (#22). For now, service endpoint URI is hardcoded as well (#24).

What does ZooKeeper class look like?

  1. ZooKeeper class uses constructor injection and hardcoding IP address and port (#23):

    It does the following tasks:

    1. Loads ZooKeeper’s host/port from a properties file
    2. Initializes Curator framework and starts it
    3. Initializes a hashmap to store the URI name to zNode mapping. This node is deleted later to unregister the service.
  2. Service registration is done using registerService method as:

    Code is pretty straight forward:

    1. Create a parent zNode, if needed
    2. Create an ephemeral and sequential node
    3. Add metadata, including URI, to this node
  3. Service discovery is done using discover method as:

    Again, simple code:

    1. Find all children for the path registered for the service
    2. Get metadata associated with this node, URI in our case, and return.The first such node is returned in this case. Different QoS parameters can be attached to the configuration data. This will allow to return the appropriate service endpoint.

Read ZooKeeper Javadocs for API.

ZooKeeper watches can be setup to inform the client about the lifecycle of the service (#27). ZooKeeper path caches can provide an optimized implementation of the children nodes (#28).

Multiple Service Discovery Implementations

Our shopping cart application has two two service discovery implementationsServiceDisccoveryStatic and ServiceDiscoveryZooKeeper. The first one has all the service URIs defined statically, and the other one retrieves them from ZooKeeper.

Other means to register and discover can be easily added by creating a new package in services module and implementing ServiceRegistry interface. For example, Snoop, etcd, Consul, and Kubernetes. Feel free to send a PR for any of those.

Run Application

  1. Make sure the ZooKeeper image is running as explained earlier.
  2. Download and run WildFly:
  3. Deploy the application:
  4. Access the application at localhost:8080/everest-web/. Learn more about the application and different components in Monolithic to Microservices Refactoring for Java EE Applications blog.

Enjoy!

Monolithic to Microservices Refactoring for Java EE Applications

Have you ever wondered what does it take to refactor an existing Java EE monolithic application to a microservices-based one?

This blog explains how a trivial shopping cart example was converted to microservices-based application, and what are some of the concerns around it. The complete code base for monolithic and microservices-based application is at: github.com/arun-gupta/microservices.

Read on for full glory!

Java EE Monolith

A Java EE monolithic application is typically defined as a WAR or an EAR archive. The entire functionality for the application is packaged in a single unit. For example, an online shopping cart may consist of User, Catalog, and Order functionalities. All web pages are in root of the application, all corresponding Java classes are in the WEB-INF/classes directory, resources in WEB-INF/classes/META-INF directory.

Lets assume that your monolith is not designed as a distributed big ball of mud and the application is built following good software architecture. Some of the common rules are:

  • Separation of concerns, possibly using Model-View-Controller
  • High cohesion and low coupling using well-defined APIs
  • Don’t Repeat Yourself (DRY)
  • Interfaces/APIs and implementations are separate, and following Law of Demeter. Classes don’t call other classes directly because they happen to be in the same archive
  • Using Domain Driven Design to keep objects related to a domain/component together
  • YAGNI or You Aren’t Going to Need It: Don’t build something that you don’t need now

Here is how a trivial shopping cart monolithic WAR archive might look like:

Java EE Monolithic

This monolithic application has:

  • Web pages, such as .xhtml files, for User, Catalog, and Order component, packaged in the root of the archive. Any CSS and JavaScript resources that are shared across different webpages are also packaged with these pages.
  • Classes for the three components are in separate packages in WEB-INF/classes directory. Any utility/common classes used by multiple classes are packed here as well.
  • Configuration files for each component are packaged inWEB-INF/classes/META-INF directory. Any config files for the application, such as persistence.xml and load.sql to connect and populate the data store respectively, are also packaged here.

It has the usual advantages of well-known architecture, IDE-friendly, easy sharing, simplified testing, easy deployment, and others. But also comes with disadvantages such as limited agility, obstacle for continuous delivery, “stuck” with a technology stack, growing technical debt, and others.

Even though microservices are all the raze these days, but monoliths are not bad. Even those that are not working for you may not benefit much, or immediately, from moving to microservices. Other approaches, such as just better software engineering and architecture, may help. Microservices is neither a free lunch or a silver bullet and requires significant investment to be successful such as service discovery, service replication, service monitoring, containers, PaaS, resiliency, and a lot more.

don’t even consider microservices unless you have a system that’s too complex to manage as a monolith.

Microservice Architecture for Java EE

Alright, I’ve heard about all of that but would like to see a before/after, i.e. how a monolith code base and how a refactored microservice codebase looks like.

First, lets look at the overall architecture:

Java EE Microservices

The key pieces in this architecture are:

  • Application should be functionally decomposed where User, Order, and Catalog components are packaged as separate WAR files. Each WAR file should have the relevant web pages (#15), classes, and configuration files required for that component.
    • Java EE is used to implement each component but there is no long term commitment to the stack as different components talk to each other using a well-defined API (#14).
    • Different classes in this component belong to the same domain and so the code is easier to write and maintain. The underlying stack can also change, possibly keeping technical debt to a minimum.
  • Each archive has its own database, i.e. no sharing of data stores. This allows each microservice to evolve and choose whatever type of datastore – relational, NoSQL, flat file, in-memory or some thing else – is most appropriate.
  • Each component will register with a Service Registry. This is required because multiple stateless instances of each service might be running at a given time and their exact endpoint location will be known only at the runtime (#17).Netflix Eureka, EtcdZookeeper are some options in this space (more details).
  • If components need to talk to each other, which is quite common, then they would do so using a pre-defined API. REST for synchronous or Pub/Sub for asynchronous communication are the common means to achieve this.In our case, Order component discovers User and Catalog service and talks to them using REST API.
  • Client interaction for the application is defined in another application, Shopping Cart UI in our case. This application mostly discover the services from Service Registry and compose them together. It should mostly be a dumb proxy where the UI pages of different components are invoked to show the interface (#18).A common look-and-feel can be achieved by providing a standard CSS/JavaScript resources.

This application is fairly trivial but at least highlights some basic architectural differences.

Monolith vs Microservice

Some of the statistics for the monolith and microservices-based applications are compared below:

Characteristic Monolith Microservice
Number of archives  1  5

  • Contracts (JAR, ~4 KB)
  • Order (WAR, ~7 KB)
  • User (WAR, ~6 KB)
  • Catalog (WAR, ~8 KB)
  • Web UI (WAR, 27 KB)
Web pages  8  8 (see below)
Configuration Files  4

  • web.xml
  • template.xhtml
  • persistence.xml
  • load.sql
 3 per archive

  • persistence.xml
  • load.sql
  • web.xml
Class files  12 26

  • Service registration for each archive
  • Service discovery classes
  • Application class for each archive
Total archive size  24 KB  ~52 KB (total)

Code base for the monolithic application is at: github.com/arun-gupta/microservices/tree/master/monolith/everest

Code base for the microservices-enabled application is at: github.com/arun-gupta/microservices/tree/master/microservice

Issues and TODOs

Here are the issues encountered, and TODOs, during refactoring of the monolith to a microservices-based application:

  • Java EE already enables functional decomposition of an application using EAR packaging. Each component of an application can be packaged as a WAR file and be bundled within an EAR file. They can even share resources that way. Now that is not a true microservices way, but this could be an interim step to get you started. However, be aware that@FlowScoped bean is not activated in an EAR correctly (WFLY-4565).
  • Extract all template files using JSF Resource Library Templates.
    • All web pages are currently in everest module but they should live in each component instead (#15).
    • Resource Library Template should be deployed at a central location as opposed to packaged with each WAR file (#16).
  • Breakup monolithic database into multiple databases require separate persistence.xml and DDL/DML scripts for each application. Similarly, migration scripts, such as using Flyway, would need to be created accordingly.
  • A REST interface for all components, that need to be accessed by another one, had to be created.
  • UI is still in a single web application. This should instead be included in the decomposed WAR (#15) and then composed again in the dumb proxy. Does that smell like portlets?
  • Deploy the multiple WAR files in a PaaS (#12)
  • Each microservice should be easily deployable in a container (#6)

Here is the complete list of classes for the monolithic application:

Here is the complete list of classes for the microservices-based application:

Once again, the complete code base is at github.com/arun-gupta/microservices.

Future Topics

Some of the future topics in this series would be:

  • Are containers required for microservices?
  • How do I deploy multiple microservices using containers?
  • How can all these services be easily monitored?
  • A/B Testing
  • Continuous Deployment using microservices and containers

What else would you like to see?

Enjoy!

 

Microservices at Red Hat Summit 2015

RedHat Summit Logo

Red Hat Summit is just a few days away. You’ll get the opportunity to learn a plethora of technologies and products in 277 sessions by 402 speakers. Complete agenda shows the list of all sessions, hands-on labs, tracks, and everything else.

When? Jun 21-25, 2015 (DevNation) and Jun 23-26, 2015 (Summit)
Where? Hynes Convention Center, Boston, MA
AgendaSummit and DevNation
Register: Summit and DevNation

Here is the list of 12 sessions that are talking about Microservices:

You’ll learn about:

  • Microservices technical and business aspects
  • Development, Testing, Production of microservices
  • Relationship with PaaS, Containers, and DevOps, Camel, Fabric8

Of course, you can catch any of the speakers in the hallway. You can also stop by at any of the booths in the exhibitor hall, and talk to us about what Red Hat offers for microservices.

I’m giving a few sessions as well, and really looking forward to see you there!

 

Docker Tools in Eclipse

Upcoming Docker Tooling for Eclipse gave a preview of Docker Tooling coming in Eclipse. This Tech Tip will show how to get started with it.

docker-logoeclipse-logo

NOTE: This is pretty bleeding edge and so some of the features may be half baked. But we are looking for all the feedback!

The Docker tooling is aimed at providing at minimum the same basic level features as the command-line interface, but also provide some advantages by having access to a full fledged UI.

Install Docker Tools Plugins

  • Download and Install JBoss Developer Studio 9.0 Nightly, take defaults through out the installation. Alternatively, download Eclipse Mars latest build and configure JBoss Tools plugin from the update site http://download.jboss.org/jbosstools/updates/nightly/mars/.
  • Open JBoss Developer Studio 9.0 Nightly or Eclipse Mars.
  • Add a new site using the menu items: Help > Install New Software… > Add…. Specify the Name: as “Docker Nightly” and Location: as http://download.eclipse.org/linuxtools/updates-docker-nightly/.Eclipse Docker Tooling
  • Expand Linux Tools, select Docker Client and Docker Tooling:Eclipse Docker Tooling
  • Click on Next >, Next >, accept the license agreement, and click on Finish. This will complete the installation of plugins.Restart the IDE for changes to take effect.

Docker Explorer

The Docker Explorer provides a wizard to establish a new connection to a Docker daemon. This wizard can detect default settings if the user’s machine runs Docker natively (such as in Linux) or in a VM using Boot2Docker (such as in Mac or Windows). Both Unix sockets on Linux machines and the REST API on other OSes are detected and supported. The wizard also allows remote connections using custom settings.

  • Use the menu Window, Show View, Other…. Type “docker” to see the output as:Docker Eclipse Tools
  • Select Docker Explorer to open the explorer.Docker Eclipse Tools
  • Click on the link in this window to create a connection to Docker Host. Specify the settings as shown:Docker Eclipse ToolsMake sure to get IP address of the Docker Host using docker-machine ip command.Also, make sure to specify the correct directory for .docker on your machine.
  • Click on Test Connection to check the connection. This should show the output as:Docker Eclipse ToolsClick on OK and Finish to exit out of the wizard.
  • Docker Explorer itself is a tree view that handles multiple connections and provides users with quick overview of the existing images and containers.Docker Eclipse Tools
  • Customize the view by clicking on the arrow in toolbar:Docker Eclipse Tools
  • Built-in filters can show/hide intermediate and dangling images, as well as stopped containers.Docker Eclipse Tools

Docker Images

The Docker Images view lists all images in the Docker host selected in the Docker Explorer view. This view allows user to manage images, including:

  • Pull/push images from/to the Docker Hub Registry (other registries will be supported as well, #469306)
  • Build images from a Dockerfile
  • Create a container from an image

Lets take a look at it.

  • Use the menu  Window, Show View, Other…, select Docker Images. It shows the list of images on Docker Host:Docker Eclipse Tools
  • Right-click on the image ending with wildfly:latest and click on the green arrow in the toolbar. This will show the following wizard:Docker Eclipse ToolsBy default, all exports ports from the image are mapped to random ports on the host interface. This setting can be changed by unselecting the first checkbox and specify exact port mapping.Click on Finish to start the container.
  • When the container is started, all logs are streamed into Eclipse Console:Docker Eclipse Tools

Docker Containers

Docker Containers view lets the user manage the containers. The view toolbar provides commands to start, stop, pause, unpause, display the logs and kill containers.

  • Use the menu Window, Show View, Other…, select Docker Containers. It shows the list of running containers on Docker Host:Docker Eclipse Tools
  • Pause the container by clicking on the pause button in the toolbar (#469310). Show the complete list of containers by clicking on the View Menu, Show all containers.

    Docker Eclipse Tools

  • Select the paused container, and click on the green arrow in the toolbar to restart the container.
  • Right-click on any running container and select Display Log to view the log for this container.

    Docker Eclipse Tools

Information and Inspect on Images and Containers

Eclipse Properties view is used to provide more information about the containers and images.

  • Just open the Properties View and click on a Connection, Container, or Image in any of the Docker Explorer View, Docker Containers View, or Docker Images View. This will fill in data in the Properties view.
  • Info view is shown as:

    Docker Eclipse Tools

  • Inspect view is shown as:

    Docker Eclipse Tools

The code is hosted in Linux Tools project.

File your bugs at: bugs.eclipse.org/bugs/enter_bug.cgi?product=Linux%20Tools and use “Docker” component. Talk to us on IRC.

Enjoy!