Category Archives: couchbase

Service Discovery with Java and Database application in Kubernetes

containers, couchbase, javaee, wildfly, , ,

This blog will show how a simple Java application can talk to a database using service discovery in Kubernetes.

 Kubernetes Logo WildFly Logo

Service Discovery with Java and Database application in DC/OS explains why service discovery is an important aspect for a multi-container application. That blog also explained how this can be done for DC/OS.

Let’s see how this can be accomplished in Kubernetes with a single instance of application server and database server. This blog will use WildFly for application server and Couchbase for database.

This blog will use the following main steps:

  • Start Kubernetes one-node cluster
  • Kubernetes application definition
  • Deploy the application
  • Access the application

Start Kubernetes Cluster

Minikube is the easiest way to start a one-node Kubernetes cluster in a VM on your laptop. The binary needs to be downloaded first and then installed.

Complete installation instructions are available at github.com/kubernetes/minikube.

The latest release can be installed on OSX as as:

It also requires kubectl to be installed. Installing and Setting up kubectl provide detailed instructions on how to setup kubectl. On OSX, it can be installed as:

Now, start the cluster as:

The kubectl version command shows more details about the kubectl client and minikube server version:

More details about the cluster can be obtained using the kubectl cluster-info command:

Kubernetes Application Definition

Application definition is defined at github.com/arun-gupta/kubernetes-java-sample/blob/master/service-discovery.yml. It consists of:

  • A Couchbase service
  • Couchbase replica set with a single pod
  • A WildFly replica set with a single pod
The key part is where the value of the COUCHBASE_URI environment variable is name of the Couchbase service. This allows the application deployed in WildFly to dynamically discovery the service and communicate with the database.

arungupta/couchbase:travel Docker image is created using github.com/arun-gupta/couchbase-javaee/blob/master/couchbase/Dockerfile.

arungupta/wildfly-couchbase-javaee:travel Docker image is created using github.com/arun-gupta/couchbase-javaee/blob/master/Dockerfile.

Java EE application waits for database initialization to be complete before it starts querying the database. This can be seen at github.com/arun-gupta/couchbase-javaee/blob/master/src/main/java/org/couchbase/sample/javaee/Database.java#L25.

Deploy Application

This application can be deployed as:

The list of service and replica set can be shown using the command kubectl get svc,rs:

Logs for the single replica of Couchbase can be obtained using the command kubectl logs rs/couchbase-rs:

Logs for the WildFly replica set can be seen using the command kubectl logs rs/wildfly-rs:

Access Application

The kubectl proxy command starts a proxy to the Kubernetes API server. Let’s start a Kubernetes proxy to access our application:

Expose the WildFly replica set as a service using:

The list of services can be seen again using kubectl get svc command:

Now, the application is accessible at:

A formatted output looks like:

Now, new pods may be added as part of Couchbase service by scaling the replica set. Existing pods may be terminated or get rescheduled. But the Java EE application will continue to access the database service using the logical name.

This blog showed how a simple Java application can talk to a database using service discovery in Kubernetes.

For further information check out:

  • Kubernetes Docs
  • Couchbase on Containers
  • Couchbase Developer Portal
  • Ask questions on Couchbase Forums or Stack Overflow
  • Download Couchbase

Service Discovery with Java and Database application in DC/OS

containers, couchbase

This blog will show how a simple Java application can talk to a database using service discovery in DC/OS.

DC/OS logo

Why Service Discovery?

An application typically consist of multiple components such as an application server, a database, a web server, caching and messaging server. Typically, multiple replicas of each component would run based upon the needs of your application. Deploying this application using a container orchestration framework means that each replica would run as a container. So, an application is typically deployed as multi-container application.

Each container is assigned a unique IP address for its lifetime. But containers are ephemeral and may terminate and rescheduled on a different host by the orchestration framework. A container is typically assigned a different IP address in that case. This means an application deployed in application server cannot rely upon the IP address of the database. This is where service discovery is required.

So, multiple replicas of a component are assigned a logical name. For example, web for all the application server containers and db for all the database containers. Now, an application can talk to the database containers using the logical service name. This allows the database containers to be rescheduled anywhere in the cluster, and also scale up and down dynamically.

Let’s see how this can be accomplished in DC/OS with a single instance of application server and database server. This blog will use WildFly for application server and Couchbase for database.

Couchbase Cluster on Mesos with DC/OS provide more details on how to setup a Couchbase cluster on DC/OS.

This blog will use the following main steps:

  • Setup DC/OS Cluster
  • Marathon application definition
  • Deploy the application

The complete source code used in this blog is at github.com/arun-gupta/dcos-java-database.

Many thanks to @unterstein for creating the Maven plugin and helping me understand the inner workings of DC/OS.

Setup DC/OS Cluster

DC/OS cluster can be easily created using the CloudFormation template. Detailed instructions, including system requirements and screenshots and setup, are available at Installing DC/OS on AWS.

CloudFormation output looks as shown:

DC/OS Cluster CloudFormation Output

Note down the value shown for the keys DnsAddress and PublicSlaveDnsAddress. The value of the first key can be used to access DC/OS GUI and looks like:

DC/OS Cluster Console Default Output

Configure DC/OS CLI as explained in CLI. In short, the following commands are used:

  • dcos config set core.dcos_url http://${DnsAddress} Replace ${DnsAddress} with the corresponding value from the CloudFormation output.
  • dcos auth login
  • dcos config show core.dcos_acs_token. If not already done, clone the repo from github.com/arun-gupta/dcos-java-database. Create a new file.dcos-token and copy the output from the command in this file.
  • dcos package install marathon-lb

Marathon Application Definition

Marathon framework is used to schedule containers in DC/OS. A marathon application can be defined by providing an application definition.

As mentioned earlier, this blog will show how a simple Java application can talk to a database. We’ll use a Java EE application deployed in WildFly and use Couchbase as the database. The application definition looks like:

What are the key points in this application definition?

  • Application has two containers: database and web. The web container has a dependency on the database container defined using dependencies attribute.
  • database container uses arungupta/couchbase:travel Docker image. This image is created from github.com/arun-gupta/couchbase-javaee/tree/master/couchbase. It uses Couchbase base image and uses Couchbase REST API to pre-configure the database. A sample bucket is also loaded in the database as well.
  • web container uses arungupta/wildfly-couchbase-javaee:travel image. This image is created from github.com/arun-gupta/couchbase-javaee/blob/master/Dockerfile. This is a Java EE 7 application bundled in WildFly. The app uses COUCHBASE_URI as an environment variable to connect to the Couchbase database. The value of this environment variable is configured to use DNS service discovery and is derived as explained in Virtual Networks.

Make sure to change the value of HAPROXY_0_VHOST to match the value of ${PublicSlaveDnsAddress} from CloudFormation output. The label HAPROXY_0_VHOST instructs Marathon-LB to expose the Docker container, the WildFly application server in our case, on the external load balancer with a virtual host. The 0 in the label key corresponds to the servicePort index, beginning from 0. If you had multiple servicePort definitions, you would iterate them as 0, 1, 2, and so on. Deploying an internally and externally load-balanced app with marathon-lb provide more details about how to configure marathon-lb.

Service Discovery and Load Balancing provide more details about service discovery and load balancing in DC/OS.

Deploy the Application using Maven

The application can be deployed using dcos-maven-plugin.

Plugin looks like:

Main points in this fragment are:

  • Plugin version is 0.2. That indicates the plugin is still in early stages of development.
  • dcosUrl is the value of ${DnsAddress} key from CloudFormation output. This address is used for deployment of the application.
  • <deployable> element enables different types of deployment – app, group or pods. This element is a hint for the plugin and should likely go away in a future version as Marathon API consolidates. Follow #11 for more details.

Other details and configuration about the plugin are at dcos-maven-plugin.

Deploy the application:

The following output is shown:

Here are some of the updated output from DC/OS console.

First updated Services tab:
DC/OS Cluster Web Application

Two applications in the service:
DC/OS Cluster Web Application

Database application has one task:
DC/OS Cluster Web Application

Status of database task:Database Service Discovery

Logs from the database task:
DC/OS Cluster Web Application

It shows the output from Couchbase REST API for configuring the server.

Status of web task:
DC/OS Cluster Web Application

Logs from web task:
DC/OS Cluster WildFly Output

It shows that the Java EE application is deployed successfully.

Access the application:

The address is the value of the key ${PublicSlaveDnsAddress} from CloudFormation output. A formatted output, for example with jq, looks like:

That’s it!

As mentioned earlier, the complete source code used in this blog is at github.com/arun-gupta/dcos-java-database.

This blog showed how a simple Java application can talk to a database using service discovery in DC/OS.

For further information check out:

  • DC/OS Docs
  • Couchbase on Containers
  • Couchbase Developer Portal
  • Ask questions on Couchbase Forums or Stack Overflow
  • Download Couchbase

Source: blog.couchbase.com/service-discovery-java-database-dcos/

NoSQL Simplifies Database DevOps

couchbase, devops

Does your organization want to simplify Database DevOps?
Is your database becoming a bottleneck to innovate rapidly?
Do you want to save millions of $$ in database licensing cost?

Read on!

State of Database DevOps

State of Database DevOps is a survey on DevOps adoption rates among SQL Server professionals. Over 1000 SQL Server professionals responded to the survey. The respondents came from across the globe and represent a wide range of job roles, company sizes and industries.

There are some good findings in the survey results. A few key findings worth highlighting here:

the greatest challenge with integrating database changes into a DevOps process would still be synchronizing application and database changes

Another one …

The greatest drawback identified with traditional siloed database development is the increased risk of failed deployments or downtime when introducing changes. This is closely followed by slow development and release cycles and the inability to respond quickly to changing business requirements

And another one …

Increasing the speed of delivery of database changes and freeing developers up to do more value added work are the key drivers for automating the delivery of database changes

The challenges highlighted here are not mentioned in the context of SQL server only, but would be applicable towards any relational database. You may be using Oracle, Postgres, MySQL, MariaDB or any other relational database for that matter and would be very much facing these issues. Why?

Why is Relational not well suited for Database DevOps?

It’s common for an application to operate on data from multiple tables in a RDBMS. For example, placing an order may use Customer, Order and Product tables. Each table has multiple columns with standard data types specific to a database. Tables may have primary, reference and foreign key constraints. Developers building applications using a relational database typically use an Object Relational Mapper (ORM), such as Hibernate or Java Persistence API for Java developers. There are similar ORM for other languages as well. ORMs captures the underlying complex database structure and allow programmers to build applications naturally using their language.

ORMs also use a persistence provider and allows your application to be independent of the underlying database. This persistence provider creates a binding between the language specific class to the database structure. For example, it maps a class to a table or multiple tables, binds the language data types to the types defined in the database and captures the relationship between tables. Theoretically, a programmer can use a different persistence provider to use a different RDBMS for the application. But this is far from a practical experience!

Any database change requires the ORM classes to be updated otherwise the application may not work. For example, adding a new table may mean a new Java class or updating an existing class. Change of a data type in a column requires the class definition to be updated otherwise the application will not even compile. Adding a new column means adding a new field in the class. Any change requires the classes to be updated and the application needs to be repackaged.

Changes in database structure are required all the time to meet the evolving needs of business. But if the DBAs make a database change and the ORM classes are not updated then there is a disconnect. Application deployment needs to be coordinated with the updating the database schema. There are tools like Flyway, Liquibase and others that integrate application and database deployment. But developers are often not allowed to make any direct changes to the production database. A disconnect would result in your application not working and the business to suffer. DevOps practices can definitely help solve these issues as it requires a close collaboration between developers that are building applications and DBAs that are updating database scripts.

But as the survey reports, more than 50% of the respondents do not have DevOps adopted today.

Database DevOps Adoption

There are challenges even if you were to integrate database changes into a DevOps process.

Database DevOps Challenges

Synchronizing application and database changes where the ORM classes need to be synchronized with the backend database structure is the biggest challenge. DBAs may want to structure the database in a certain way which may not be optimal for application development. Applying consistency across application and database development is the next major challenge for ensuring a seamless database DevOps.

A siloed development has serious issues on your ability to rapidly innovate and deliver value to your business.

Database DevOps Drawbacks

As shown in this image, failed deployments when introducing changes, slow development/release cycles and inability to respond to business needs account for over 60% of the drawbacks.

Speed of delivery of database changes is the biggest concern for database DevOps.

Database DevOps Driver

So what do you do?

How does NoSQL simplify Database DevOps?

NoSQL document database helps to simplify database DevOps!

How does NoSQL simplify database DevOps?

  • Schema flexibility – Developers need a single database that can store rapidly changing structured, semi-structured and unstructured data. NoSQL document database offers schema flexibility by allowing developers operate directly on JSON data and derive meaning out of it
  • No impedance mismatch – With no ORM for the application, there is no impedance mismatch between domain classes and database structure. Only the application code needs to be updated and no coordination is required with the schema changes
  • Scalability –  One of the drawbacks mentioned in the report is the inability to adapt to changing business requirements. This highlights scalability as a major DevOps challenge. If the volume of data, the number of queries, or the types of indexes required to support the application changes the database needs to change to accommodate those changes. Not in weeks or months, but today! No SQL databases run on commodity hardware and has a scale-out architecture as opposed to scale-up with RDBMS. Sharding can help with scalability in RDBMS but that’s an extra complexity that now need to be dealt with.

Learn more about why enterprises move to NoSQL.

Which NoSQL database is preferred by GE, Marriott, Verizon, United, LinkedIn, DIRECTV and many others?

What are some other advantages of Couchbase?

  • Homogeneous distributed architecture – no master/slave
  • SQL-like query language to query JSON documents
  • Auto-sharding using vBuckets
  • Memory-first architecture reduces the need for an additional caching layer
  • Cross-data center replication
  • Multiple SDKs
  • Database on server or mobile device, with a complete synchronization between them
  • Different container orchestration frameworks

NoSQL is not a panacea by any means. If you are building a system that needs complex transaction logic or real-time data warehousing, then RDBMS may be a better fit. However it addresses your scalability and agility concerns and simplifies database DevOps.

Here is a great video on migrating from relational database to NoSQL:

Here is another interesting video that shows why Marriott transitioned from Relational to NoSQL:

A lot more videos are available on Couchbase Connect 2016.

And some more relevant blogs:

  • Making the Shift from Relational to NoSQL: How to Get Started (whitepaper)
  • Moving from Oracle database to Couchbase
  • Moving from MySQL to Couchbase
  • Moving from SQL Server to Couchbase – Part 1 (data modeling), Part 2 (data migration), Part 3 (coming)
  • Moving from MongoDB to Couchbase
  • Migrate your MongoDB with Mongoose REST API to Couchbase with Ottoman
  • Migrating from Relational Database to Couchbase
  • Top 5 reasons companies replace Oracle with Couchbase

Talk to us:

  • Couchbase Developer Portal
  • Couchbase Forums
  • @couchhasedev and @couchbase

Source: blog.couchbase.com/nosql-simplifies-database-devops/

Stateful Containers using Portworx and Couchbase

containers, couchbase

Portworx Logo Couchbase Logo

Containers are meant to be ephemeral and so scale pretty well for stateless applications. Stateful containers, such as Couchbase, need to be treated differently. Managing Persistence for Docker Containers provide a great overview of how to manage persistence for stateful containers.

This blog will explain how to use Docker Volume Plugins and Portworx to create a stateful container.

Why Portworx?

Portworx is an easy-to-deploy container data services that provide persistence, replication, snapshots, encryption, secure RBAC and much more. Some of the benefits are:

  1. Container granular volumes – Portworx can take multiple EBS volumes per host and aggregate the capacity and derive container granular virtual (soft) volumes per container.
  2. Cross Availability Zone HA – Portworx will protect the data, at block level, across multiple compute instances across availability zones. As replication controllers restart pods on different nodes, the data will still be highly available on those nodes.
  3. Support for enterprise data operations – Portworx implements container granular snapshots, class of service, tiering on top of the available physical volumes.
  4. Ease of deployment and provisioning – Portworx itself is deployed as a container and integrated with the orchestration tools. DevOps can programmatically provision container granular storage with any property such as size, class of service, encryption key etc.

Setup AWS EC2 Instance

Portworx runs only on Linux or CoreOS. Setup an Ubuntu instance on AWS EC2:

  1. Start Ubuntu 14.04 instance with m3.medium instance type. Make sure to add port 8091 to inbound security rules. This allows Couchbase Web Console to be accessible afterwards.
  2. Login to the EC2 instance using the command: ssh -i ~/.ssh/arun-cb-west1.pem ubuntu@<public-ip>
  3. Update the Ubuntu instance: sudo apt-get update
  4. Install Docker: curl -sSL https://get.docker.com/ | sh. More detailed instructions are available at Get Docker for Ubuntu.
  5. Enable non-root access for the docker command: sudo usermod -aG docker ubuntu
  6. Logout from the EC2 instance and log back in

Create AWS EBS Volume

  1. Create an EBS volume for 10GB using EC2 console as explained in docs.
  2. Get the instance id from the EC2 console. Attach this volume to EC2 instance using this instance id, use the default device name /dev/sdf.
    Portworx EC2 Create Volume
  3. Use lsblk command in EC2 instance to verify that the volume is attached to the instance:

Portworx Container

  1. Physical storage makeup of each node, all the provisioned volumes in the cluster as well as their container mappings is stored in an etcd cluster. Start an etcd cluster:
  2. By default root mounted volumes are not allowed to be shared. Enable this using the command:
    This is explained more at Ubuntu Configuration and Shared Mounts.
  3. PX-Developer (px-dev) container on a server with Docker Engine turns that server into a scale-out storage node. PX-Enterprise, on the other hand, provides multi-cluster and multi-cloud support, where storage under management can be on-premise or in a public cloud like AWS.
    For this blog, we’ll start a px-dev container:
    Complete details about this command are available at Run PX with Docker.
  4. Look for logs using docker container logs -f px and watch out for the following statements:
  5. Check the status of attached volumes that are available to Portworx using sudo /opt/pwx/bin/pxctl status to see the output:
    It shows the total capacity available and used.

Docker Volume

  1. Let’s create a Docker volume:
    More details about this command are at Create Volumes with Docker.
  2. Check the list of volumes available using docker volume ls command:
    As shown, cbvol is created with pxd driver.

Couchbase with Portworx Volume

  1. Create a Couchbase container using the Portworx volume:
    Notice how /opt/couchbase/var where all Couchbase data is stored in the container is mapped to the cbvol volume on the host. This volume is mapped by Portworx.
  2. Login to Couchbase Web Console at http://<public-ip>:8091, use the login Administrator and password as password.
  3. Go to Data Buckets and create a new data bucket pwx:
    Couchbase Bucket with Portworx
  4. In EC2 instance, see the list of containers:
    etcd, px-dev and db containers are running.
  5. Kill the db container:
  6. Restart the database container as:
    Now, because cbvol is mapped to /opt/couchbase/var again, the data is preserved across restarts. This can be verified by accessing the Couchbase Web Consoleand checking on the pwx bucket created earlier.

Another interesting perspective is also at why database are not for containers?. Just because there is Docker, does not mean all your database needs should be Dockerized. But if you need to, then there are plenty of options and can be used in production-grade applications.

Want to learn more about running Couchbase in containers?

  • Couchbase on Containers
  • Couchbase Developer Portal
  • @couchhasedev and @couchbase

Source: blog.couchbase.com/stateful-docker-containers-portworx-couchbase/

Start Couchbase Using Docker Compose

containers, couchbase

Couchbase Forums has a question Can’t use N1QL on docker-compose. This blog will show how to run Couchbase using Docker Compose and run a N1QL query.

Docker ComposeN1QL

What is Docker Compose?

Docker Compose allows you to define your multi-container application with all of its dependencies in a single file, then spin your application up in a single command.

Docker Compose introduced v3 in Docker 1.13. How do you know what version of Docker are you running?

docker version command gives you that information:

Couchbase Docker Compose File

Now if you see this version of Docker, then you can use the following Compose file:

In this Compose file:

  • v3 version of Compose file. If you are using an older version of Docker, then you can consider using v2 version of Compose file.
  • arungupta/couchbase Docker image is used to start Couchbase server.  This image is created as explained at github.com/arun-gupta/docker-images/tree/master/couchbase. It uses Couchbase REST API to pre-configure the Couchbase server.
  • Ports 8091, 8092, 8093, 8094, 11210 are exposed.
  • Only a single replica of Couchbase server is started.

Couchbase can be started in a couple of ways using this Compose file.

Couchbase using Docker Compose on Single Docker Host

If you want to start Couchbase on a single host (such as provisioned by Docker for Mac or a single Docker Machine), then use the command:

This will show the warning message but starts Couchbase server:

Check the status of started service using the command docker-compose ps:

All the exposed ports are shown and Couchbase is accessible at http://localhost:8091. Use the credentials Administrator/password to access the web console.

Now you can create buckets and connect from CBQ and run N1QL queries. For example:

Typically, you may be able to scale the services started by Docker Compose using docker-compose scale command. But this will not be possible in our case as the ports are exposed. Scaling a service will cause port conflict.

The container can be brought down using the command docker-compose down.

Couchbase using Docker Compose on Multi-host Swarm-mode Cluster

Docker allows multiple hosts to be configured in a cluster using Swarm-mode. This can be configured using the command docker swarm init.

Once the cluster is initialized, then the Compose file can be used to start the cluster:

It shows the output:

This creates a Docker service and the status can be seen using the command docker service ls:

Check the tasks/containers running inside the service using the command docker service ps couchbase_db:

Here again, you can connect to the Couchbase server and run N1QL queries:

The service, and thus the container running in the service, can be terminated using the command docker service couchbase_db.

Any more questions? Catch us on Couchbase Forums.

You may also consider running Couchbase Cluster using Docker or read more about Deploying Docker Services to Swarm.

Want to learn more about running Couchbase in containers?

  • Couchbase on Containers
  • Couchbase Developer Portal
  • @couchhasedev and @couchbase

Source: blog.couchbase.com/couchbase-using-docker-compose/

Getting Started with Oracle Container Cloud Service

containers, couchbase, ,

Oracle Cloud Container Logo

Oracle Container Cloud Service is Oracle’s entry into the the world of managed container service. There are plenty of existing options:

  • Docker for AWS or Azure
  • Amazon Elastic Container Service
  • Google Container Engine
  • Azure Container Service
  • DC/OS by Mesosphere
  • OpenShift by Red Hat

This blog will explain how to get started with Oracle Container Cloud Service. A comparison of different managed services is started at Managed Container Service.

Before we jump into all the details, let’s try to clarify a couple of things about this offering from Oracle.

First, a bit about the name. “Oracle Cloud Container Service” seems more natural and intuitive since its a Container Service in Oracle Cloud. Wonder why is it called “Oracle Container Cloud Service”? Is it because “Oracle Container” is Oracle’s container orchestration framework and its a Cloud Service? Could that mean other orchestration frameworks be offered as a service as well?

Second, don’t confuse it with Oracle Application Container Cloud Service that allows to build cloud-native 12-factor applications using polyglot platform. Now, that confuses me further. Can the Container Service not be used to build 12-factor apps? Are cloud-native and containers mutually exclusive?

Anyway, this is causing more confusion than clarification :) Let’s move on!

One last thing before we dig in. Many thanks to Bruno Borges (@brunoborges) for pushing the buttons for cloud service activation. I don’t know the normal time for the free trial to be activated otherwise. And a much bigger thank to Mike Raab (@mikeraab) for helping me understand the details of Container Service.

Let’s get started!

  1. Get a Free Trial for Oracle Cloud. It takes a few days for the trial to be activated. The trial time bombs after 30 days so make sure you’ve time planned for evaluation. Each free trial comes with 6 OC3 nodes. OC3 is one of the compute node types available on Oracle Cloud. OC3 particularly is 1 OCPU (think vCPU on Amazon Web Services) and 7.5 GB RAM.
  2. Once the account is activated, you get an email as shown:oracle-cloud-welcome-emailThe important piece of information is username, temporary password, identity domain and My Services URL. The My Account URL link is only for account administration.
  3. Click on My Service URL, login using the values from email:oracle-cloud-services-login
     You get an opportunity to change your password afterwards
  4. Oracle Cloud dashboard shows up after logging in:oracle-cloud-services-dashboardA default set of services and their status is shown. The dashboard can also be customized by clicking on Customize Dashboard button on the top right.
  5. Getting to Oracle Container Cloud Service Console is a bit non-intuitive but you get it once you know it. Select Container Cloud Service tab, click on top-right corner and select Open Service Console:oracle-cloud-container-service-console-accessOr you can directly click on the link for Oracle Container Cloud Service Console in the welcome email. Service console looks like:oracle-cloud-container-service-console
  6. Click on Create Service:oracle-cloud-container-service-definitionOracle Container Container Service Instance Details provide more details about each of the field.What is a worker node? We’ll talk about it a bit later. But essentially this is where the container runs. We are asking for only one worker node.

    Its worth noting different capacities for the worker node:
    Oracle Cloud CPUs

    Confirm all the settings:

    oracle-cloud-container-service-definition-confirmation

    and click on Create> to start the service creation.

  7. Wait for about 30 minutes for the service to be created. After that the Service Console looks like:
    oracle-cloud-container-service-console-with-serviceWait, we asked for  one worker node and how come two OCPUs are being consumed.Each Oracle Container Cloud Service has at least two nodes – a manager node and one or more worker nodes. Manager node is responsible for administration of all the workers and and orchestrate containers on different worker nodes. Worker nodes can be organized in different resource pools to meet different workflow needs.

    And, so ~30 minutes are spent provisioning two nodes and installing container service components on each node. This is also evident in the service logs shown in Service Create and Delete History shown in the main Console page:

    No timestamp in the activity feels a bit too clean.

  8. One main question that I kept wondering all along is “when am I ready to deploy the containers?“. Apparently, not yet!A couple of more steps so hang in there …

    In your service, click on the top-right icon to select another menu:
    oracle-cloud-container-console-open

    Select Container Console.  So, now you are transitioning from Oracle Container Cloud Service Console to Container Console. Make sure to use the right terminology otherwise it gets confusing fast.

  9. This attempts to open Container Console but prompts the usual warningoracle-cloud-container-console-open-warning

    Just click on Proceed link. In a typical production setup, this will setup correctly using certificates and so this warning would not happen.

  10. This brings up a login screen:oracle-cloud-container-console-login
  11. Use the username and password specified during service creation earlier. Click on Login to see Container Console:oracle-cloud-container-console-default

Are we there yet?

Yes, now is the time to deploy containers. But we’ll cover that in a subsequent blog!

Just to recap on what is needed to get started with Oracle Container Cloud Service …

  1. Register for Oracle Cloud trial
  2. Login to main Oracle Cloud Dashboard
  3. Create a Oracle Container Cloud Service Instance
  4. Oracle Container Cloud Service Instance Console
  5. Container Console

All the steps need to be done once but a console inside a console inside a dashboard feels like Inception. The good thing is that the IP address of Container Console is a public IP address served by Oracle Cloud and can be used from anywhere.

Oracle Container Cloud Service Docs have lot more details about building and deploying applications using this Console.

In the next blog, we’ll see what it takes to run a Couchbase container using this console? Possibly a cluster of Couchbase across multiple hosts?

Want to learn more about running Couchbase in containers?

  • Couchbase on Containers
  • Couchbase Forums
  • Couchbase Developer Portal
  • @couchhasedev and @couchbase

Source: https://blog.couchbase.com/2017/february/getting-started-oracle-container-cloud-service

Microservice using Docker stack deploy – WildFly, Java EE and Couchbase

containers, couchbase, wildfly, , ,

There is plenty of material on microservices, just google it! I gave a presentation on refactoring monolith to microservices at Devoxx Belgium a couple of years back and it has good reviews:

This blog will show how Docker simplifies creation and shutting down of a microservice.

All code used in this blog is at github.com/arun-gupta/couchbase-javaee.

Microservice Definition using Compose

Docker 1.13 introduced a v3 of Docker Compose. The changes in the syntax are minimal but the key difference is addition of deploy attribute. This attribute allows to specify replicas, rolling update and restart policy for the container.

Our microservice will start a WldFly application server with a Java EE application pre-deployed. This application will talk to a Couchbase database to CRUD application data.

Here is the Compose definition:

In this Compose file:

  1. Two services in this Compose are defined by the name db and web attributes
  2. Image name for each service defined using image attribute
  3. The arungupta/couchbase:travel image starts Couchbase server, configures it using Couchbase REST API, and loads travel-sample bucket with ~32k JSON documents.
  4. The arungupta/couchbase-javaee:travel image starts WildFly and deploys application WAR file built from https://github.com/arun-gupta/couchbase-javaee. Clone that project if you want to build your own image.
  5. envrionment attribute defines environment variables accessible by the application deployed in WildFly. COUCHBASE_URI refers to the database service. This is used in the application code as shown at https://github.com/arun-gupta/couchbase-javaee/blob/master/src/main/java/org/couchbase/sample/javaee/Database.java.
  6. Port forwarding is achieved using ports attribute

  7. depends_on attribute in Compose definition file ensures the container start up order. But application-level start up needs to be ensured by the applications running inside container. In our case, WildFly starts up rather quickly but takes a few seconds for the database to start up. This means the Java EE application deployed in WildFly is not able to communicate with the database. This outlines a best practice when building micro services applications: you must code defensively and ensure in your application initialization that the micro services you depend on have started, without assuming startup order. This is shown in the database initialization code at https://github.com/arun-gupta/couchbase-javaee/blob/master/src/main/java/org/couchbase/sample/javaee/Database.java. It performs the following checks:

    1. Bucket exists
    2. Query service of Couchbase is up and running
    3. Sample bucket is fully loaded

This application can be started using docker-compose up -d command on a single host. Or a cluster of Docker engines in swarm-mode using docker stack deploy command.

Setup Docker Swarm-mode

Initialize Swarm mode using the following command:

This starts a Swarm Manager. By default, manager node are also worker but can be configured to be manager-only.

Find some information about this one-node cluster using the command docker info command:

This cluster has 1 node, and that is manager.

Alternatively, a multi-host cluster can be easily setup using Docker for AWS.

Deploy Microservice

The microservice can be started as:

This shows the output:

WildFly and Couchbase services are started on this node. Each service has a single container. If the Swarm mode is enabled on multiple nodes then the containers will be distributed across multiple nodes.

A new overlay network is created. This allows multiple containers on different hosts to communicate with each other.

Verify that the WildFly and Couchbase services are running using docker service ls:

Logs for the service can be seen using docker service logs -f webapp_web:

Make sure to wait for the last log statement to show.

Access Microservice

Get 10 airlines from the microservice:

This shows the results as:

Docker for Java Developers workshop is a self-paced hands-on lab and allows you to get started with Docker easily.

Get a single resource:

Create a new resource:

Update a resource:

Delete a resource:

Detailed output from each of these commands is at github.com/arun-gupta/couchbase-javaee.

Delete Microservice

The microservice can be removed using  the command docker stack rm webapp:

Want to get started with Couchbase? Look at Couchbase Starter Kits.

Want to learn more about running Couchbase in containers?

  • Couchbase on Containers
  • Couchbase Forums
  • Couchbase Developer Portal
  • @couchhasedev and @couchbase

Source: https://blog.couchbase.com/2017/february/microservice-using-docker-stack-deploy-wildfly-javaee-couchbase

Deploy Docker Compose Services to Swarm

containers, couchbase, techtip, , ,

Docker 1.13 introduced a new version of Docker Compose. The main feature of this release is that it allow services defined using Docker Compose files to be directly deployed to Docker Engine enabled with Swarm mode. This enables simplified deployment of multi-container application on multi-host.

Docker 1.13

This blog will show use a simple Docker Compose file to show how services are created and deployed in Docker 1.13.

Here is a Docker Compose v2 definition for starting a Couchbase database node:

This definition can be started on a Docker Engine without Swarm mode as:

This will start a single replica of the service define in the Compose file. This service can be scaled as:

If the ports are not exposed then this would work fine on a single host. If swarm mode is enabled on on Docker Engine, then it shows the message:

Docker Compose gives us multi-container applications but the applications are still restricted to a single host. And that is a single point of failure.

Swarm mode allows to create a cluster of Docker Engines. With 1.13, docker stack deploy command can be used to deploy a Compose file to Swarm mode.

Here is a Docker Compose v3 definition:

As you can see, the only change is the value of version attribute. There are other changes in Docker Compose v3. Also, read about different Docker Compose versions and how to upgrade from v2 to v3.

Enable swarm mode:

Other nodes can join this swarm cluster and this would easily allow to deploy the multi-container application to a multi-host as well.

Deploy the services defined in Compose file as:

A default value of Compose file here would make the command a bit shorter. #30352 should take care of that.

List of services running can be verified using docker service ls command:

The list of containers running within the service can be seen using docker service ps command:

In this case, a single container is running as part of the service. The node is listed as moby which is the default name of Docker Engine running using Docker for Mac.

The service can now be scaled as:

The list of container can then be seen again as:

Note that the containers are given the name using the format <service-name>_n. Both the containers are running on the same host.

Also note, the two containers are independent Couchbase nodes and are not configured in a cluster yet. This has already been explained at Couchbase Cluster using Docker and a refresh of the steps is coming soon.

A service will typically have multiple containers running spread across multiple hosts. Docker 1.13 introduces a new command docker service logs <service-name> to stream the log of service across all the containers on all hosts to your console. In our case, this can be seen using the command docker service logs couchbase_db and looks like:

The preamble of the log statement uses the format <container-name>.<container-id>@<host>. And then actual log message from your container shows up.

At first instance, attaching container id may seem redundant. But Docker services are self-healing. This means that if a container dies then the Docker Engine will start another container to ensure the specified number of replicas at a given time. This new container will have a new id. And thus it allows  to attach the log message from the right container.

So a quick comparison of commands:

 Docker Compose v2  Docker compose v3
 Start services docker-compose up -d docker stack deploy --compose-file=docker-compose.yml <stack-name> 
 Scale service docker-compose scale <service>=<replicas> docker service scale <service>=<replicas>
 Shutdown docker-compose down docker stack rm <stack-name>
 Multi-host No Yes

Want to get started with Couchbase? Look at Couchbase Starter Kits.

Want to learn more about running Couchbase in containers?

  • Couchbase on Containers
  • Couchbase Forums
  • Couchbase Developer Portal
  • @couchhasedev and @couchbase

Source: https://blog.couchbase.com/2017/deploy-docker-compose-services-swarm

Analyze Donald Trump Tweets with Couchbase and N1QL

couchbase, , , ,

AWS Serverless Lambda Scheduled Events to Store Tweets in Couchbase explained how to store tweets in Couchbase using AWS Serverless Lambda. Now, this Lambda Function has been running for a few days and has collected 269 tweets from @realDonaldTrump. This blog , inspired by SQL on Twitter: Analysis Made Easy Using N1QL, will show how these tweets can be analyzed using N1QL.

trump-tweets
 N1QL is a SQL-like query language from Couchbase that operates on JSON documents. N1QL and SQL Differences provide differences between N1QL and SQL. Let’s use N1QL to reveal some interesting information from @realDonaldTrump‘s tweets.

Many thanks to Sitaram from N1QL team to help hack the queries.

How Many Tweets

First query is to find out how many tweets are available in the database. The query is pretty simple:

Query:

As you notice, the syntax is very similar to SQL. SELECT, COUNT and FROM clauses are what you are already familiar with from SQL syntax. tweet_count is an alias defined for the returned result. twitter is the bucket where all the JSON documents are stored.

Results:

The result is a JSON document as well.

Tweet Sample JSON Document

In order to write queries on a JSON document, you need to know the structure of the document. The next query will give you that.

Query:

The new clause introduced here is LIMIT. This allows to restrict the number of objects that are returned in a result set of SELECT.

Results:

Top 5 Tweeting Days

After the basic queries are out of the way, let’s look at some interesting data now.

What are the top 5 days on which @realDonaldTrump tweeted and the tweet count?

Query:

Usual GROUP BY and ORDER BY SQL clauses perform the same function.

N1QL Functions apply a function to values. The createdAt field is returned a number as a String. TO_NUM function converts the String to a number. MILLIS_TO_STR function converts the String to a date. Finally, SUBSTR function extracts the relevant part of the date.

Results:

Jan 17th, 2017 is the most tweeted day. Now, this result is of course restricted to the data from the JSON documents stored in the database.

Does anybody have a more comprehensive database of @realDonaldTrump tweets?

Tweet Frequency

OK, our database shows that that maximum number of tweets in a day were 13. How do I find out how many days @realDonaldTrump tweeted a certain number of times?

Query:

This is easily achieved using N1QL nested queries.

Results:

In 47 days, there is only one day with a single tweet. A sum total of tweet_count shows that there is no single day without a tweet :)

Most Common Hour In a Day To Tweet

@realDonaldTrump is known to tweet at 3am. Let’s take a look what are the most common hours for him to tweet.

Query:

Results:

Now seems like the controversial tweets come at 3am. But 39 tweets are coming at 1pm ET, likely right after lunch and while having a dessert 😉

Common Day of The Week to Tweet

Let’s find out what are the most common day of the week to tweet.

Query:

DATE_PART_STR is a new function returns date part of the date. Further day_of_week attribute is used to get day of the week.

Results:

Seems like Tuesday is the most common day to tweet. Then comes Sunday and Wednesday at the same level. The performance tends to fizzle out closer to the weekend.

Here is a nice chart that shows the same trend:

realdonaldtrump-tweets-per-day

#22417 should allow to report the weekday part in English.

Top 5 Mentions in Tweets

Query:

userMentionEntities is a nested array in the JSON document. UNNEST conceptually performs a join of the nested array with its parent object. Each resulting joined object becomes an input to the query.

Results:

Needless to say, he mentions his own name the most in tweets! And his two favorite TV stations Fox News and CNN.

Top 5 Tweets with RTs

Lambda Function wakes up every 3 hours and fetches the latest tweets. So the database is a snapshot of tweets and associated information such as RTs and Favorites. So depending upon when the tweet was archived, the RTs and Favorites may not be an accurate representation. But given this information, let’s take a look at the tweets with most RTs.

Query:

Pretty straight forward query.

Results:

Original vs RTs

How many of tweets were written vs retweeted?

Query:

Results:

Most of the tweets are original with only a few RTs.

Most Common Words in Tweet

Query:

This query uses SPLIT function that

Results:

Frequency of words “media”, “fake” and “America” in tweets

Query:

LOWER function is used to compare words independent of the case.

Result:

Lambda function will continue to store tweets in the database.

Try these queries yourself?

N1QL References

  • N1QL Interactive Tutorial
  • N1QL Cheatsheet
  • N1QL Language Reference
  • Run Your First N1QL Query

Source: https://blog.couchbase.com/2017/january/analyze-donald-trump-tweets-couchbase-n1ql

AWS Serverless Lambda Scheduled Events to Store Tweets in Couchbase

couchbase, , , ,

This blog has explained a few Serverless concepts with code samples:

  • Serverless FaaS with AWS Lambda and Java
  • AWS IoT Button, Lambda and Couchbase
  • Microservice using AWS API Gateway, AWS Lambda and Couchbase
  • Microservice using AWS Serverless Application Model and Couchbase

This particular blog entry will show how to use AWS Lambda to store tweets of a tweeter in Couchbase. Here are the high level components:

 

lambda-twitter-couchbase

The key concepts are:

  • Lambda Function deployed using Serverless Application Model
  • Triggered every 3 hours using Scheduled Events
  • Uses Twitter4J API to query new tweets since the last fetch
  • Use Couchbase Java SDK API to store JSON documents in the Couchbase Server

Complete sample code for this blog is available at github.com/arun-gupta/twitter-n1ql.

Serverless Application Model

Serverless Application Model, or SAM, defines simplified syntax for expressing serverless resources. SAM extends AWS CloudFormation to add support for API Gateway, AWS Lambda and Amazon DynamoDB. Read more details in Microservice using AWS Serverless Application Model and Couchbase.

For our application, SAM template is available at github.com/arun-gupta/twitter-n1ql/blob/master/template-example.yml and shown below:

What do we see here?

  • Function is packaged and available in a S3 bucket
  • Handler class is org.sample.twittter.TwitterRequestHandler and is at github.com/arun-gupta/twitter-n1ql/blob/master/twitter-feed/src/main/java/org/sample/twitter/TwitterRequestHandler.java. It looks like:
    By default, this class reads the twitter handle of Donald Trump. More fun on that coming in a subsequent blog.
  • COUCHBASE_HOST and COUCHBASE_BUCKET_PASSWORD are environment variables that provide EC2 host where Couchbase database is running and the password of the bucket.
  • Function can be triggered by different events. In our case, this is triggered every three hours. More details about the expression used here are at Schedule Expressions Using Rate or Cron.

Fetching Tweets using Twitter4J

Tweets are read using Twitter4J API. It is an unofficial Twitter API that provides a Java abstraction over Twitter REST API. Here is a simple example:

 

Twitter4J Docs and Javadocs are pretty comprehensive.

Twitter API allows to read only last 200 tweets. Lambda function is invoked every 3 hours. The tweet frequency of @realDonaldTrump is not 200 every 3 hours, at least yet. If it does reach that dangerous level then we can adjust the rate to trigger Lambda function more frequently.

JSON representation of each tweet is stored in Couchbase server using Couchbase Java SDK. AWS Lambda supports Node, Python and C#. And so you can use Couchbase Node SDK, Couchbase Python SDK or Couchbase .NET SDK to write these functions as well.

Twitter4J API allows to fetch tweets since the id of a particular tweet. This allows to ensure that duplicate tweets are not fetched. This requires us to sort all tweets in a particular order and then pick the id of the most recent tweet. This was solved using the simple N1QL query:

The syntax is very SQL-like. More on this in a subsequent blog.

Store Tweets in Couchbase

The final item is to store the retrieved tweets in Couchbase.

Value of COUCHABSE_HOST environment variable is used to connect to the Couchbase instance. The value of COUCHBASE_BUCKET_PASSWORD environment variable is to connect to the secure bucket where all JSON documents are stored. Its very critical that the bucket be password protected and not directly specified in the source code. More on this in a subsequent blog.

The JSON document is upserted (insert or update) in Couchbase using the Couchbase Java API:

 

This Lambda Function has been running for a few days now and has captured 258 tweets from @realDonaldTrump.

serverless-lambda-couchbase-twitter-bucket

An interesting analysis of his tweets is coming shortly!

Talk to us:

  • Couchbase Forums
  • Couchbase Database Developer Portal
  • @couchbasedev and @couchbase

Complete sample code for this blog is available at github.com/arun-gupta/twitter-n1ql.

Source: https://blog.couchbase.com/2017/january/aws-serverless-lambda-scheduled-events-tweets-couchbase

Couchbase Customer Advisory Note – Security

couchbase

In light of the recent widespread news about security vulnerabilities in MongoDB and Elasticsearch, we want to proactively remind our customers of Security Best Practices for Couchbase.

At this time there have been no known ransomware attacks on Couchbase, and no new security vulnerabilities have been identified in the product. This advisory is in the spirit of ‘forewarned is forearmed’.

Comprehensive security planning is a complex topic, but getting started with Security Basics is not. This Advisory Note is intended as a heads-up and reminder of general security best practices as well as Couchbase security capabilities available to you. First of all, let’s start with the basics. All Couchbase Server installations should ensure that:

  • Proper physical security (server access and backup storage) is maintained.
  • Couchbase Server nodes are behind a firewall so that they are not publically accessible. Here is how to configure network access to Couchbase using IP tables.
  • The server operating system is up to date with the latest security patches.
  • Delete the “default” bucket.
  • Secure in-transit data by using SSL connections for client/server and server/server communication.
  • Use a strong and unique bucket password for all data buckets.
  • Add security to your Couchbase mobile application
  • Encrypt Couchbase Lite databases

Additionally, customers should consult the following Couchbase resources in order to build a comprehensive security plan:

Documentation

  • Couchbase Server Security Considerations
  • Couchbase Server Introduction to Security
  • Couchbase Server Security Best Practices

Blogs

  • Configuring IPSec
  • N1QL Security
  • Skipping Default Bucket Creation
  • Security for Mobile Data Synchronization
  • Decentralized Security with Couchbase Mobile

As always, please reach out to us if you have any questions.

How to contact?

Source: https://blog.couchbase.com/2017/couchbase-customer-advisory-note-security

Microservice using AWS Serverless Application Model and Couchbase

couchbase, techtip, , , , ,

Amazon Web Services introduced Serverless Application Model, or SAM, a couple of months ago. It defines simplified syntax for expressing serverless resources. SAM extends AWS CloudFormation to add support for API Gateway, AWS Lambda and Amazon DynamoDB. This blog will show how to create a simple microservice using SAM. Of course, we’ll use Couchbase instead of DynamoDB!

This blog will also use the basic concepts explained in Microservice using AWS API Gateway, AWS Lambda and Couchbase. SAM will show the ease with which the entire stack for microservice can be deployed and managed.

As a refresher, here are key components in the architecture:

serverless-microservice

  • Client could be curl, AWS CLI/Console, Postman client or any other tool/API that can invoke a REST endpoint.
  • AWS API Gateway is used to provision APIs. The top level resource is available at path /books. HTTP GET and POST methods are published for the resource.
  • Each API triggers a Lambda function. Two Lambda functions are created, book-list function for listing all the books available and book-create function to create a new book.
  • Couchbase is used as a persistence store in EC2. All the JSON documents are stored and retrieved from this database.

Other blogs on serverless:

  • Microservice using AWS API Gateway, AWS Lambda and Couchbase
  • AWS IoT Button, Lambda and Couchbase
  • Serverless FaaS with Lambda and Java

Let’s get started!

Serverless Application Model (SAM) Template

An AWS CloudFormation template with serverless resources conforming to the AWS SAM model is referred to as a SAM file or template. It is deployed as a CloudFormation stack.

Let’s take a look at our SAM template:

This template is available at github.com/arun-gupta/serverless/blob/master/aws/microservice/template.yml.

SAM template Specification provide complete details about contents in the template. The key parts of the template are:

  • Defines two resources, both of Lambda Function type identified by AWS::Serverless::Function attribute. Name of the Lambda function is defined by Resources.<resource>.
  • Class for each handler is defined by the value of Resources.<resource>.Properties.Handler attribute
  • Java 8 runtime is used to run the Function defined by Resources.<resource>.Properties.Runtime attribute
  • Code for the class is uploaded to an S3 bucket, in our case to s3://serverless-microservice/microservice-http-endpoint-1.0-SNAPSHOT.jar
  • Resources.<resource>.Properties.Environment.Variables.COUCHBASE_HOST attribute value defines the host where Couchbase is running. This can be easily deployed on EC2 as explained at Setup Couchbase.
  • Each Lambda function is triggered by an API. It is deployed using AWS API Gateway. The path is defined by Events.GetResource.Properties.Path. HTTP method is defined using Events.GetResource.Properties.Method attribute.

Java Application

The Java application that contains the Lambda functions is at github.com/arun-gupta/serverless/tree/master/aws/microservice/microservice-http-endpoint.

Lambda function that is triggered by HTTP GET method is shown:

A little bit of explanation:

  • Each Lambda function needs to implement the interface com.amazonaws.services.lambda.runtime.RequestHandler.
  • API Gateway and Lambda integration require a specific input format and output format. These formats are defined as GatewayRequest and GatewayResponse classes.
  • Function logic uses Couchbase Java SDK to query the Couchbase database. N1QL query is used to query the database. The results and exception are then wrapped in GatewayRequest and GatewayResponse.

Lambda function triggered by HTTP POST method is pretty straightforward as well:

A bit of explanation:

  • Incoming request payload is retrieved from GatewayRequest
  • Document inserted in Couchbase is returned as response.
  • Like the previous method, Function logic uses Couchbase Java SDK to query the Couchbase database. The results and exception are then wrapped in GatewayRequest and GatewayResponse.

Build the Java application as:

Upload Lambda Function to S3

SAM template reads the code from an S3 bucket. Let’s create a S3 bucket:

us-west-2 region is one of the supported regions for API Gateway. S3 bucket names are globally unique but their location is region specific.

Upload the code to S3 bucket:

The code is now uploaded to S3 bucket. SAM template is ready to be deployed!

Deploy SAM Template

Deploy the SAM template:

It shows the output:

This one command deploys Lambda functions and REST Resource/APIs that trigger these Lambda functions.

Invoke the Microservice

API Gateway publishes a REST API that can be invoked by curl, wget, AWS CLI/Console, Postman or any other app that can call a REST API. This blog will use AWS Console to show the interaction.

API Gateway home at us-west-2.console.aws.amazon.com/apigateway/home?region=us-west-2#/apis shows:

AWS SAM Microservice API

Click on the API to see all the APIs in this resource:

AWS SAM Microservice API Resources

Click on POST to see the default page for POST method execution:

AWS SAM Microservice API POST

Click on Test to test the API:

AWS SAM Microservice API POST Input

Add the payload in Request Body and click on Test to invoke the API. The results are shown as below:

AWS SAM Microservice API POST Output

Now click on GET to see the default execution page:

AWS SAM Microservice API GET

Click on Test to test the API:

AWS SAM Microservice API GET Input

No request body is needed, just click on Test the invoke the API. The results are as shown:

AWS SAM Microservice API GET Output

Output from the Couchbase database is shown in the Response Body.

References

  • Deploying Lambda-based Applications
  • Serverless Architectures
  • AWS API Gateway
  • Creating a simple Microservice using Lambda and API Gateway
  • Couchbase Server Docs
  • Couchbase Forums
  • Follow us at @couchbasedev

Source: blog.couchbase.com/2017/january/microservice-aws-serverless-application-model-couchbase

Microservice using AWS API Gateway, AWS Lambda and Couchbase

couchbase, , , , , ,

This blog has explained the following concepts for serverless applications so far:

  • Serverless FaaS with AWS Lambda and Java
  • AWS IoT Button, Lambda and Couchbase

The third blog in serverless series will explain how to create a simple microservice using Amazon API Gateway, AWS Lambda and Couchbase.

Read previous blogs for more context on AWS Lambda.

Amazon API Gateway is a fully managed service that makes it easy for developers to create, publish, maintain, monitor, and secure APIs at any scale. Amazon API Gateway handles all the tasks involved in accepting and processing up to hundreds of thousands of concurrent API calls, including traffic management, authorization and access control, monitoring, and API version management.

Here are the key components in this architecture:

serverless-microservice

  • Client could be curl, AWS CLI, Postman client or any other tool/API that can invoke a REST endpoint.
  • API Gateway is used to provision APIs. The top level resource is available at path /books. HTTP GET and POST methods are published for the resource.
  • Each API triggers a Lambda function. Two Lambda functions are created, book-list function for listing all the books available and book-create function to create a new book.
  • Couchbase is used as a persistence store in EC2. All the JSON documents are stored and retrieved from this database.

Let’s get started!

Create IAM Role

IAM roles will have policies and trust relationships that will allow this role to be used in API Gateway and execute Lambda function.

Let’s create a new IAM role:

--assume-role-policy-document defines the trust relationship policy document that grants an entity permission to assume the role. trust.json is at github.com/arun-gupta/serverless/blob/master/aws/microservice/trust.json and looks like:

This trust relationship allows Lambda functions and API Gateway to assume this role during execution.

Associate policies with this role as:

policy.json is at github.com/arun-gupta/serverless/blob/master/aws/microservice/policy.json and looks like:

This generous policy allows any permissions over logs generated in CloudWatch for all resources. In addition it allows all Lambda and API Gateway permissions to all resources. In general, only required policy would be given to specific resources.

Create Lambda Functions

Detailed steps to create Lambda functions are explained in Serverless FaaS with AWS Lambda and Java. Let’s create the two Lambda functions as required in our case:

Couple of key items to note in this function are:

  • IAM role microserviceRole created in previous step is explicitly specified here
  • Handler is org.sample.serverless.aws.couchbase.BucketGetAll class. This class queries the Couchbase database defined using the COUCHBASE_HOST environment variable.

Create the second Lambda function:

The handler for this function is org.sample.serverless.aws.couchbase.BucketPost class. This class creates a new JSON document in the Couchbase database identified by COUCHBASE_HOST environment variable.

The complete source code for these classes is at github.com/arun-gupta/serverless/tree/master/aws/microservice/microservice-http-endpoint.

API Gateway Resource

Create an API using Amazon API Gateway and Test It and Build an API to Expose a Lambda Function provide detailed steps and explanation on how to use API Gateway and Lambda Functions to build powerful backend systems. This blog will do a quick run down of the steps in case you want to cut the chase.

Let’s create API Gateway resources.

  1. The first step is to create an API:
    This shows the output as:
    The value of id attribute is API ID. In our case, this is lb2qgujjif.
  2. Find ROOT ID of the created API as this is required for the next AWS CLI invocation:
    This shows the output:
    Value of id attribute is ROOT ID. This is also the PARENT ID for the top level resource.
  3. Create a resource
    This shows the output:
    Value of id attribute is RESOURCE ID.

API ID and RESOURCE ID are used for subsequent AWS CLI invocations.

API Gateway POST Method

Now that the resource is created, let’s create HTTP POST method on this resource.

  1. Create a POST method
    to see the response:
  2. Set Lambda function as destination of the POST method:
    Make sure to replace <act-id> with your AWS account id. API ID and RESOURCE ID from previous section are used here as well. --uri is used to specify the URI of integration input. The format of the URI is fixed. This CLI will show the result as:
  3. Set content-type of POST method response:
    to see the response:
  4. Set content-type of POST method integration response:
    to see the response:
  5. Deploy the API
    to see the response
  6. Grant permission to allow API Gateway to invoke Lambda Function:
    Also, grant permission to the deployed API:
  7. Test the API method:
    to see the response:
    Value of status attribute is 200 and indicates this was a successful invocation. Value of log attribute shows the log statement from CloudWatch Logs. Detailed logs can also be obtained using aws logs filter-log-events --log-group /aws/lambda/MicroservicePost.
  8. This command stores a single JSON document in Couchbase. This can be easily verified using the Couchbase CLI Tool cbq.Connect to the Couchbase server as:
    Create a primary index on default bucket as this is required to query the bucket with no clauses:
  9. Write a N1QL query to access the data:
    The results show the JSON document that was stored by our Lambda function.

API Gateway GET Method

Let’s create HTTP GET method on the resource:

  1. Create a GET method:
  2. Set correct Lambda function as destination of GET:
  3. Set content-type of GET method response:
  4. Set content-type of GET method integration response:
  5. Grant permission to allow API Gateway to invoke Lambda Function
  6. Grant permission to the deployed API:
  7. Test the method:
    to see the output:
    Once again, 200 status code shows a successful invocation. Detailed logs can be obtained using aws logs filter-log-events --log-group /aws/lambda/MicroservicePost.

This blog only shows one simple POST and GET methods. Other HTTP methods can be very easily included in this microservice as well.

API Gateway and Lambda References

  • Serverless Architectures
  • AWS API Gateway
  • Creating a simple Microservice using Lambda and API Gateway
  • Couchbase Server Docs
  • Couchbase Forums
  • Follow us at @couchbasedev

Source: blog.couchbase.com/2016/december/microservice-aws-api-gateway-lambda-couchbase

AWS IoT Button, Lambda and Couchbase

couchbase, techtip, , ,

Getting Started with Serverless FaaS and AWS Lambda shows how to use a simple Java function to store a JSON document to Couchbase using AWS Lambda. This blog builds upon that and shows how an AWS IoT Button can be used as a trigger for that Lambda function.

By end of this blog, you’ll learn:

  • How to configure AWS IoT Button
  • Use IoT Button as trigger for Lambda Function
  • Test IoT button

The overall flow will be:

serverless-iot-couchbase

Iot button click will invoke HelloCouchbaseLambda Lambda function. This function is uses Couchbase Java SDK to create a JSON document in Couchbase.

This blog is also playing catch up with Collecting iBeacon Data with Couchbase and Raspberry Pi IoT Devices by Nic and The CouchCase by Matthew on their summer projects. One last blog will be published in this series. That will show how multiple AWS IoT buttons can be used for some fun.

Let’s get started!

Configure IoT Button

The fastest way to configure IoT button  is using the mobile app for iOS or Android.

 

More details about configuring IoT Button using mobile app.

Here are some snapshots from configuring button using the mobile app.

Bring up the app, click on + to start configuring a new button:

aws-iot-button-configure-1

Enter button’s serial number:

aws-iot-button-configure-2

Register the button:

aws-iot-button-configure-3

Configure the button with wifi network:

aws-iot-button-configure-4

Upload all the certificates etc:

aws-iot-button-configure-5

After this, the button is configured and ready to use. This blog skipped the part where a template Lambda Function is associated with the button click.

If  mobile app cannot be used then the button can be configured manually.

Use IoT Button as Trigger for Lambda Function

The aws lambda create-event-source-mapping CLI allows to create an event source for Lambda function. As of AWS CLI version 1.11.21, only Amazon Kinesis stream or an Amazon DynamoDB stream can be used. But for this blog, we’ll use IoT button as a trigger. And this has to be configured using AWS Lambda Console.

IoT Button is only supported in a limited number of regions. For example, it is not supported in the us-west-1 region but us-west-2 region works.

The list of regions not supported are greyed out in the following list:

aws-iot-buttons-supported-region

Lambda Function can be triggered by several events. Lambda Function is invoked when any of these events occur. By default, no triggers are associated with a Lambda Function. For our HelloCouchbaseLambda function, these can be seen at us-west-2.console.aws.amazon.com/lambda/home?region=us-west-2#/functions/HelloCouchbaseLambda?tab=triggers.

AWS Lambda Default Triggers

Click on Add trigger to add a new trigger:

AWS Lambda Add Trigger

Select on the empty square to create a new trigger, and select AWS IoT:

AWS Lambda Add IoT Trigger

For the button previously registered, get the serial number from us-west-2.console.aws.amazon.com/iotv2/home?region=us-west-2#/thinghub:

aws-iot-things-hub

Specify the serial number of the button in the AWS IoT trigger:

aws-iot-add-trigger

Click on Submit to create the trigger:

aws-iot-added-trigger

And this confirms that the trigger has been added.

Test IoT Button

Before testing the button, let’s login to the Couchbase instance and verify the number of JSON documents in the bucket:

aws-iot-button-couchbase-console-default

This can be verified at http://<EC2-IP-Address>:8091/index.html#sec=buckets. As expected, no documents exists in the bucket.

Press the button once, and refresh the page. It shows that one document is now stored in the bucket. This is verified in the Couchbase Web Console:

aws-iot-button-couchbase-console-one-document

Click on Documents to see the complete list of documents:

aws-iot-button-couchbase-one-document-2

Click on the document ID to see more details about the document:

aws-iot-button-couchbase-one-document-details

Only timestamp is stored in this JSON document.

Now, let’s update HelloCouchbaseLambda code to include request id in the document as well. This can be achieved by adding the following line of code in the Java class:

A new deployment package can be built and uploaded using the following command:

Now clicking the button will update the number of documents. But the updated document will have an additional attribute populated as shown:

aws-iot-button-couchbase-second-document-details

How are you going to take AWS IoT button and use it with Lambda and Couchbase? Let us know at Couchbase Forums.

References

  • AWS IoT Button
  • AWS IoT Button Developer Guide
  • Couchbase Server Docs
  • Couchbase Forums
  • Follow us at @couchbasedev

Source: https://blog.couchbase.com/2016/december/aws-iot-button-lambda-couchbase

Serverless FaaS with AWS Lambda and Java

couchbase, techtip, ,

What is Serverless Architecture?

Serverless architecture runs custom code in ephemeral containers that are fully managed by a 3rd party. The custom code is typically a small part of a complete application. It is also called as function. This gives another name for serverless architecture as Function as a Service (FaaS). The container is ephemeral because it may only last for one invocation. The container may be reused but that’s not something you can rely upon. As a developer, you upload the code to FaaS platform, the service then handles all the capacity, scaling, patching and administration of the infrastructure to run your code.

An application built using Serverless Architecture follows the event-driven approach. For example, an activity happened in the application such as a click. This is

This is very different from a classical architecture where the application code is typically deployed in an application server such as Tomcat or WildFly. Scaling your application means starting additional instances of the application server or spinning up additional containers with the packaged application server. The Load Balancer need to be updated with the new IP addresses. Operating system need to be patched, upgraded and maintained.

Serverless Architectures explain the difference between the classical programming model and this new serverless architecture.

FaaS platform takes your application is divided into multiple functions. Each function is deployed in FaaS. The service spins up additional compute instances to meet the scalability demands of your application. FaaS platform provides the execution environment and takes care of starting and tearing down the containers to run your function.

Read Serverless Architectures for more details about these images.

One of the big advantages of FaaS is that you are only charged for the compute time, i.e. the time your code is running. There is no charge when your code is not running.

Another way to look at how Functions are different from VMs and Containers:

vm-containers-serverless

Note that Linux containers instead of Docker containers are used as an implementation for AWS Lambda.

How is FaaS different from PaaS?

As quoted at Serverless Architectures, a quick answer is provided by the following tweet:

In other words most PaaS applications are not geared towards bringing entire applications up and down for every request, whereas FaaS platforms do exactly this.

Abstracting the Back-end with FaaS explain the difference with different *aaS offerings. The image from the blog is captured below:

faas

Serverless Architectures also provide great details about what FaaS is and is not.

AWS Lambda, Google Cloud Functions and Azure Functions are some of the options for running serverless applications.

This blog will show how to write your first AWS Lambda function.

What is AWS Lambda?

AWS Lambda is FaaS service from Amazon Web Services. It runs your code on a high-availability compute infrastructure and performs all of the administration of the compute resources, including server and operating system maintenance, capacity provisioning and automatic scaling, code monitoring and logging.

AWS Lambda charges you for the duration your code runs in increments of 100ms. There is no cost associated with storing the Lambda function in AWS. First million requests per month are free and the pricing after that is nominal. Read more details on Lambda pricing. It also provides visibility into performance by providing real time metrics and logs to AWS CloudWatch. All you need to do is write the code!

Here is a quick introduction:

Also check out What’s New in AWS Lambda from AWS ReInvent 2016:

Also checkout Serverless Architectural Patterns and Best Practices from AWS ReInvent 2016:

The code you run on AWS Lambda is called a Lambda Function. You upload your code as a zip file or design it using the AWS Lambda Management Console. There is a built-in support for AWS SDK and this simplifies the ability to call other AWS services.

In short, Lambda is scalable, serverless, compute in the cloud.

AWS Lambda provides several execution environments:

  • Node.js – v0.10.36, v4.3.2 (recommended)
  • Java – Java 8
  • Python – Python 2.7
  • .NET Core – .NET Core 1.0.1 (C#)

This blog will show:

  • Build a Java application that stores a JSON document to Couchbase
  • Use Maven to create a deployment package for Java application
  • Create a Lambda Function
  • Update the Lambda Function

The complete code in this blog is available at github.com/arun-gupta/serverless/tree/master/aws/hellocouchbase.

Java Application for AWS Lambda

First, lets look at a Java application that will be used for this Lambda function. Programming Model for Lambda Functions in Java provide more details about how to write your Lambda function code in Java.

Our Lambda function will implemented the pre-defined interface com.amazonaws.services.lambda.runtime.RequestHandler. The code looks like:

handleRequest method is where the function code is implemented. Context provides useful information about Lambda execution environment. Some of the information from the context is stored a JSON document. Finally, Couchbase Java SDK API upsert is used to write a JSON document to the identified Couchbase instance. Couchbase on Amazon EC2 provide complete instructions to install Couchbase on AWS EC2.

Information about the Couchbase server is obtained as:

This is once again using Couchbase Java API CouchbaseCluster as a main entry point to the Couchbase cluster. The COUCHBASE_HOST environment variable is passed when the Lambda function is created. In our case, this would point to a single node Couchbase cluster running on AWS EC2. Environment variables were recently introduced in AWS Lambda.

Finally, you need to access bucket in the server:

The bucket name is serverless and all JSON documents are stored in this.

A simple Hello World application may be used for creating this function as well.

Create AWS Lambda Deployment Package

AWS Lambda function needs a deployment package. This package is either a .zip or .jar file that contains all the dependencies of the function. Our application is packaged using Maven, and so we’ll use a Maven plugin to create a deployment package.

The application has pom.xml with the following plugin fragment:

More details about Maven configuration are available in Creating a .jar Deployment Package Using Maven without any IDE. The maven-shade-plugin allows to create an uber-jar including all the dependencies. The shade goal is tied to the package phase. So the mvn package command will generate a single deployment jar.

Package the application using mvn package command. This will show the output:

The target/hello-couchbase-1.0-SNAPSHOT.jar is the shaded jar that will be deployed to AWS Lambda.

More details about creating a deployment package are at Creating a Deployment Package.

Create AWS Lambda Function

Create AWS Lambda Function using AWS CLI. The CLI command in this case looks like:

In this CLI:

  • create-function creates a Lambda function
  • --function-name provides the function name. The function name is case sensitive.
  • --role specifies Amazon Resource Name (ARN) of an IAM role that Lambda assume when it executes your function to access any other AWS resources. If you’ve executed a Lambda function using AWS Console then this role is created for you.
  • --zip-file points to the deployment package that was created in previous step. fileb is an AWS CLI specific protocol to indicate that the content uploaded is binary.
  • --handler is the Java class that is called to begin execution of the function
  • --publish request AWS Lambda to create the Lambda function and publish a version as an atomic operation. Otherwise multiple versions may be created and may be published at a later point.

Lambda Console shows:

servleress-couchbase-lambda-function

Test AWS Lambda Function

Test the AWS Lambda Function using AWS CLI.

It shows the output as:

The output from the command is stored in hellocouchbase.out and looks like:

Invoking this function stores a JSON document in Couchbase. Documents stored in Couchbase can be seen using Couchbase Web Console. The password is Administrator and password is the EC2 instance id.

All data buckets in this Couchbase instance are shown below:

serverless-couchbase-bucket-overview

Note that the serverless bucket is manually created.

Clicking on Documents shows details of different documents stored in the bucket:

serverless-couchbase-bucket-documents

Clicking on each document shows more details about the JSON document:

serverless-couchbase-bucket-document

Lambda function can also be tested using the Console:

serverless-couchbase-console-test

Update AWS Lambda Function

If the application logic changes then a new deployment package needs to be uploaded for the Lambda function. In this case, mvn package will create a deployment package and aws lambda CLI command is used to update the function code:

Shows the result:

The function can then be invoked again.

During writing of this blog, this was often used to debug the function as well. This is because Lambda functions do not have any state or box associated with them. And so you cannot log in to a box to check out if the function did not deploy correctly. You can certainly use CloudWatch log statements once the function is working.

AWS Lambda References

  • Serverless Architectures
  • AWS Lambda: How it works
  • Couchbase Server Docs
  • Couchbase Forums
  • Follow us at @couchbasedev

Source: https://blog.couchbase.com/2016/december/serverless-faas-aws-lambda-java