Tech Tip #25 showed how to configure WildFly on Raspberry Pi. This tech tip will show how to setup a WildFly managed domain over two hosts running on Raspberry Pi.

Lets understand some basic concepts first.

WildFly can run in two modes:

• Managed Domain allows you to run and manage a multi-server domain topology
• Standalone allows to run a single instance of server

Multiple standalone instances can be configured to form a highly available cluster. It is up to the user to coordinate management across multiple servers though.

Servers running in managed domain mode are referred to as the members of a “domain”. A single Domain Controller acts as the central management control point for the domain. A domain can span multiple physical or virtual hosts, with all WildFly instances on a given host under the control of a Host Controller process. One Host Controller instance is configured to act as the Domain Controller. The Host Controller on each host interacts with the Domain Controller to control the lifecycle of the application server instances running on its host and to assist the Domain Controller in managing them.

It is important to understand that administration of servers (standalone or managed domain) is orthogonal to clustering and high availability. In the managed domain mode, a server instance always belong to a Server Group. A domain can have multiple Server Groups. Each group can be configured with different profiles and deployments.

For example, default WildFly installation comes with “main-server-group” and “other-server-group” as shown:

<servers>
  <server name="server-one" group="main-server-group" auto-start="false"/>
  <server name="server-two" group="main-server-group" auto-start="false">
    <socket-bindings port-offset="150"/>
  </server>
  <server name="server-three" group="other-server-group" auto-start="true">
    <socket-bindings port-offset="250"/>
  </server>
</servers>

“main-server-group” has two servers: “server-one” and “server-two”. “other-server-group” has one server: “server-three”.

By default, these are all configured on a single machine (localhost) with Host Controller and Domain Controller co-located. This is defined in “domain/configuration/host.xml”.

Each Server Group is configured with a profile. These profiles are defined, and associated with a Server Group, in “domain/configuration/domain.xml”. Default WildFly installation comes with two profiles: “full” and “full-ha”.

<server-groups>
  <server-group name="main-server-group" profile="full">
    . . .
  </server-group>
  <server-group name="other-server-group" profile="full-ha">
    . . .
  </server-group>
</server-groups>

“main-server-group” is configured with “full” profile and “other-server-group” is configured with “full-ha” profile.

Profile is a named set of subsystem configurations that adds capabilities like Servlet, EJB, JPA, JTA, etc. The “full-ha” profile also enable all subsystems needed to establish cluster (infinispan, jcluster, and mod_cluster).

OK, enough explanation, lets get into action!

Make sure to install WildFly on each of the Raspberry Pi following Tech Tip #25.

docs.jboss.org/author/display/WFLY8/WildFly+8+Cluster+Howto explain in detail on how to setup Domain Controller and Host Controller on two hosts and enable clustering. This tech tip follows these instructions and adapt them for WildFly. This tech tip will show how to configure WildFly managed domain over two Raspberry Pis.

A subsequent blog will show how to configure cluster over this managed domain.

Here is what we’ll do:

• Call one host as “master” and another as “slave”.
• Both will run WildFly 8.1 CR2, master will run as Domain Controller and slave will run under the domain management of master
• Deploy a project into domain, and verify that the application is deployed on both master and slave hosts.

Each WildFly was connected to a display to obtain the IP address and enable SSH access. It can be enabled by invoking

sudo raspi-config

Scroll down to the SSH option and enable it. Then the two Raspberry Pis were configured to run in headless mode (no keyboard, mouse, or monitor).

1. Domain Configuration – Configure master and slave “host.xml”
   1. Master configuration
      1. Login to the master Raspberry pi:

         ~> ssh 10.0.0.27 -l pi
         [email protected]'s password: raspberry
         
         Linux raspberrypi 3.10.25+ #622 PREEMPT Fri Jan 3 18:41:00 GMT 2014 armv6l
         
         The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright.
         
         Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law.
         Last login: Wed May 28 19:03:19 2014
         pi@raspberrypi ~ $

         The default password is “raspberry”.

      2. By default, WildFly is configured to run in server VM. As mentioned in Tech Tip #25, this is not support on JDK bundled on Raspbian. This needs to be updated at two places. Edit “bin/domain.sh”:

         80 # If -server not set in JAVA_OPTS, set it, if supported
         81 SERVER_SET=`echo $JAVA_OPTS | $GREP "\-server"`
         82 if [ "x$SERVER_SET" = "x" ]; then
         83
         84     # Check for SUN(tm) JVM w/ HotSpot support
         85     if [ "x$HAS_HOTSPOT" = "x" ]; then
         86         HAS_HOTSPOT=`"$JAVA" $JVM_OPTVERSION -version 2>&1 | $GREP -i HotSpot`
         87     fi
         88
         89     # Check for OpenJDK JVM w/server support
         90     if [ "x$HAS_OPENJDK" = "x" ]; then
         91         HAS_OPENJDK=`"$JAVA" $JVM_OPTVERSION 2>&1 | $GREP -i OpenJDK`
         92     fi
         93
         94     # Check for IBM JVM w/server support
         95     if [ "x$HAS_IBM" = "x" ]; then
         96         HAS_IBM=`"$JAVA" $JVM_OPTVERSION 2>&1 | $GREP -i "IBM J9"`
         97     fi
         98
         99     # Enable -server if we have Hotspot or OpenJDK, unless we can't
         100     if [ "x$HAS_HOTSPOT" != "x" -o "x$HAS_OPENJDK" != "x" -o "x$HAS_IBM" != "x" ]; then
         101         # MacOS does not support -server flag
         102         if [ "$darwin" != "true" ]; then
         103             PROCESS_CONTROLLER_JAVA_OPTS="-server $PROCESS_CONTROLLER_JAVA_OPTS"
         104             HOST_CONTROLLER_JAVA_OPTS="-server $HOST_CONTROLLER_JAVA_OPTS"
         105             JVM_OPTVERSION="-server $JVM_OPTVERSION"
         106         fi
         107     fi
         108 else
         109     JVM_OPTVERSION="-server $JVM_OPTVERSION"
         110 fi

         and remove the lines marked 80 through 110.

      3. Edit host.xml

         vi domain/configuration/host.xml

         Remove “-server” option by removing lines 75 through 77 as shown below:

         71     <jvms>
         72         <jvm name="default">
         73             <heap size="64m" max-size="256m"/>
         74             <permgen size="256m" max-size="256m"/>
         75             <jvm-options>
         76                 <option value="-server"/>
         77             </jvm-options>
         78         </jvm>
         79     </jvms>

          

      4. The default setting for <interfaces> in this file looks like:

         <interfaces>
           <interface name="management">
             <inet-address value="${jboss.bind.address.management:127.0.0.1}"/>
           </interface>
           <interface name="public">
             <inet-address value="${jboss.bind.address:127.0.0.1}"/>
           </interface>
           <interface name="unsecure">
             <inet-address value="${jboss.bind.address.unsecure:127.0.0.1}"/>
           </interface>
         </interfaces>

         Change this to:

         <interfaces>
           <interface name="management">
             <inet-address value="${jboss.bind.address.management:10.0.0.27}"/>
           </interface>
           <interface name="public">
             <inet-address value="${jboss.bind.address:10.0.0.27}"/>
           </interface>
           <interface name="unsecure">
             <inet-address value="${jboss.bind.address.unsecure:10.0.0.27}"/>
           </interface>
         </interfaces>

         10.0.0.27 is master’s IP address.

   2. Slave configuration
      1. Login to the slave Raspberry pi:

         ~> ssh 10.0.0.28 -l pi
         [email protected]'s password: raspberry
         
         Linux raspberrypi 3.10.25+ #622 PREEMPT Fri Jan 3 18:41:00 GMT 2014 armv6l
         
         The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright.
         
         Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law.
         Last login: Tue May 27 21:28:02 2014 from 10.0.0.8
         pi@raspberrypi ~ $

         The default password is “raspberry”.

      2. Edit “domain.sh” and remove the lines as done for server. This will ensure that non-server, or client, JVM is used for running the domain.
      3. Edit “host.xml”

         vi domain/configuration/host.xml

      4. Set the host name by changing

         <host name="master" xmlns="urn:jboss:domain:2.1">

         to

         <host name="slave" xmlns="urn:jboss:domain:2.1">

      5. Remove “-server” option by removing lines 75 through 77 as explained above.
      6. Modify <domain-controller> such that slave can connect to master’s management port. Change

         <domain-controller>
         <local/>
         <!-- Alternative remote domain controller configuration with a host and port -->
         <!-- <remote host="${jboss.domain.master.address}" port="${jboss.domain.master.port:9999}" security-realm="ManagementRealm"/> -->
         </domain-controller>

         to

         <domain-controller>
         <local/>
         <!-- Alternative remote domain controller configuration with a host and port -->
         <remote host="10.0.0.27" port="9999" security-realm="ManagementRealm"/>
         </domain-controller>

         10.0.0.27 is master’s IP address.

      7. Change the default <interfaces> from:

         <interfaces>
           <interface name="management">
             <inet-address value="${jboss.bind.address.management:127.0.0.1}"/>
           </interface>
           <interface name="public">
             <inet-address value="${jboss.bind.address:127.0.0.1}"/>
           </interface>
           <interface name="unsecure">
             <inet-address value="${jboss.bind.address.unsecure:127.0.0.1}"/>
           </interface>
         </interfaces>

         to:

         <interfaces>
           <interface name="management">
             <inet-address value="${jboss.bind.address.management:10.0.0.28}"/>
           </interface>
           <interface name="public">
             <inet-address value="${jboss.bind.address:10.0.0.28}"/>
           </interface>
           <interface name="unsecure">
             <inet-address value="${jboss.bind.address.unsecure:10.0.0.28}"/>
           </interface>
         </interfaces>

         10.0.0.28 is slave’s IP address.

2. Security Configuration
   1. Master
      1. Using “add-user.sh” script, create a user in Management Realm for master.

         pi@raspberrypi ~/wildfly-8.1.0.CR2 $ ./bin/add-user.sh
         
         What type of user do you wish to add?
         a) Management User (mgmt-users.properties)
         b) Application User (application-users.properties)
         (a):  <ENTER>
         
         Enter the details of the new user to add.
         Using realm 'ManagementRealm' as discovered from the existing property files.
         Username : master
         Password recommendations are listed below. To modify these restrictions edit the add-user.properties configuration file.
         - The password should not be one of the following restricted values {root, admin, administrator}
         - The password should contain at least 8 characters, 1 alphabetic character(s), 1 digit(s), 1 non-alphanumeric symbol(s)
         - The password should be different from the username
         Password : master
         JBAS015269: Password must have at least 8 characters!
         Are you sure you want to use the password entered yes/no? yes
         Re-enter Password : master
         What groups do you want this user to belong to? (Please enter a comma separated list, or leave blank for none)[  ]:
         About to add user 'master' for realm 'ManagementRealm'
         Is this correct yes/no? yes
         Added user 'master' to file '/home/pi/wildfly-8.1.0.CR2/standalone/configuration/mgmt-users.properties'
         Added user 'master' to file '/home/pi/wildfly-8.1.0.CR2/domain/configuration/mgmt-users.properties'
         Added user 'master' with groups  to file '/home/pi/wildfly-8.1.0.CR2/standalone/configuration/mgmt-groups.properties'
         Added user 'master' with groups  to file '/home/pi/wildfly-8.1.0.CR2/domain/configuration/mgmt-groups.properties'
         Is this new user going to be used for one AS process to connect to another AS process? e.g. for a slave host controller connecting to the master or for a Remoting connection for server to server EJB calls. yes/no? no

      2. Using “add-user.sh” script, create a user in Management Realm for slave. The username must be equal to the name given in the slave’s <host> element in “host.xml”.

         pi@raspberrypi ~/wildfly-8.1.0.CR2 $ ./bin/add-user.sh
         
         What type of user do you wish to add?
         a) Management User (mgmt-users.properties)
         b) Application User (application-users.properties)
         (a):  <ENTER>
         
         Enter the details of the new user to add.
         Using realm 'ManagementRealm' as discovered from the existing property files.
         Username : slave
         Password recommendations are listed below. To modify these restrictions edit the add-user.properties configuration file.
         - The password should not be one of the following restricted values {root, admin, administrator}
         - The password should contain at least 8 characters, 1 alphabetic character(s), 1 digit(s), 1 non-alphanumeric symbol(s)
         - The password should be different from the username
         Password : slave
         JBAS015269: Password must have at least 8 characters!
         Are you sure you want to use the password entered yes/no? yes
         Re-enter Password : slave
         What groups do you want this user to belong to? (Please enter a comma separated list, or leave blank for none)[  ]:
         About to add user 'master' for realm 'ManagementRealm'
         Is this correct yes/no? yes
         Added user 'master' to file '/home/pi/wildfly-8.1.0.CR2/standalone/configuration/mgmt-users.properties'
         Added user 'master' to file '/home/pi/wildfly-8.1.0.CR2/domain/configuration/mgmt-users.properties'
         Added user 'master' with groups  to file '/home/pi/wildfly-8.1.0.CR2/standalone/configuration/mgmt-groups.properties'
         Added user 'master' with groups  to file '/home/pi/wildfly-8.1.0.CR2/domain/configuration/mgmt-groups.properties'
         Is this new user going to be used for one AS process to connect to another AS process? e.g. for a slave host controller connecting to the master or for a Remoting connection for server to server EJB calls. yes/no? yes
         To represent the user add the following to the server-identities definition <secret value="c2xhdmU=" />

         The answer to the last question is “yes” as this slave will be connecting to master.

         Note that output’s last line contains a <secret …> element. Copy this string as this will need to be used in slave’s “host.xml”.

   2. Slave
      1. Configure “domain/configuration/host.xml” for authentication by changing the security-realms element from:

         <management>
           <security-realms>
             <security-realm name="ManagementRealm">
             <authentication>
               <local default-user="$local" skip-group-loading="true" />
               <properties path="mgmt-users.properties" relative-to="jboss.domain.config.dir"/>
             </authentication>

         to

         <management>
           <security-realms>
             <security-realm name="ManagementRealm">
                    <server-identities>
                       <secret value="c2xhdmU=" />
                    </server-identities>
              <authentication>
               <local default-user="$local" skip-group-loading="true" />
               <properties path="mgmt-users.properties" relative-to="jboss.domain.config.dir"/>
              </authentication>

         The <secret …> element added here is obtained when “add-user.sh” is invoked for slave in the previous step. Slave’s host name is “slave”, master has a username by “slave”, and adding <secret …> element (base64 code for the password) allows the slave to connect to master.

3. Change the default username/password for HornetQ otherwise it will throw a pesky warning in the server log as:

   [Server:server-three] 19:56:54,746 ERROR [org.hornetq.core.server]
   (Thread-1 (HornetQ-client-netty-threads-6928453)) HQ224058: Stopping
   ClusterManager. As it failed to authenticate with the cluster:
   HQ119099: Unable to authenticate cluster user:
   HORNETQ.CLUSTER.ADMIN.USER
   [Server:server-three] 19:56:54,982 INFO  [org.hornetq.core.server]
   (Thread-6 (HornetQ-server-HornetQServerImpl::serverUUID=2b53cedf-e6a2-11e3-9d57-71d44db5d6ff-27664265))
   HQ221029: stopped bridge
   sf.my-cluster.5e79c85f-e69d-11e3-a06c-37b4b1e261b8
   [Server:server-three] 19:56:55,833 ERROR [org.hornetq.core.server]
   (default I/O-1) HQ224018: Failed to create session:
   HornetQClusterSecurityException[errorType=CLUSTER_SECURITY_EXCEPTION
   message=HQ119099: Unable to authenticate cluster user:
   HORNETQ.CLUSTER.ADMIN.USER]

   Edit “domain/configuration/domain.xml” in master and change:

   <subsystem xmlns="urn:jboss:domain:messaging:2.0">
   <hornetq-server>
   <cluster-password>${jboss.messaging.cluster.password:CHANGE ME!!}</cluster-password>

   to

   <subsystem xmlns="urn:jboss:domain:messaging:2.0">
   <hornetq-server>
   <cluster-password>newClusterPassword</cluster-password>

   Make this change in “domain/configuration/domain.xml” for slave as well.

4. Disable the firewall on master and slave by giving the following command on each host.

   sudo iptables --flush

5. Start the master first as:

   ./wildfly-8.1.0.CR2/bin/domain.sh

   The last message in the log will look like:

   [Server:server-three] 17:32:01,641 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015874: WildFly 8.1.0.CR2 "Kenny" started in 137565ms - Started 321 of 430 services (186 services are lazy, passive or on-demand)

   After the master has completely started, start the slave as:

   ./wildfly-8.1.0.CR2/bin/domain.sh

   The last message in the log will look like:

   [Server:server-three] 17:35:43,947 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015874: WildFly 8.1.0.CR2 "Kenny" started in 127576ms - Started 322 of 431 services (187 services are lazy, passive or on-demand)

   The server log for master will also show a message indicating that the slave is now a registered host with master:

   [Host Controller] 17:33:31,545 INFO  [org.jboss.as.domain] (Host Controller Service Threads - 56) JBAS010918: Registered remote slave host "slave", WildFly 8.1.0.CR2 "Kenny"

6. Deploy the application
   1. Check out a simple application that puts/gets some HTTP session data.

      git clone https://github.com/arun-gupta/wildfly-samples.git

      This workspace has other samples related to WildFly. But the one that we care about it is in “clustering/http” directory. Change to that directory and build the sample as:

      cd clustering/http
      mvn package

      This will generate a WAR file in the “target” directory.

   2. “jboss-cli” is a Command Line management tool for standalone and managed domains. It is bundled as a script in the “bin” directory. It can connect to remote servers as well. Now that we’ve Domain Controller running on master and Host Controller running on slave, lets connect it from your local machine as:

      ~/tools/wildfly-8.1.0.CR2/bin/jboss-cli.sh -c --controller=10.0.0.27:9990
      Authenticating against security realm: ManagementRealm
      Username: master
      Password: master
      [[email protected]:9990 /]

      10.0.0.27 is IP address of the master. The generated WAR file can now be deployed as:

      [[email protected]:9990 /] deploy http-1.0-SNAPSHOT.war --server-groups=other-server-group

      This shows the following output in master’s log:

      [Host Controller] 20:20:27,058 INFO  [org.jboss.as.repository] (management-handler-thread - 6) JBAS014900: Content added at location /home/pi/wildfly-8.1.0.CR2/domain/data/content/c4/f73f651f03c68d3a7b29686519e6142bccdfa4/content
      [Server:server-three] 20:20:31,497 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-1) JBAS015876: Starting deployment of "http-1.0-SNAPSHOT.war" (runtime-name: "http-1.0-SNAPSHOT.war")
      
      . . .
      
      [Server:server-three] 20:20:38,351 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 71) JBAS010281: Started dist cache from web container
      [Server:server-three] 20:20:38,361 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 70) JBAS010281: Started default-host/http-1.0-SNAPSHOT cache from web container
      [Server:server-three] 20:20:38,599 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-1) JBAS017534: Registered web context: /http-1.0-SNAPSHOT
      [Server:server-three] 20:20:42,474 INFO  [org.jboss.as.server] (ServerService Thread Pool -- 68) JBAS018559: Deployed "http-1.0-SNAPSHOT.war" (runtime-name : "http-1.0-SNAPSHOT.war")

      And slave’s log shows:

      [Server:server-three] 20:20:31,475 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-2) JBAS015876: Starting deployment of "http-1.0-SNAPSHOT.war" (runtime-name: "http-1.0-SNAPSHOT.war")
      [Server:server-three] 20:20:35,690 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 70) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.
      
      . . .
      
      [Server:server-three] 20:20:41,028 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 71) JBAS010281: Started default-host/http-1.0-SNAPSHOT cache from web container
      [Server:server-three] 20:20:41,214 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-1) JBAS017534: Registered web context: /http-1.0-SNAPSHOT
      [Server:server-three] 20:20:42,472 INFO  [org.jboss.as.server] (ServerService Thread Pool -- 69) JBAS018559: Deployed "http-1.0-SNAPSHOT.war" (runtime-name : "http-1.0-SNAPSHOT.war"

Accessing the application on master (http://10.0.0.27:8330/http-1.0-SNAPSHOT/index.jsp) shows:

Accessing the application on slave (http://10.0.0.28:8330/http-1.0-SNAPSHOT/index.jsp) shows:

So we could easily deploy a Java EE application to multiple WildFly instances, running on RaspberryPi, in managed domain mode, with a single command. How cool ?

Next blog will explain how to setup cluster on these multiple instances.