Tuesday, October 29, 2013

Creating Multiple Server Instances on JBoss 7

Problem Space

If you are a typical Java Developer, chances are that you are highly likely to be working on more than one project at a time. Even if you are lucky enough to be working on only one project, you may find situations where you need to work on 2 different versions of the code at the same time (Fixing current Production issues and developing new functionality).

If you are in a support role, you should have testing environments with each hosting multiple versions of the code(DEV, SIT, QA or Production) or even multiple projects independent of each other.

The point that I am trying to make is that; in almost every situation, there is a need for more than one Application Server instance on a machine. You can get away with a single instance but then you would spend an enormous amount of time reconfiguring your server instance when developing or testing code.

You could also get away with "cloning" the complete product in multiple directories but this is very messy, amateur and tough to maintain.

Application Servers Today

Application servers are built to address the issues mentioned above. There are basically 2 mechanisms which are used:
  1. The application server is contained as a package. You can use a wizard or some script that will assist you in creating a new server instance. The newly created server instance can exist anywhere on the machine but it will reference libraries and scripts in the base package. 
  2. Other application servers are shipped as a package which contains a few sample server instances. You can create a new server instance by simply copying one the the sample server instances, renaming it and pasting it in the same directory or location as the other server instances. 
JBoss 7 falls in the second category mentioned above. The sample server instances in previous versions of JBoss are "Default", "Minimal" & "All". In JBoss 7 the sample instance is called "standalone".

In previous versions of JBoss, the sample server instances("Default", "Minimal" & "All") represented the JBoss services that you required. In JBoss 7 this is addressed by multiple configuration files(standalone-full-ha.xml, standalone-full.xml, standalone-ha.xml & standalone.xml) all located in the configuration folder of the server instance. This is a separate topic on its own but for now, this is enough information to get you going.

How to create multiple server instances on JBoss 7

Cloning(Copy & Paste) the Standalone directory is the quickest way to create a separate or new server instance. You would need to rename the newly created instance as the operating system will not allow 2 files or directories in the same location with the same name. The directory name will be the server instance name.

You can then use this command in the bin directory of JBoss to run your newly created server instance:

 ./standalone.sh -Djboss.server.base.dir=<Instance_Name>  

If you want JBoss to bind to your IP Address instead of localhost you would need to run the following command:

 ./standalone.sh -Djboss.server.base.dir=<Instance_Name> -Djboss.bind.address=<your_ip_address> -Djboss.bind.address.management=<your_ip_address>  

And one step further, if you required to run a specific configuration; lets say for example you wanted to use the JMS queues you would need to run the following command with Configuration_file being standalone-full.xml:

 ./standalone.sh -Djboss.server.base.dir=<Instance_Name> -Djboss.bind.address=<your_ip_address> -Djboss.bind.address.management=<your_ip_address> -c <Configuration_File>  

You would theoretically have 2 JBoss 7 server instances but they would not be able to run concurrently because of port conflicts(Both instances use the same ports).

To address the above issue we can change the port offset.

Locate this line in the configuration file:
Change the value of the port-offset to a positive number. All ports will be offset by this value. As an example, if we were to set this port-offset value to 10.

  • The http port would be 8090 instead of 8080
  • The ajp port would be 8019 instead of 8009
  • The https port would be 8453 instead of 8443

I trust that you understand this point. I think that the engineers at JBoss have done a great job using this mechanism. I remember me having to update ports all over the show in a previous version and it was a nightmare to maintain or maybe, I was just not well informed.

My Solution to a Robust Distribution

I suggest that you start off with a clean or new installation.

1.) Copy the standalone directory and clone it 8 times. This means that you should have your original standalone directory and 8 other standalone directories. Name them as follows: standalone_1, standalone_2, ..., standalone_8.

2.) Set the port offset in:
standalone_1 to 100
standalone_2 to 200
standalone_3 to 300
standalone_8 to 800

3.) Create a scripts directory inside the bin directory. Create start and stop scripts to each of the standalone instances. You would want to dynamically get the IP address instead of hard coding it.

A Windows batch script to start standalone instance 1 should look something like this:

1:  @echo off  
2:  for /f "tokens=1-2 delims=:" %%a in ('ipconfig^|find "IPv4"') do set ip=%%b  
3:  set address=%ip:~1%  
4:  set server=standalone_1  
6:  echo ****************************************  
7:  echo The server ip address is: %address%  
8:  echo The server instance is: %server%  
9:  echo ****************************************  
10:  cd ..  
11:  CALL standalone.bat -Djboss.server.base.dir=%server% -Djboss.bind.address=%address% -Djboss.bind.address.management=%address% -c standalone-full.xml  
12:  cd scripts  

A Unix bash script to start standalone instance 1 should look something like this:

1:  #!/bin/sh  
2:  address=`ifconfig | grep "inet " | grep -v | cut -f2 -d':' | cut -f1 -d' '`  
3:  server=standalone_1  
5:  echo "****************************************"  
6:  echo "The server ip address is: "$address  
7:  echo "The server instance is: "$server  
8:  echo "****************************************"  
9:  cd ..;  
10:  ./standalone.sh -Djboss.server.base.dir=$server -Djboss.bind.address=$address -Djboss.bind.address.management=$address -c standalone-full.xml &
12:  cd scripts  

The biggest issue that I had creating these scripts were that there was a different way of retrieving the IP address dynamically from almost all of the Operating Systems. Here are some scripts to retrieve the IP address from the OS:

Linux ifconfig Example
 ifconfig | grep 'inet addr:'| grep -v '' | cut -d: -f2 | awk '{ print $1}'  

FreeBSD/OpenBSD ifconfig Example
 ifconfig | grep -E 'inet.[0-9]' | grep -v '' | awk '{ print $2}'  

Sun / Oracle Solaris Unix Example
 ifconfig -a | grep inet | grep -v '' | awk '{ print $2}'  

MAC OSX Mountain Lion Example
 ifconfig | grep -E 'inet.[0-9]' | grep -v '' | grep -i 'broadcast' |awk '{ print $2}'  

4.) Create stop Scripts for each of the standalone instances. 

A Windows batch script to stop standalone instance 1 should look something like this:

1:  @echo off  
2:  for /f "tokens=1-2 delims=:" %%a in ('ipconfig^|find "IPv4"') do set ip=%%b  
3:  set address=%ip:~1%  
4:  set port=10099  
5:  echo ****************************************  
6:  echo The server ip address is: %address%  
7:  echo The server port number is: %port%  
8:  echo ****************************************  
9:  cd ..  
10:  CALL jboss-cli.bat --connect --controller=%address%:%port% --command=:shutdown  
11:  cd scripts  

A Unix bash script to stop standalone instance 1 should look something like this:

1:  #!/bin/sh  
2:  address=`ifconfig | grep "inet " | grep -v | cut -f2 -d':' | cut -f1 -d' '`  
3:  port=10099  
4:  echo "****************************************"  
5:  echo "The server ip address is: "$address  
6:  echo "The server port number is: "$port  
7:  echo "****************************************"  
8:  cd ..;  
9:  ./jboss-cli.sh --connect --controller=$address:$port --command=:shutdown  
10:  cd scrips  

5.) JBoss-as-7.1.1.Final is packaged with a bug on user creation. I addressed this by creating an administrative user on the standard standalone instance using the bin/add-user.sh script and then copying the mgmt-users.properties to the configuration folders of each of the newly created server instances. When you try to connect to the admin console and are prompted for the username and password, you can use the same credentials in every server instance.

6.) Do not modify or use the the original standalone server instance. This way you can easily extend your installation. I use the original standalone server instance for trouble shooting purposes.


I had this problem and worked in a team of at least 10 developers who would eventually run into the same issues. Instead of wasting time and money I created a company specific distribution.

Contact me and I can assist you with a custom distribution for your organisation.

Again, I would love to hear your questions or comments!

Wednesday, October 23, 2013

Jenkins Jobs to Deploy Artefact to Amazon Cloud Server hosting JBoss Instance

Problem background:

We develop IT solutions for clients. Mainly Java applications running on a JEE application server and connecting to a database. Development and support is done at Head office. The clients are based throughout the world. We have a Continuous Integration process hosted and running at Head office.

Our Continuous Integration Process contains the following:

  • GIT Repository
  • Jenkins Build Server
  • Maven Build Tool (Maven projects)
  • Nexus Artefact Repository

Our  clients host the production solutions on their own infrastructure. We configure the environments at project initiation time and then a typical deployment consists of only deploying a new java artefact(ear, war or jar).

We traditionally connect to our Client's server environments via a VPN connection but this time our client went for a Cloud Hosting Solution(Amazon).


My usual advice to my colleagues that are attempting to build any Jenkins job is, to first get the process working manually. If you are able to execute the process manually, then building a Jenkins job is very focused and you are able to break down your specific problem area. It is also a form of decoupling the problem.

When I tried to execute my process manually for this particular scenario, I have a few issues. I was provided with a security key file, a hostname and a username. I tried to use the : ssh -i <path_to_key> <user>@<Cloud_server> command but it kept prompting me for a password. I initially thought that it was looking for a passphrase for the security key but I was assured that there was no passphrase. The problem turned out to be that the security keys were in .ppk format. Someone converted the .PEM file to .ppk format so that it could be loaded into Putty(Telnet, ssh, ... ,keygen utility). My problem was that I use a MAC and use the native ssh client which requires a .PEM file and not a .ppk file.

I tried to convert the .ppk file back into .PEM with no success. It was easier asking the server administrator for the .PEM file.

The Solutions available:

Open Up Ports to the Cloud server 

The first option available is getting ports opened on the Cloud server to the internet. The advantage of this approach is that we can create our deployment job on Jenkins using the JBoss CLI utility to deploy the artefact onto the application server. I usually just create a "free-style software project", use the "wget" command to download the artefact from Nexus and run the JBoss command line utility to deploy the application. The other option is to use the Maven jboss-as plugin to do the deployment. This would work well if you are deploying a Snapshot version built from a fresh maven install. The only other snag that you might encounter is, to configure Proxy settings if connecting to the internet using a Proxy Server.

The disadvantages are far too many in my opinion. Our Internet connection is not that great. This means that the JBoss CLI utility can timeout. The next problem is that we need to secure the JBoss server. Remember now, everyone on the internet has access to the administration ports on JBoss, which is a huge security risk. One way of countering this issue, is to only open up access to these ports to specific public IP addresses. Not to forget, someone needs to keep account of which ports have been opened up and to which IP addresses are they accessible. Migrating your Jenkins Server to an alternative IP address can easily turn into a nightmare

The simplicity to this mechanism is that, you can build your Job in the same way as you would connect to any other server hosted in your own network.

SSH into the Server

The other option is to use the Publish over SSH Jenkins plugin.
Create a "free-style software project". Use the "wget" command to download the deployment artefact from Artefact Repository(Nexus) into the Job's workspace. Use the ssh plugin to copy over the artefact to the Cloud server. Once the artefact is on the Cloud server,  use the same plugin to execute a script hosted on the Cloud server. This script needs to either copy the artefact into the JBoss server's deployment directory or executes the JBoss CLI utility to do the deployment. The Jenkins Publish over SSH plugin is capable of both connecting via ssh and securely copying files or scp.

One disadvantage of this mechanism is that the script that is executed on the Cloud server should be able to return both positive and negative results. In this way the job can report successful or unsuccessful execution of the project. Another disadvantage is loading and maintaining the SSL keys.


I am pretty sure that if we had a perfect internet connection I would have had a much simpler life. Unfortunately this is IT, always have a plan and a backup plan!

This is my first Blog Posting and I would love to hear your opinions or questions.