Monday, July 30, 2012

Internal DNS resolution in a Windows Network for Ubuntu or OSX

It is common for developers to use Ubuntu or other Linux/Unix flavors in Corporate environments where Windows Servers perform different Network fundamental operations like Domain Name System (DNS) and Dynamic Host Configuration Protocol (DHCP) Services.

For some reason I keep on finding that DHCP Servers are not configured to pass the domain to be used for searches. If that is your case you should wake your administrator up so they properly configure their DHCP Server but while you wait if you are running OSX (you probably know this) just change Network Preferences "Search Domain" option. If you are running Ubuntu Desktop AFAICS you will not be able to force a "Search domain" using the "Network Settings" applet while using DHCP. You will have to use command line (despites the warnings about not editing /etc/resolv.conf this trick does work still in version 12.04:
#!/bin/bash -e
# change-search-domain.sh

domain=$1

USAGE="Usage: `basename $0` <domain>"

if [ $# -ne "1" ] 
then
 echo $USAGE
  exit 1 
fi

filePath=/etc/resolv.conf

sed "/search/d" $filePath  > tmp && echo >> tmp && echo "search $domain" >> tmp && mv tmp $filePath
service resolvconf restart
Of course this can be run locally, but also remotely through Remoto-IT.

Monday, July 23, 2012

Mediawiki AJAX autocomplete Search POB

Here a couple of POB Recipes you can run with Remoto-IT to upgrade your Mediawiki Search functionality so it uses AJAX to autocomplete for case insensitive results. The first is for installing:
#!/bin/bash -e
# mediawiki-autocomplete-search-install.sh

hostname=$1

USAGE="Usage: `basename $0` <mediawikiHome> <titleKeyUrl> <titleKeyFileName>"

if [ $# -ne "3" ] 
then
 echo $USAGE
  exit 1 
fi

mediawikiHome=$1
titleKeyUrl=$2
titleKeyFileName=$3

wget -O "$titleKeyFileName" $titleKeyUrl
tar -xzf $titleKeyFileName -C $mediawikiHome/extensions
The second is for configuring:
#!/bin/bash -e
# mediawiki-autocomplete-search-config.sh

hostname=$1

USAGE="Usage: `basename $0` <mediawikiHome>"

if [ $# -ne "1" ] 
then
 echo $USAGE
  exit 1 
fi

mediawikiHome=$1

sed -i '/$wgEnableMWSuggest/d' $mediawikiHome/LocalSettings.php && sed -i '/TitleKey/d' $mediawikiHome/LocalSettings.php && echo -e 'require_once( "$IP/extensions/TitleKey/TitleKey.php" );' >> $mediawikiHome/LocalSettings.php && echo '$wgEnableMWSuggest = true;' >> $mediawikiHome/LocalSettings.php
cd $mediawikiHome/maintenance/
php update.php 
cd $mediawikiHome/extensions/TitleKey/
php rebuildTitleKeys.php
And here is how to call it. For example suppose your tools.sample.com server has two mediawiki installations like samplewiki and just wiki. Then your host recipe script will look like:
#!/bin/bash -e
#tools.sample.com.sh

common/mediawiki/mediawiki-autocomplete-search-install.sh /var/www/samplewiki http://upload.wikimedia.org/ext-dist/TitleKey-MW1.17-81394.tar.gz TitleKey-MW1.17-81394.tar.gz 
common/mediawiki/mediawiki-autocomplete-search-config.sh /var/www/samplewiki
common/mediawiki/mediawiki-autocomplete-search-config.sh /var/www/wiki

Ubuntu: Run apt-get update before installing Sources

Errors like the below usually mean an outdated Ubuntu:
libmysqlclient-dev : Depends: libmysqlclient16 (= 5.1.49-1ubuntu8.1) but 5.1.58-1ubuntu1 is to be installed
Do always update first:
sudo apt-get update

Sunday, July 22, 2012

Hadoop for Business and Security Intelligence - BI and SI with the help of POB - A Hadoop Tutorial

To generate comprehensive near to real time reports for Business Intelligence (BI) and Security Intelligence (SI) you will need to collect data from different sources, mash it up and aggregate it. As the term suggests "Big Data" refers to a huge amount of data generated by several systems in a company. While that is the reality of most companies few actually analyze their big data (like server logs) which in turn carry valuable information that can be used to make decisions on the Business side and on the Security side.

Specially Security is such an important aspect for Business (so many times underestimated) that I prefer to give it a very special treatment.

Intrusion Detection is one of those areas where analyzing different logs from switches, routers, OS and applications could give a good non-false-positive alert by the time an intruder gains access to important resources.

Hadoop allows us to achieve this important goal (SI) the same way it allows us to increase our BI solutions portfolio. Specially Hadoop Streaming API allows your Plain Old Bash ( POB ) scripts to run now in a cluster analyzing not only a local log but logs from any appliance in your Data Center (DC).

Not only you can use bash scripts to define Map and Reduce functionality but you can use POB recipes to automate the installation and configuration of Hadoop Clusters.

Here is my attempt to bring Hadoop to System and Data Base Administrators, to Software Engineers and Developers, to Security experts and Executives. You need no java knowledge, just bash knowledge which I assume is mandatory for any Linux operator. Hadoop is officially supported in Linux for production systems and bash scripting is a fundamental skill to master Linux.

What is Hadoop?

Hadoop is a distributed computing platform that includes a distributed file system and a parallel programming model which allows to distribute the load of long time taking tasks through inexpensive nodes in a cluster. Both data storage and data processing are distributed so if you need to process more data or if you need to provide more complex processing you just need to add more cluster nodes. The same applies if you want to speed up the time taken to provide results. Distributed computing is of course not just about speed but also about fault tolerance and Hadoop addresses that issue as well. If one node goes down the results of computations will still be available.

What Hadoop is not?

Transactional systems are not well suited for hadoop however analytics systems are. Do not think about hadoop as a database but rather a Platform that allows to extract valuable information in a NoSQL way: Using Keys and Value Pairs Mapping plus Reduce functions (MapReduce Pattern)

Hadoop Topology

Social issues aside Hadoop uses a Master/Slave approach. As you figured the Master will command what the Slaves will do, it will coordinate and assign the job a Slave. If a Slave dies failing to execute some logic then Hadoop will re-assign the job to a different Slave node. A Master runs two daemons (NameNode and JobTracker) while a Slave runs another two (DataNode and TaskTracker).

You can start with just one machine that process your data (using standalone or pseudo distributed mode) and as data or computation complexity grows you can add more nodes to split the work. Of course just one machine will not provide fault tolerance/redundancy plus most likely the performance will be the same if not worst than just running scripts (bash+find+awk+grep+cut+sort+uniq for example) from command line so it would be impractical to say the least. On the other side if you are considering Hadoop it is because your current data processing is taking too much time and you want it to scale horizontally.

While a NameNode and a JobTracker service (Master) can run in the same machine you will find that as you need more nodes to process bigger data the two services will need to be separated and as you need at least two machines as DataNodes/TaskTrackers (Slaves) to have failover and redundancy then at a minimum Hadoop typical production installation will start with three or four separated machines: Each DataNode machine will also be a TaskTracker machine, this is to ensure the map or reduce jobs happen in the same local file system. Of course with only two nodes to store data and perform the jobs you end up with a replication factor of 2 (dfs.replication). A three way replication is more reliable and the defacto used by big players like Google in their GFS. So that will make your minimum four or five machines.

If the DN/TT machines are virtualized VMs it is recommended to have at least the first dfs.replication machines in different hosts. The same applies for storage, their disks should be in different hardware. In fact Hadoop works better with Just One Bunch Of Disks (JBOD) rather than RAID or SAN storage. Remember, Hadoop concept is to use commodity machines which means machines that are shipped with common available components (not necessarily unreliable but definitely cheaper than high end hardware) but if those are VMs in the same host you will probably end up having a limitation that you could avoid distributing in different physical machine.

VMWARE claims with their new open sourced project named serengeti to resolve the single point of failure problems inherent to Hadoop (read below) and at the same time claims ESXi-VM-deployed-Hadoop has performance which is comparable with that of native (physical) deployments. Local disks against traditional SAN will be needed in any case so the discussions on the topic will continue I guess for a while. One thing for sure is that the competition is furious and Hadoop Appliances, optimized servers and custom built rackmount servers configured as Hadoop appliances will be considered as well by CTOs.

Take a look at this image gallery for an idea of how big Hadoop clusters can be. Clearly configuration here plays an important role and tweaking without documenting would be a big mistake but if you can configure from POB with Remote-IT then you get both, the documentation and the automated configuration at the same time.

Backup considerations

Hadoop is so far a single point of failure system. If the NameNode goes down then you will need to try to restart it, create a new one from its latest image which will hopefully be still available in the crashed NameNode or will need to be pulled from the "Checkpoint Node" or from a "Backup Node.

Rebalancing maintenance tasks could be needed depending on how uniformly data is placed across the DataNode. Rack awareness is important as all DataNodes involved in a job should ideally be collocated in the same rack because of network traffic, however replicas should live in separated racks for proper redundancy, it is important then to tell hadoop which rack the node is. There might be safemode conditions that should be understood and manual action could be required. Tools like fsck for hadoop will need to be run to check for the health of the cluster and then manually take actions if required. Upgrade and rollback are important procedures that must be considered up front to avoid losing data. This means you will probably have 5 initial machines (or 6 if you decide to go with three-way replication (as I already mentioned)

Some details of Hadoop

Hadoop offers a programming paradigm to resolve tasks faster through a Distributed File System (HDFS) and MapReduce jobs.

HDFS uses NameNode (Master node which stores the files metadata) to find block locations (dir, mkdir) and Datanodes (Slaves nodes that store the real data blocks) to write and retrieve the data from the specific blocks (cat, get, put…).

MapReduce pattern allows any data processing problem to be divided in a mapping step where just a key and a value is pooled from the whole data set and a second step which later reduces the result (where data gets aggregated to provide a total value like an average through grouping for example). It uses a JobTracker (Master node in charge of scheduling, rescheduling and keeping track of all map reduce jobs running in all nodes of the cluster) and TaskTrackers (Slaves running the map reduce jobs and reporting back progress to the JobTracker).

An HDFS master node will run a NameNode daemon and will control slaves (Configured in $HADOOP_HOME/conf/slaves which hosts a line per slave node) running DataNode daemons. In addition a MapRed master node will run JobTracker daemon and will control slaves (configured in $HADOOP_HOME/conf/slaves as well) running TaskTracker daemons.

Some important concepts that support the work of HDFS and MapReduce are:
  1. Combiner: Optionally if you can provide some aggregation in the same node on a Map result a combiner allows to free network resources as less data gets transferred to the Reduce nodes. Clearly if you are just getting a total as a summation you can save time summing up within the Map step to save network resources later when applying a Reduce step.
  2. Distributed Data Flow Engine: Supporting the execution of the MapReduce algorithm across different machines through the use of input splits (dividing the input data in chunks)
  3. Partition: Reducing through different Reducers which are in charge of applying the reduce algorithm to a particular set of keys.
A typical Hadoop project needs two main different concerns: A person wearing the software engineer hat must write two functions (a Map and a Reduce Function) and another wearing the systems engineer hat must offer a production environment optimized for handling the totality of the data within certain time constraints.

Typically Hadoop nodes will have 4 to 12 drives per machine, Non-RAID configuration and the disks themselves will be commodity hardware, no need to use top-notch high performance disks.

A sample problem

Imagine you have to combine the authorization logs coming from different servers and respond two questions:
  1. List how many failed login attempts we have per user and type (wrong password, wrong username for example)
  2. What exact day, hour and minute we had the most bad-user-login requests (potential unknown-user brute force attack)
To solve this problem we will need to learn how to install hadoop, how to write map and reduce jobs, how to run a hadoop job and how to retrieve the results.

Installing Hadoop

The Hadoop documentation is pretty straightforward (Look always for the specific version of hadoop you are trying to install, in my case for example I read http://hadoop.apache.org/common/docs/r1.0.3/single_node_setup.html) We should try to automate the installation using POB recipes though.

I have shared a project to aid on automatically installing hadoop in standalone, pseudo-distributed and fully distributed modes. You just need to look into constants.sh and hadoop-preconditions.sh to use custom configuration. I have tried to make it as generic as possible so it can be used by the community. I have open sourced it with the hope that others will make it better while forking of course.

Configuration

By default you get a Standalone Operation Mode configuration (for development purposes) but for Production you will need a Fully Distributed Mode Configuration. A Pseudo Distributed Mode Configuration is an alternative in the middle, while it is not suitable for production it simulates real production.

Standalone

Use this mode to rapidly test MapReduce jobs. There is nothing to configure. As per the original documentation here is how you can test Hadoop running a MapReduce example that will search a regex in several XML files:
mkdir -p ~/hadoop-tests/input
cd ~/hadoop-tests/
cp $HADOOP_HOME/conf/*.xml input
hadoop jar $HADOOP_HOME/hadoop-examples*.jar grep input output 'dfs[a-z.]+' 

Pseudo Distributed

As a precondition we need to authorize our public key in our own host. If "ssh localhost" does not login without prompting for password then install the ssh public key using the recipe below the hadoop directory. It should be included anyway in the host recipe for the box where you are configuring hadoop in this mode.

After installation you should be able to access NameNode from http://localhost:50070/ and JobTracker from http://localhost:50030/

To run now the same example we run for the Standalone installation we will need to have the input folder in HDFS (instead of the local file system):
hadoop fs -mkdir input
hadoop fs -put $HADOOP_HOME/conf/*.xml input
hadoop jar $HADOOP_HOME/hadoop-examples*.jar grep input output 'dfs[a-z.]+' 
The result will be exactly the same you got from the standalone installation. You can inspect now the input (/user/hadoop/input) and output (/user/hadoop/output) directory using "Browse the filesystem" link from http://localhost:50070 or from command line:
hadoop fs -cat output/part-00000
Hadoop uses 4 daemons you can start separately using the commands below:
hadoop-daemon.sh start namenode
hadoop-daemon.sh start jobtracker
hadoop-daemon.sh start tasktracker
hadoop-daemon.sh start datanode
Or as we saw in the recipe you can just start them all:
start-all.sh
To stop use just 'stop instead of 'start'
Troubleshooting dev environment
I found myself running into lot of issues related to corrupted file system. Look at the page http://localhost:50070/dfshealth.jsp and make sure you have "DFS Remaining", if you do not, then try freeing DHFS space or just recreate the file system. As this is a development environment that option is reasonable of course. Notice an alternative to stop-all.sh below. I had to use this approach because I found hadoop java processes running even after running the stop all command:
ps -ef|grep hadoop|grep -v grep|awk '{print $2}'|xargs kill
rm -fr /tmp/hadoop-`whoami`/
hadoop namenode -format
start-all.sh 

Fully Distributed

There is a lot that can be tweaked in configuration files but the pob-recipes project I have shared is the bare minimum to run NameNode+JobTracker (Master) in one machine and DataNode+TaskTracker (Workers) in a couple of other machines. My hadoop-constants.sh you will notice uses 3 Ubuntu VMs (hadoop-master, hadoop-slave1 and hadoop-slave2) but you can easily separate hadoop-master into hadoop-namenode and hadoop-jobtracker to get the 4 minimum production machines I was referring to. The only reason for me choosing just one master machine is my current notebook memory resources.

Make no mistake, developing a Hadoop Job is not as big deal as setting correctly your cluster. The cluster administrator must keep an eye on hints, tips and tricks but also must monitor the cluster to make sure it performs the best possible way and NameNode is backed up correctly (and the restore tested of course). We must ensure there is no single point of failure and in the case of an undesired event the cluster can be back up and running ASAP.

The hadoop POB recipes should allow you to configure your details in hadoop/hadoop-constants.sh (1 hadoop master and two hadoop slaves by default). You just need to run the slaves and master recipes in that order and you should get the below responses (between bunch of other feedback). For the master:
11756 JobTracker
8834 NameNode
For the slaves:
6333 DataNode
9712 TaskTracker
Clearly the cluster is running with the four expected daemons. Running a job in your cluster is as easy as already explained in the pseudo distributed section. Run the job in master while inspecting the log file in both slaves, for example:
tail -f /usr/local/hadoop/logs/hadoop-hadoop-datanode-hadoop-slave2.log


Pointing your browser to http://hadoop-master:50070 should show you two Live Nodes and the amount of disk available (DFS Remaining).

Here are the default web Administrative/Informative URLs to get information about your hadoop cluster:
http://hadoop-master:50070/ – NameNode (HDFS Information)
http://hadoop-master:50030/ – JobTracker (Map Reduce Administration)
http://hadoop-slave1:50060/ – TaskTracker (Task Tracker status)

Problem Solution

Our authentication auditing exercise can be for sure solved with some unix power tools run from command line as I show below however the power of Hadoop will allow the tasks to run faster for big files coming from different servers in addition to ensure the data is kept saved thanks to the HDFS redundancy. So from command line:
root@hadoop-master:~# cat /var/log/auth.log | grep "Failed password" | awk '{if($9 == "invalid") print $1"-"$2"-"substr($3,1,5)" "$11" "$9; else print $1"-"$2"-"substr($3,1,5)" "$9" WrongPassword"}'
Jul-17-15:42 hadoop WrongPassword
Jul-17-15:45 hadoop WrongPassword
Jul-17-15:46 hadoop WrongPassword
Jul-17-17:14 hadoop WrongPassword
Jul-17-17:32 hadoop WrongPassword
Jul-17-17:33 hadoop WrongPassword
Jul-17-17:47 hadoop WrongPassword
Jul-18-03:57 hadoop WrongPassword
Jul-19-19:43 nestor invalid
Jul-19-19:44 root WrongPassword
Jul-19-20:29 attacker invalid
Jul-19-20:29 attacker invalid
Jul-19-20:29 attacker invalid
Jul-19-20:29 attacker invalid
Hadoop by default will use a tab character (\t) as deliniter for key value pairs. We will cheat for this exercise and instead we will generate from the mapper just lines of fields separated by spaces, hence hadoop will see the lines as keys with null values.

Let us then put in a shell script our map code to extract the fields we are interested in:
root@hadoop-master:~# vi mapper.sh
grep "Failed password" | awk '{if($9 == "invalid") print $1"-"$2"-"substr($3,1,5)" "$11" "$9; else print $1"-"$2"-"substr($3,1,5)" "$9" WrongPassword"}'
root@hadoop-master:~# chmod +x mapper.sh 
root@hadoop-master:~# cat /var/log/auth.log | ./mapper.sh
root@hadoop-master:~# cat /var/log/auth.log | ./mapper.shtime, user failure
Jul-17-15:42 hadoop WrongPassword
Jul-17-15:45 hadoop WrongPassword
Jul-17-15:46 hadoop WrongPassword
Jul-17-17:14 hadoop WrongPassword
Jul-17-17:32 hadoop WrongPassword
Jul-17-17:33 hadoop WrongPassword
Jul-17-17:47 hadoop WrongPassword
Jul-18-03:57 hadoop WrongPassword
Jul-19-19:43 nestor invalid
Jul-19-19:44 root WrongPassword
Jul-19-20:29 attacker invalid
Jul-19-20:29 attacker invalid
Jul-19-20:29 attacker invalid
Jul-19-20:29 attacker invalid
Now with a reduce function we can obtain the number of failed attempts per user and failure:
root@hadoop-master:~# cat /var/log/auth.log | ./mapper.sh | awk 'BEGIN{print "UserFailure count"} {userFailures[$2"-"$3]++}END{for (userFailure in userFailures) printf("%s %i\n", userFailure, userFailures[userFailure])}'
UserFailure count
root-WrongPassword 1
hadoop-WrongPassword 8
attacker-invalid 4
nestor-invalid 1
root@hadoop-master:~# vi failedAttemptsPerUserAndFailureReduce.sh
awk 'BEGIN{print "UserFailure count"} NR!=1 {userFailures[$2"-"$3]++}END{for (userFailure in userFailures) printf("%s %i\n", userFailure, userFailures[userFailure])}'
root@hadoop-master:~# chmod +x failedAttemptsPerUserAndFailureReduce.sh 
root@hadoop-master:~# cat /var/log/auth.log | ./mapper.sh | ./failedAttemptsPerUserAndFailureReduce.sh 
UserFailure count
root-WrongPassword 1
hadoop-WrongPassword 8
attacker-invalid 4
nestor-invalid 1
Our second exercise can be solved with another reduce example (to detect during which period of time we had the most attack attempts):
root@hadoop-master:~# cat /var/log/auth.log | ./mapper.sh | awk '{if($3 == "invalid") invalids[$1]++}END{for (invalid in invalids) print invalids[invalid]" "invalid}'| sort -n -r | head -1
4 Jul-19-20:29
root@hadoop-master:~# vi maxAttacksInAMinuteReduce.sh
awk 'NR!=1{if($3 == "invalid") invalids[$1]++}END{for (invalid in invalids) print invalids[invalid]" "invalid}'| sort -n -r | head -1
root@hadoop-master:~# chmod +x maxAttacksInAMinuteReduce.sh 
root@hadoop-master:~# cat /var/log/auth.log | ./mapper.sh | ./maxAttacksInAMinuteReduce.sh 
4 Jul-19-20:29
Now let's run both map and reduce jobs in a hadoop cluster. The first thing is we need to copy the file in HDFS:
hadoop fs -put auth.log input/authLog/auth.log
Then we need to delete any results in case we are running our jobs again:
hadoop fs -rmr output/authLog
Finally we run our jobs:
$HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/contrib/streaming/hadoop-streaming* -file ./mapper.sh -mapper "mapper.sh" -file ./failedAttemptsPerUserAndFailureReduce.sh -reducer "failedAttemptsPerUserAndFailureReduce.sh" -input input/authLog/auth.log -output output/authLog/failedAttemptsPerUserAndFailure
$HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/contrib/streaming/hadoop-streaming* -file ./mapper.sh -mapper "mapper.sh" -file ./maxAttacksInAMinuteReduce.sh -reducer "maxAttacksInAMinuteReduce.sh" -input input/authLog/auth.log -output output/authLog/maxAttacksInAMinute
We can get the results as explained before from command line or web interface.

We could sync our log files from different servers and copy them into HDFS later on, then run the jobs to obtain a result across all of our servers but there should be a more direct way.

What about putting the files in HDFS directly from remote machines? To query across multiple server logs this would be ideal. In bash this is easy indeed. Using ssh you can pipe any stream into a command:
echo "Hello" | ssh hadoop@hadoop-master 'while read data; do echo $data from `hostname`; done'
Then use the hadoop fs put command to push the log file from the remote server into the hadoop-master NameNode. You will need first authorize the server key for a particular user using the POB recipe I previously posted like:
ssh-keygen -t rsa -N '' -f /root/.ssh/id_rsa
common/tools/ssh-copy-id-uniq.sh root hadoop hadoop-master /root/.ssh/id_rsa.pub /root/.ssh/id_rsa
Then push logs from that server like in:
cat /var/log/auth.log | ssh hadoop@hadoop-master "hadoop fs -rm input/authLogs/`hostname``date "+%Y%m%d"`; hadoop fs -put - input/authLogs/`hostname``date "+%Y%m%d"`" 
Finally run the jobs:
hadoop fs -rmr output/authLogs/failedAttemptsPerUserAndFailure
hadoop fs -rmr output/authLogs/maxAttacksInAMinute
hadoop jar $HADOOP_HOME/contrib/streaming/hadoop-streaming* -file ./mapper.sh -mapper "mapper.sh" -file ./failedAttemptsPerUserAndFailureReduce.sh -reducer "failedAttemptsPerUserAndFailureReduce.sh" -input input/authLogs -output output/authLogs/failedAttemptsPerUserAndFailure
hadoop jar $HADOOP_HOME/contrib/streaming/hadoop-streaming* -file ./mapper.sh -mapper "mapper.sh" -file ./maxAttacksInAMinuteReduce.sh -reducer "maxAttacksInAMinuteReduce.sh" -input input/authLogs -output output/authLogs/maxAttacksInAMinute
POB has allowed us to automate hadoop installations and to build and run remote hadoop jobs that can be used to improve our Security Intelligence, an aspect that directly affects Business and so indirectly becomes part of Business Intelligence as well. Not to mention Hadoop power to aid directly on Business Intelligence for example while analyzing Server Application logs and identifying user trends.

Friday, July 20, 2012

Monitoring Ubuntu logs

There is a main file you should be monitoring in Ubuntu (not the only one though) and it is /var/log/kern.log. Here is as a reminder a poor man script to send alerts under certain situations.

Thursday, July 19, 2012

Security: Firefox connection to website partially encrypted and eavesdropping

What happens when an SSL Certificate has not been signed by some Intermediate CA and Apache does not include the certificate with a SSLCACertificateFile directive?

I was surprised to learn today Windows and Linux different browsers were complaining including Chrome while my OSX Chrome version (Version 22.0.1207.1 dev) was failing to do so.

Not sure if this is a Chrome vulnerability but certainly I have filled a Chrome feedback request for this issue.

In OSX Safari and Opera would fail to notice the issue and the only Browser that would complaint was Firefox with a "Your connection to this website is only partially encrypted, and does not prevent eavesdropping."

SSLLabs continues to rock as the ultimate online tool to check for your certificates and they have recently added a check (still in beta) for Certificate chain problems. So do check it out!

Do always test in all OS and all Browsers. OMG this fragmented market really ends up costing a lot of money to companies, especially those that take security seriously.

Wednesday, July 18, 2012

Avoid duplicates in authorized_keys after ssh-copy-id

This is a question that comes up every so often and here is a proposal through which you can override any existing authorized ssh public key. I have used this POB recipe with Remoto-IT while configuring Hadoop Clusters:
#!/bin/bash -ex
# ssh-copy-id-uniq.sh

localUser=$1
remoteUser=$2
remoteHost=$3
publicKey=$4
privateKey=$5

LOCAL_HOST_NAME=`hostname`

USAGE="Usage: `basename $0` <localUser> <remoteUser> <remoteHost> <publicKey> <privateKey>"

if [ $# -ne "5" ] 
then
 echo $USAGE
  exit 1 
fi

su $localUser -c "ssh-copy-id -i $publicKey $remoteUser@$remoteHost"
ssh -i $privateKey $remoteUser@$remoteHost "sed -i \"\\\$!{/$user@$LOCAL_HOST_NAME/d;}\" ~/.ssh/authorized_keys"
The below command will then deploy the public key from one server (For example using Remoto-IT in the hadoop master server) in the remote server (in this case a hadoop slave server) and will use the private key to password-less login and delete all lines referring to the current host for the given user:
common/tools/ssh-copy-id-uniq.sh $HADOOP_USER $HADOOP_USER $SLAVE_HOSTNAME /home/$HADOOP_USER/.ssh/id_rsa.pub /home/$HADOOP_USER/.ssh/id_rsa

Monday, July 16, 2012

echo-ing new lines in bash

For the sake of our memories:
$ echo "line\nnewline"
line\nnewline
$ echo -e "line\nnewline"
line
newline
$ lines=`echo -e "line\nnewline"`
$ echo $lines
line newline
$ echo "$lines"
line
newline
Clearly you must use -e if you want echo to recognize newline escape character (\n). In addition notice how quotes are needed to print the new line stored in a variable while the -e option is not needed.

Friday, July 13, 2012

NBSP versus SP: The non breaking space which breaks

Yup the Non Break Space (NBSP) character (octal 240, hex A0, dec 160) does not play good with scripts. Sometimes you copy and paste snippets of code from websites like Thinking In Software and the code simply does not work and you get errors like:
'\240`: command not found
Clearly you need to clean the code and here is one of the methods to do so. It is just a POB as usual: The character combination \302\240 (U+00A0) is usually found as well when copying code from websites.

However if you are serious about posting code snippets I recommend using gist. Just paste your code or add modifications in the future. You can copy the content from "Embed this gist" which is a script tag you can directly paste in your blog. The code above is coming from nbsp2sp.sh gist.

Thursday, July 12, 2012

POB Recipe to refresh Ubuntu servers repo - solving 404 Not Found

From time to time Ubuntu servers from where packages are downloaded (repository) become unavailable. The below POB recipe can be run remotely to solve this problem in your servers:
#!/bin/bash -e
# ubuntu-refresh-servers-repo.sh

sudo mv /var/lib/apt/lists{,bak`date "+%Y%m%d"`}
sudo mv /var/cache/apt/archives/partial{,bak`date "+%Y%m%d"`}
sudo apt-get update

POB recipe to install SVN credentials in a remote Server

If you are keeping configuration files in SVN you will need to retrieve them from your servers while managing or configuring them. After all you should automate everything but also keep track of your changes. Remember The palest ink is better than the best memory.

You do not want to be running svn passing credentials every time you need to pull a configuration file (or even a script if you practice bootstrapping ;-) so here is a POB recipe to remotely install credentials for SVN in a remote server:
#!/bin/bash -e
# svn.sh

svnrepo=$1

USAGE="Usage: `basename $0` <svnrepo>"

if [ $# -ne "1" ] 
then
 echo $USAGE
  exit 1 
fi

echo -n "SVN User: "
read svnuser
echo -n "SVN Password: "
read -s svnpassword
echo

apt-get -q -y install subversion || svn --version
sed -i 's/# store-passwords = no/store-passwords = yes/g' .subversion/servers
sed -i 's/# store-plaintext-passwords = no/store-plaintext-passwords = yes/g' .subversion/servers
echo $svnpassword | svn info $svnrepo --username $svnuser
Now you can run POB recipes that use svn repo remotely without the need to supply credentials.

Avoid slow SSH response using POB

Here is a quick POB recipe that you can run remotely in those servers taking unbearable 10 seconds to respond to a simple remote ssh command (including login).
#!/bin/bash -e
# ssh-config.sh

#Avoid slow SSH response due to reverse client IP lookup
sed -i "/UseDNS/d" /etc/ssh/sshd_config && bash -c "echo 'UseDNS no' >> /etc/ssh/sshd_config"
service ssh restart

Change hostname in Ubuntu with POB

Every now and then some services don't work just because Linux has a wrong hostname. A hostname has a short form and a long (FQDN) form. The commands `hostname` and `hostname -f` should not output the same. They correspond to short and long form respectively.

This is a POB recipe to change the host name in Ubuntu (and probably other non Debian systems). I have tested this to remotely change machine names through Remoto-IT.
#!/bin/bash -e
# common/change-hostname.sh

hostname=$1

USAGE="Usage: `basename $0` "

if [ $# -ne "1" ] 
then
 echo $USAGE
 exit 1 
fi

#It is mandatory to provide an FQDN which at the minimum is a subdomain address 
if [[ ! "$hostname" =~ ^.*\..*\..*$ ]]
then
 echo $USAGE
 exit 1
fi

#bash substring removal
short=${hostname%%.*}

sed -i "/$short/d" /etc/hosts
bash -c "echo '127.0.0.1 $hostname $short' >> /etc/hosts"
echo $short > /etc/hostname
hostname -F /etc/hostname
Now you can run it to change your hostname (I use remoto-it to deploy it remotely):
common/change-hostname.sh myhost.sample.com
In your remote host the below commands should correctly output the short name and the FQDN depending on the flags passed:
$ hostname
myhost
$ hostname -f
myhost.sample.com

Tuesday, July 10, 2012

Patching or Installing Java in Ubuntu from a POB recipe

Patching Java is just about reinstalling it. I commonly use symlinks so changing from version to version should be easy. Massively updating Java is important especially when a security update is placed by Oracle.

Remoto-IT can be used to remotely deploy the below recipe. Note that it uses a mounting point and some simple common recipes that allows to mount to the correct CIFS mounting point:
#!/bin/bash -e

USAGE="Usage: `basename $0`      "
POB_LOCAL_RES_DIR="/mnt/pob-resource-repository"
CREDENTIALS_PATH="/root/projects/cifs/cifs_smbmount.txt"

distributionFileName=$1
distributionVersion=$2
windowsDomain=$3
linuxUser=$4
linuxGroup=$5
cifsDirectory=$6

if [ $# -ne "6" ] 
then
 echo $USAGE
  exit 1 
fi

common/umount-path.sh $POB_LOCAL_RES_DIR
common/mount-cifs.sh $cifsDirectory $POB_LOCAL_RES_DIR $CREDENTIALS_PATH $windowsDomain $linuxUser $linuxGroup
cp /mnt/pob-resource-repository/$distributionFileName /opt/
cd /opt
rm -fr /opt/$distributionVersion
chmod +x $distributionFileName
echo "yes" "\n" | ./$distributionFileName 1>/dev/null
rm -f $distributionFileName
rm -f /usr/bin/java /usr/bin/javac /usr/bin/jar
ln -s /opt/$distributionVersion/bin/java /usr/bin/java
ln -s /opt/$distributionVersion/bin/javac /usr/bin/javac
ln -s /opt/$distributionVersion/bin/jar /usr/bin/jar
chown -R $linuxUser:$linuxGroup /opt/$distributionVersion/
java -version
The above is invoked as in:
common/java.sh jdk-6u33-linux-x64.bin jdk1.6.0_33 myWindowsDomain myLinuxUser myLinuxGroup //filer.sample.com/pob-resource-repository
And as you already noticed there are two recipes it relies upon. One to mount:
#!/bin/bash -e
# mount-cifs.sh

USAGE="Usage: `basename $0` <cifsPath> <localPath> <credentialsPath> <domain> <uid> <gid>"

cifsPath=$1
localPath=$2
credentialsPath=$3
domain=$4
uid=$5
gid=$6


if [ $# -ne "6" ] 
then
 echo $USAGE
  exit 1 
fi

if ! grep -qs "$localPath" /proc/mounts; then
    mount -t cifs "$cifsPath" "$localPath" -o credentials=$credentialsPath,domain=$domain,file_mode=0600,dir_mode=0700,uid=$uid,gid=$gid
fi
And the one to unmount:
#!/bin/bash -e
# umount-path.sh

USAGE="Usage: `basename $0` <localPath>"

localPath=$1


if [ $# -ne "1" ] 
then
 echo $USAGE
  exit 1 
fi

if grep -qs "$localPath" /proc/mounts; then
    umount "$localPath"
fi
Of course it is expected CIFS to be available and a file with the credentials to exist. Here is a sample POB recipe that I use to make sure CIFS is installed:
#!/bin/bash -e
# cifs.sh

apt-get -q -y install smbfs
mkdir -p /mnt/pob-resource-repository
mkdir -p  /root/projects/cifs
cd /root/projects/cifs
svn export http://svn.sample.com/repos/reporting/environment/production/settings/bhub-tomcat-all/cifs_smbmount.txt

Monday, July 09, 2012

Cygwin SSH to manage Windows from Linux

Managing Windows from Linux servers might be tricky but our old friend SSH could help us run remote commands. Here I present a Plain Old Bash Recipe (POB) prepared just for Cygwin to get SSH working. It should be of course idempotent. In addition I am presenting a VBS script to install cygwin from MS command line. It would be great if both could be integrated but at the moment I have no time for more.

Unfortunately fully automation of a cygwin plus OpenSSH installation in Windows is really difficult to achieve. For one we need to run the command line prompt as administrator and that is not available in "unnatended mode". Interaction will be needed. With Virtualization you can build templates and use them at least to avoid normal (not disaster related) installations or re-installations but having idempotent recipes should be the way to go, anyway move on!.

Cygwin uses Windows users and so this procedure assumes you provide a valid local Windows user and password.
  1. Download the cygwin into c:\scripts\setup.exe
  2. Install cygwin from Windows command prompt by running the below script (cscript c:\scripts\cygwin-install.vbs):
    '''''''''''''''''''''''''''''''''''''''''''''''
    '
    ' c:\scripts\cygwin-install.vbs
    
    ' @author Nestor Urquiza
    ' @date 20120706
    ' @description: Based on several snippets of code from the web:
    '               https://gist.github.com/2053179
    '
    '
    '
    ''''''''''''''''''''''''''''''''''''''''''''''''
    
    
    '
    ' Functions 
    '
    
    'Necessary as cygwin setup aborts when a package is already installed
    On Error Resume Next
    
    Function wget(URL)
      ' Fetch the file
      Set objXMLHTTP = CreateObject("MSXML2.XMLHTTP")
    
      objXMLHTTP.open "GET", URL, false
      objXMLHTTP.send()
      'Wscript.Echo objXMLHTTP.Status
      If objXMLHTTP.Status = 200 Then
        Set objADOStream = CreateObject("ADODB.Stream")
        objADOStream.Open
        objADOStream.Type = 1 'adTypeBinary
    
        objADOStream.Write objXMLHTTP.ResponseBody
        objADOStream.Position = 0    'Set the stream position to the start
    
        Set objFSO = Createobject("Scripting.FileSystemObject")
        parts = split(URL,"/") 
        saveTo = parts(ubound(parts))
        If objFSO.Fileexists(saveTo) Then objFSO.DeleteFile saveTo
        Set objFSO = Nothing
    
        objADOStream.SaveToFile saveTo
        objADOStream.Close
        Set objADOStream = Nothing
      Else
        Err.Raise 8 'Bad response ' objXMLHTTP.Status & ' from ' & URL
      End if
    
      Set objXMLHTTP = Nothing  
    End Function
    
    Function installPackage(package)
      Wscript.Echo "-----------Installing " & package
      Set objShell = CreateObject("WScript.Shell")
      objShell.run "c:\scripts\setup.exe -n -q -s ftp://lug.mtu.edu/cygwin -P " & package, 0, True
      objShell = Nothing
    End Function 
    
    '
    'Main
    '
    
    'currentPath = CreateObject("Scripting.FileSystemObject").GetAbsolutePathName(".")
    'Wscript.Echo currentPath
    
    'Download cygwin
    wget "http://www.cygwin.com/setup.exe"
    
    'Install Cygwin if not present and download specific packages
    installPackage("wget")
    
    ' Done
    WScript.Quit
    
  3. Start cygwin as Administrator. Right click C:\cygwin\Cygwin.bat and select Run as Administrator"
  4. Run the below script from cygwin (/usr/tmp/cygwin-ssh-setup.sh)
    #!/usr/bin/bash -e
    #
    # /usr/tmp/cygwin-ssh-setup.sh
    #
    # @author Nestor Urquiza
    # @date 20120709
    # @description Set up SSH access for Cygwin
    #
    SSH_USER=$1
    SSH_PWD=$2
     
    USAGE="Usage: `basename $0` <ssh user> <ssh password>"
     
    if [ $# -ne "2" ] 
    then
     echo $USAGE
         exit 1 
    fi
     
    #install a package manager
    wget http://apt-cyg.googlecode.com/svn/trunk/apt-cyg -O apt-cyg
    chmod +x apt-cyg
    mv apt-cyg /usr/bin/
     
    #install needed packages via apt-cyg
    #/usr/bin/apt-cyg remove openssh
    /usr/bin/apt-cyg install openssh
    /usr/bin/apt-cyg install tcl
    /usr/bin/apt-cyg install expect
    /usr/bin/apt-cyg install pwgen
    /usr/bin/apt-cyg install vim
    /usr/bin/apt-cyg install screen
     
    #clean any previous openssh configuration
    net start|grep sshd && net stop sshd
    reg_sshd_key="HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\services\sshd"
    reg query $reg_sshd_key 2&>1 && reg delete $reg_sshd_key /f
    sc query ssd 2&>1 && sc delete sshd
    ps -ef | grep sshd | awk  '{print $2}' | xargs kill -9
    cygrunsrv --list | grep sshd && cygrunsrv --stop sshd && cygrunsrv --remove sshd
    net user | grep sshd && net user sshd /delete
    net user | grep cyg_server && net user cyg_server /delete
    grep $SSH_USER /etc/passwd || mkpasswd -l -u $SSH_USER >> /etc/passwd
    sed "/sshd.*/d"  /etc/passwd > tmp_passwd &&   mv tmp_passwd  /etc/passwd
    sed "/cyg_server.*/d"  /etc/passwd > tmp_passwd &&   mv tmp_passwd  /etc/passwd
     
    #configure openssh
    ssh-host-config -y -w `pwgen -y -N 1 10`
     
     
    #start ssh
    net start sshd
     
    /usr/bin/expect -c "
      spawn /usr/bin/passwd $SSH_USER
      expect \"password:\" {
        send \"$SSH_PWD\n\"
        expect \"password:\" {
          send \"$SSH_PWD\n\"
        }
      }
    "
    
  5. Authorize your public key in the remote windows server. Run the below from the client machine:
    ssh-copy-id -i ~/.ssh/id_rsa.pub user@remote.domain.com
    
  6. You should be able to run a remote command in windows from your linux box. For example let us list which version of VBS the windows box is running:
    ssh user@remote.domain.com "/cygdrive/c/Windows/System32/cscript.exe"
    
This is of course handy for absolutely every automation in Windows Servers. One showcase of it would be backup procedures for MSSQL Server which run in Windows and which you want to trigger from Linux for example.

cygwin fork: Resource temporarily unavailable

Not claiming the titled error will be solved 100% of the cases but probably it will go away after you ...

Restart Windows

Before you get lost into hundreds of posts I would give it a try.

After all Windows restart has been *the solution* for so many "problems" for years that I have a hard time thinking it will be any different ever (at least in my earth lifetime;-)

Thursday, July 05, 2012

Round Half Up using Annotation-Driven Formatting from Spring 3.1

Even though bankers prefer to round half even (the default in Java Decimal Format) in Accounting (surprisingly?) rounding half up is preferred at least in some scenarios. Besides the monetary implications which should concern only business there is a fundamental problem on the technology side and it is: Where should the rounding be done?

My attention was brought to a JSTL fmt:formatNumber which is still missing the apparently necessary rounding method. This made me think twice if rounding actually belongs to front end at all.

I believe in separation of concerns as one of the most important concepts a good Architect must master and in this case I can't see why the rounding strategy and even how many decimal places should be a front end concern. It looks wrong to have to format numbers in a native Ipad app plus a native Android app plus a WEB/WAP/HTML5 app. That cannot be the way. Let us don't even talk about testing and maintainability.

There is a process called binding which is applied one way or the other when a Controller passes or accepts content to a form in the View. I clearly can see how the number conversions (if any) should be done as part of Binding.

Below is how you achieve this from a Spring Controller using the InitBinder annotation and a Formatter:
 @InitBinder
    public void initBinder(WebDataBinder dataBinder) {
        dataBinder.registerCustomEditor(Date.class, new MultipleDateEditor());
        DecimalFormat df = new DecimalFormat();
        df.setMaximumFractionDigits(3); //Round always to the third digit
        df.setRoundingMode(RoundingMode.HALF_UP); //Default is HSLF_EVEN
        dataBinder.registerCustomEditor(BigDecimal.class, "mtdQtdNetReturn", new CustomNumberEditor(BigDecimal.class, df, true));
        ...
    }
The above is valid only for forms if you want to persist the rounded numbers. But formatting is needed in non form pages as well and in fact you probably need to keep your forms presenting and persisting data with all decimal places while listing and detail pages need certain rounding (BTW persisting rounded values is probably a bad practice but that is a subject for a different discussion).

One way or the other for informational pages there should be a solution of course. It makes sense to have a Formatter applied to all relevant fields.

Spring 3.1 allows to use annotations like @NumberFormat. If you are using JPA make sure the Spring annotation is the closest to the field otherwise it won't work. Here is a the right approach:
@Column(precision = 8, scale = 6)
@NumberFormat(pattern = "###,###.###")
private BigDecimal mtdQtdNetReturn;
@NumberFormat is missing though several important features of a number formatter when it comes to Decimal/BigDecimal format. One of those is rounding. Another handy one is multiplier and while pattern allows to specify the fraction digits it does not work for Currency or Percent types. I had to hook into Spring Conversion API to get rounding/multiplier/fractionDigits supported and here is how I did it with a CustomNumberFormatter (and some other necessary Spring classes).

Let us consider the number 454.2225. Here are some examples of how the number can be formatted (US-EN locale) and the corresponding options for the annotation:
  1. 454.223
    @CustomNumberFormat(style = Style.NUMBER, fractionDigits = 3, roundingMode = RoundingMode.HALF_UP, multiplier = 100)
    
  2. 454.223%
    @CustomNumberFormat(style = Style.PERCENT, fractionDigits = 3, roundingMode = RoundingMode.HALF_UP, multiplier = 100)
    
  3. $454.223
    @CustomNumberFormat(style = Style.CURRENCY, fractionDigits = 3, roundingMode = RoundingMode.HALF_UP, multiplier = 100)
    
In spring configuration file:
<mvc:annotation-driven conversion-service="conversionService"/>
    
    <bean id="conversionService"
          class="org.springframework.format.support.FormattingConversionServiceFactoryBean">
        <property name="formatters">
            <set>
                <bean class="com.nestorurquiza.format.number.CustomNumberFormatAnnotationFormatterFactory"/>
            </set>
        </property>
    </bean>
In JSP:
<spring:eval expression="clientAssetValue.mtdQtdNetReturn"/>
Here is the Spring Factory:
package com.nestorurquiza.format.number;

import java.math.BigDecimal;
import java.math.RoundingMode;
import java.util.Collections;
import java.util.HashSet;
import java.util.Set;

import org.springframework.context.EmbeddedValueResolverAware;
import org.springframework.format.AnnotationFormatterFactory;
import org.springframework.format.Formatter;
import org.springframework.format.Parser;
import org.springframework.format.Printer;
import org.springframework.format.number.CustomCurrencyFormatter;
import org.springframework.util.StringUtils;
import org.springframework.util.StringValueResolver;

import com.nestorurquiza.format.annotation.CustomNumberFormat;
import com.nestorurquiza.format.annotation.CustomNumberFormat.Style;

public class CustomNumberFormatAnnotationFormatterFactory implements AnnotationFormatterFactory<CustomNumberFormat>, EmbeddedValueResolverAware {

    private final Set<Class<?>> fieldTypes;
    
    private StringValueResolver embeddedValueResolver;
    
    public CustomNumberFormatAnnotationFormatterFactory() {
        Set<Class<?>> rawFieldTypes = new HashSet<Class<?>>(1);
        rawFieldTypes.add(BigDecimal.class);
        this.fieldTypes = Collections.unmodifiableSet(rawFieldTypes);
    }
    
    @Override
    public void setEmbeddedValueResolver(StringValueResolver resolver) {
        this.embeddedValueResolver = resolver;
    }

    @Override
    public final Set<Class<?>> getFieldTypes() {
        return this.fieldTypes;
    }

    @Override
    public Printer<?> getPrinter(CustomNumberFormat annotation, Class<?> fieldType) {
        return configureFormatterFrom(annotation);
    }

    @Override
    public Parser<?> getParser(CustomNumberFormat annotation, Class<?> fieldType) {
        return configureFormatterFrom(annotation);
    }
    
    protected String resolveEmbeddedValue(String value) {
        return (this.embeddedValueResolver != null ? this.embeddedValueResolver.resolveStringValue(value) : value);
    }
    
    /**
     * Style.NUMBER accepts pattern, multiplier and roundingMode
     * Style.CURRENCY accepts 
     * @param annotation
     * @return
     */
    private Formatter<Number> configureFormatterFrom(CustomNumberFormat annotation) {
        RoundingMode roundingMode = annotation.roundingMode();
        int multiplier = annotation.multiplier();
        int fractionDigits = annotation.fractionDigits();
        
        Style style = annotation.style();
        if (style == Style.PERCENT) {
            CustomPercentFormatter bigDecimalPercentFormatter = new CustomPercentFormatter();
            bigDecimalPercentFormatter.setRoundingMode(roundingMode);
            bigDecimalPercentFormatter.setFractionDigits(fractionDigits);
            bigDecimalPercentFormatter.setMultiplier(multiplier);
            return bigDecimalPercentFormatter;
        }
        else if (style == Style.CURRENCY) {
            CustomCurrencyFormatter currencyFormatter = new CustomCurrencyFormatter();
            currencyFormatter.setRoundingMode(roundingMode);
            currencyFormatter.setFractionDigits(fractionDigits);
            currencyFormatter.setMultiplier(multiplier);
            return currencyFormatter;
        }
        else {
            CustomNumberFormatter numberFormatter = new CustomNumberFormatter();
            if (StringUtils.hasLength(annotation.pattern())) {
                numberFormatter.setPattern(resolveEmbeddedValue(annotation.pattern()));
            }
            numberFormatter.setRoundingMode(roundingMode);
            numberFormatter.setFractionDigits(fractionDigits);
            numberFormatter.setMultiplier(multiplier);
            
            return numberFormatter;
        }
        

    }
    
    
}
As you can see the Factory refers to an annotation and three formatters. Here is the annotation:
package com.nestorurquiza.format.annotation;

import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;
import java.math.RoundingMode;

@Target({ ElementType.METHOD, ElementType.FIELD, ElementType.PARAMETER })
@Retention(RetentionPolicy.RUNTIME)
public @interface CustomNumberFormat {
    /**
     * The style pattern to use to format the field.
     * Defaults to {@link Style#NUMBER} for general-purpose number formatter.
     * Set this attribute when you wish to format your field in accordance with a common style other than the default style.
     */
    Style style() default Style.NUMBER;

    /**
     * The custom pattern to use to format the field.
     * Defaults to empty String, indicating no custom pattern String has been specified.
     * Set this attribute when you wish to format your field in accordance with a custom number pattern not represented by a style.
     */
    String pattern() default "";
    
    int multiplier() default 1;
    
    RoundingMode roundingMode() default RoundingMode.HALF_EVEN; 

    int fractionDigits() default 2;

    /**
     * Common number format styles.
     * @author Keith Donald
     * @since 3.0
     */
    public enum Style {

        /**
         * The general-purpose number format for the current locale.
         */
        NUMBER,
        
        /**
         * The currency format for the current locale.
         */
        CURRENCY,

        /**
         * The percent format for the current locale.
         */
        PERCENT
    }
}
The formatters:
package com.nestorurquiza.format.number;

import java.math.RoundingMode;
import java.text.DecimalFormat;
import java.text.NumberFormat;
import java.util.Locale;

import org.springframework.format.number.NumberFormatter;

public class CustomNumberFormatter extends NumberFormatter {

    private int multiplier;
    private RoundingMode roundingMode; 
    private int fractionDigits = 2;
    
    public int getMultiplier() {
        return multiplier;
    }


    public void setMultiplier(int multiplier) {
        this.multiplier = multiplier;
    }


    public RoundingMode getRoundingMode() {
        return roundingMode;
    }


    public void setRoundingMode(RoundingMode roundingMode) {
        this.roundingMode = roundingMode;
    }

    

    public int getFractionDigits() {
        return fractionDigits;
    }


    public void setFractionDigits(int fractionDigits) {
        this.fractionDigits = fractionDigits;
    }


    @Override
    public NumberFormat getNumberFormat(Locale locale) {
        NumberFormat format = super.getNumberFormat(locale);
        DecimalFormat decimalFormat = (DecimalFormat) format;
        decimalFormat.setMultiplier(multiplier);
        decimalFormat.setRoundingMode(roundingMode);
        decimalFormat.setMaximumFractionDigits(fractionDigits);
        return decimalFormat;
    }

}  
package com.nestorurquiza.format.number;

import java.math.RoundingMode;
import java.text.DecimalFormat;
import java.text.NumberFormat;
import java.util.Locale;

import org.springframework.format.number.AbstractNumberFormatter;

public class CustomPercentFormatter extends AbstractNumberFormatter {

    private RoundingMode roundingMode = RoundingMode.HALF_EVEN;
    private int multiplier = 1;
    private int fractionDigits = 2;
    
    public RoundingMode getRoundingMode() {
        return roundingMode;
    }


    public void setRoundingMode(RoundingMode roundingMode) {
        this.roundingMode = roundingMode;
    }


    public int getMultiplier() {
        return multiplier;
    }


    public void setMultiplier(int multiplier) {
        this.multiplier = multiplier;
    }

    

    public int getFractionDigits() {
        return fractionDigits;
    }


    public void setFractionDigits(int fractionDigits) {
        this.fractionDigits = fractionDigits;
    }


    protected NumberFormat getNumberFormat(Locale locale) {
        NumberFormat format = NumberFormat.getPercentInstance(locale);
        if (format instanceof DecimalFormat) {
            DecimalFormat decimalFormat = ((DecimalFormat) format);
            decimalFormat.setParseBigDecimal(true);
            decimalFormat.setMultiplier(multiplier);
            decimalFormat.setRoundingMode(roundingMode);
            decimalFormat.setMaximumFractionDigits(fractionDigits);
        }
        return format;
    }

}
package org.springframework.format.number;

import java.text.DecimalFormat;
import java.text.NumberFormat;
import java.util.Locale;

public class CustomCurrencyFormatter extends CurrencyFormatter {

    private int multiplier = 1;
    
    @Override
    protected NumberFormat getNumberFormat(Locale locale) {
        DecimalFormat format = (DecimalFormat) super.getNumberFormat(locale);
        format.setMultiplier(multiplier);
        return format;
    }

    public int getMultiplier() {
        return multiplier;
    }

    public void setMultiplier(int multiplier) {
        this.multiplier = multiplier;
    }
    
    
}
Note that I am forced to use a Spring package for the CustomCurrencyFormatter to be able to extend the original class. Hopefully Spring will include these options to their @NumberFormat implementation in future versions.

Followers