Tuesday, December 22, 2009

Random Numbers For Android

PRIVACY POLICY: This app does not collect any personal information neither shares the random numbers generated by the device. It simply generates numbers as random as they can possibly be.
RandNum (Random Numbers) is available on Android. Please request fixes and features for the Android version on this page.

You can read about the application on my previous post where comments are welcome for the IPhone version.



Available in English, Chinese and Spanish it is not limited to those. New languages will be available following requests on this page.

This is my fifth Android app.

Saturday, December 12, 2009

Random Numbers


I just finished the development of a new IPhone application (RandNum) that accepts a range of numbers and a total amount of random numbers to be generated. Numbers are shown either by shaking or touching the device screen. Get it from here.

Random Numbers can be used to:

* Generate random numbers to play Lottery (Lotto and Lottery might be trademarks and they are not associated at all with this program).
* Play Lottery (Lotto and Lottery might be trade marks and they are not associated at all with this program).
* Pick a random page of a poetry book.
* Roll a dice.
* Make Raffle Tickets.
* Play your favorite depending-on-luck game.
* You name it ...


Available in English, Chinese and Spanish it is not limited to those. New languages will be available following requests on this page.

Get the Android version here.

Friday, November 13, 2009

Amazon Feeds Parser: Parsing Huge XML files

Amazon Feeds are huge and they are needed to locally search and browse their catalog as Amazon still does not allow more than 1 call per second to their PA-API. In any case even if you decide to go for the PA-API directly things like ordering a search result for a keyword in all categories will not be possible as simply that option is not available by the time of this writting.

I have opened a project which you can use to get tab delimited data from those huge feeds.

The tool I use is xgawk and I have to say that I have really enjoyed coding the *awk* way.

Sunday, October 18, 2009

Liferay for backend developers

This is a tutorial for Back End developers (Java code) in charge of developing Portlets for Liferay Portal.

As a backend developer needs to understand how the look and feel will be applied, this tutorial is a continuation of my recent "Liferay for Designers" tutorial. So, if you haven't done it please follow steps on the Designers tutorial before continuing.

Note that I am providing instructions here for two bundles, tomcat and jboss+tomcat. You better use tomcat for development purposes as it is lighter and just use JBoss if you really need J2EE capabilities not available from Tomcat.

Extension Environment

You will need to use this development path either when:
* You are extending an existing portlet portal built-in portlet or a portlet which has been built using the extension environment
* You have found the plugin environment will be not enough for the necessary logic.

Be sure you understand that no changes are done inside the ~/liferay/portal folder which we will create right away.

1. Uncompress liferay-portal-src-5.2.3.zip into ~/liferay/portal

2. Many parameters from ~/liferay/portal/portal-impl/src/portal.properties can be modified but you do so inserting the new values into ~/liferay/ext/ext-impl/src/portal-ext.properties. Many of the properties are actually modified using ~/liferay/ext/ext-impl/src/portal-developer.properties instead (if you do not have it there already just copy it from liferay/portal/portal-impl/src/portal-developer.properties) and deploy it using "ant deploy-properties" from ~/liferay/ext/ext-impl

3. Let us install p6spy so we can sniff JDBC.

Tomcat

Add the below at the end of file ~/liferay/tomcat-6.0.18/bin/setenv.sh:


JAVA_OPTS="-Dp6.home=$HOME/liferay/tomcat-6.0.18/lib/ext $JAVA_OPTS"
Jboss

Insert the below line at the very beginning of ~/liferay/jboss-tomcat-5.0.0/bin/run.sh


JBOSS_CLASSPATH="$HOME/liferay/jboss-tomcat-5.0.0/server/default/db"
Then create file spy.properties

Tomcat


~/liferay/tomcat-6.0.18/lib/ext/spy.properties
 Jboss


~/liferay/jboss-tomcat-5.0.0/server/default/db/spy.properties
Below is the content of spy.properties file. It can be further configured but with the settings below you will be just fine.


#################################################################
# P6Spy Options File #
# See documentation for detailed instructions #
#################################################################

#################################################################
# MODULES #
# #
# Modules provide the P6Spy functionality. If a module, such #
# as module_log is commented out, that functionality will not #
# be available. If it is not commented out (if it is active), #
# the functionality will be active. #
# #
# Values set in Modules cannot be reloaded using the #
# reloadproperties variable. Once they are loaded, they remain #
# in memory until the application is restarted. #
# #
#################################################################

module.log=com.p6spy.engine.logging.P6LogFactory
#module.outage=com.p6spy.engine.outage.P6OutageFactory

#################################################################
# REALDRIVER(s) #
# #
# In your application server configuration file you replace the #
# "real driver" name with com.p6spy.engine.P6SpyDriver. This is #
# where you put the name of your real driver P6Spy can find and #
# register your real driver to do the database work. #
# #
# If your application uses several drivers specify them in #
# realdriver2, realdriver3. See the documentation for more #
# details. #
# #
# Values set in REALDRIVER(s) cannot be reloaded using the #
# reloadproperties variable. Once they are loaded, they remain #
# in memory until the application is restarted. #
# #
#################################################################

# oracle driver
# realdriver=oracle.jdbc.driver.OracleDriver

# mysql Connector/J driver
# realdriver=com.mysql.jdbc.Driver

# informix driver
# realdriver=com.informix.jdbc.IfxDriver

# ibm db2 driver
# realdriver=COM.ibm.db2.jdbc.net.DB2Driver

# the mysql open source driver
#realdriver=org.gjt.mm.mysql.Driver

# mssql open source driver
realdriver=com.mysql.jdbc.Driver

#specifies another driver to use
realdriver2=
#specifies a third driver to use
realdriver3=


#the DriverManager class sequentially tries every driver that is
#registered to find the right driver. In some instances, it's possible to
#load up the realdriver before the p6spy driver, in which case your connections
#will not get wrapped as the realdriver will "steal" the connection before
#p6spy sees it. Set the following property to "true" to cause p6spy to
#explicitily deregister the realdrivers
deregisterdrivers=false

################################################################
# P6LOG SPECIFIC PROPERTIES #
################################################################
# no properties currently available

################################################################
# EXECUTION THRESHOLD PROPERTIES #
################################################################
# This feature applies to the standard logging of P6Spy. #
# While the standard logging logs out every statement #
# regardless of its execution time, this feature puts a time #
# condition on that logging. Only statements that have taken #
# longer than the time specified (in milliseconds) will be #
# logged. This way it is possible to see only statements that #
# have exceeded some high water mark. #
# This time is reloadable. #
#
# executionthreshold=integer time (milliseconds)
#
executionthreshold=

################################################################
# P6OUTAGE SPECIFIC PROPERTIES #
################################################################
# Outage Detection
#
# This feature detects long-running statements that may be indicative of
# a database outage problem. If this feature is turned on, it will log any
# statement that surpasses the configurable time boundary during its execution.
# When this feature is enabled, no other statements are logged except the long
# running statements. The interval property is the boundary time set in seconds.
# For example, if this is set to 2, then any statement requiring at least 2
# seconds will be logged. Note that the same statement will continue to be logged
# for as long as it executes. So if the interval is set to 2, and the query takes
# 11 seconds, it will be logged 5 times (at the 2, 4, 6, 8, 10 second intervals).
#
# outagedetection=true|false
# outagedetectioninterval=integer time (seconds)
#
outagedetection=false
outagedetectioninterval=

################################################################
# COMMON PROPERTIES #
################################################################

# filter what is logged
filter = true
#sqlexpression = .*statement.*

# comma separated list of tables to include when filtering
include =
# comma separated list of tables to exclude when filtering
exclude =

# sql expression to evaluate if using regex filtering
sqlexpression =


# turn on tracing
autoflush = true

# sets the date format using Java's SimpleDateFormat routine
dateformat=HH:mm:ss

#list of categories to explicitly include
includecategories =

#list of categories to exclude: error, info, batch, debug, statement,
#commit, rollback and result are valid values
#excludecategories=info,debug,result,batch,resultset
excludecategories=info,debug,result,batch,resultset,prepared

#allows you to use a regex engine or your own matching engine to determine
#which statements to log
#
#stringmatcher=com.p6spy.engine.common.GnuRegexMatcher
#stringmatcher=com.p6spy.engine.common.JakartaRegexMatcher
stringmatcher=

# prints a stack trace for every statement logged
stacktrace=false
# if stacktrace=true, specifies the stack trace to print
stacktraceclass=

# determines if property file should be reloaded
reloadproperties=false
# determines how often should be reloaded in seconds
reloadpropertiesinterval=60

#if=true then url must be prefixed with p6spy:
useprefix=false

#specifies the appender to use for logging
#appender=com.p6spy.engine.logging.appender.Log4jLogger
#appender=com.p6spy.engine.logging.appender.StdoutLogger
appender=com.p6spy.engine.logging.appender.FileLogger

# name of logfile to use, note Windows users should make sure to use forward slashes in their pathname (e:/test/spy.log) (used for file logger only)
logfile = spy.log

# append to the p6spy log file. if this is set to false the
# log file is truncated every time. (file logger only)
append=true

#The following are for log4j logging only
log4j.appender.STDOUT=org.apache.log4j.ConsoleAppender
log4j.appender.STDOUT.layout=org.apache.log4j.PatternLayout
log4j.appender.STDOUT.layout.ConversionPattern=p6spy - %m%n

#log4j.appender.CHAINSAW_CLIENT=org.apache.log4j.net.SocketAppender
#log4j.appender.CHAINSAW_CLIENT.RemoteHost=localhost
#log4j.appender.CHAINSAW_CLIENT.Port=4445
#log4j.appender.CHAINSAW_CLIENT.LocationInfo=true

log4j.logger.p6spy=INFO,STDOUT


#################################################################
# DataSource replacement #
# #
# Replace the real DataSource class in your application server #
# configuration with the name com.p6spy.engine.spy.P6DataSource,#
# then add the JNDI name and class name of the real #
# DataSource here #
# #
# Values set in this item cannot be reloaded using the #
# reloadproperties variable. Once it is loaded, it remains #
# in memory until the application is restarted. #
# #
#################################################################
#realdatasource=/RealMySqlDS
#realdatasourceclass=com.mysql.jdbc.jdbc2.optional.MysqlDataSource

#################################################################
# DataSource properties #
# #
# If you are using the DataSource support to intercept calls #
# to a DataSource that requires properties for proper setup, #
# define those properties here. Use name value pairs, separate #
# the name and value with a semicolon, and separate the #
# pairs with commas. #
# #
# The example shown here is for mysql #
# #
#################################################################
#realdatasourceproperties=port;3306,serverName;ibmhost,databaseName;mydb


#################################################################
# JNDI DataSource lookup #
# #
# If you are using the DataSource support outside of an app #
# server, you will probably need to define the JNDI Context #
# environment. #
# #
# If the P6Spy code will be executing inside an app server then #
# do not use these properties, and the DataSource lookup will #
# use the naming context defined by the app server. #
# #
# The two standard elements of the naming environment are #
# jndicontextfactory and jndicontextproviderurl. If you need #
# additional elements, use the jndicontextcustom property. #
# You can define multiple properties in jndicontextcustom, #
# in name value pairs. Separate the name and value with a #
# semicolon, and separate the pairs with commas. #
# #
# The example shown here is for a standalone program running on #
# a machine that is also running JBoss, so the JDNI context #
# is configured for JBoss (3.0.4). #
# #
#################################################################
#jndicontextfactory=org.jnp.interfaces.NamingContextFactory
#jndicontextproviderurl=localhost:1099
#jndicontextcustom=java.naming.factory.url.pkgs;org.jboss.nameing:org.jnp.interfaces

#jndicontextfactory=com.ibm.websphere.naming.WsnInitialContextFactory
#jndicontextproviderurl=iiop://localhost:900
Then copy the driver

Tomcat

cp ~/liferay/ext/lib/development/p6spy.jar ~/liferay/tomcat-6.0.18/lib/ext
Jboss

cp ~/liferay/ext/lib/development/p6spy.jar ~/liferay/jboss-tomcat-5.0.0/server/default/lib/
Create the liferay datasources file

Tomcat
 
Edit file ~/liferay/tomcat-6.0.18/conf/context.xml


<Context>
...
<Resource name="jdbc/LiferayPool" auth="Container" type="javax.sql.DataSource" driverClassName="com.p6spy.engine.spy.P6SpyDriver" url="jdbc:mysql://localhost:3306/lportal?useUnicode=true&amp;characterEncoding=UTF-8&amp;useFastDateParsing=false" username="root" password="root" maxActive="100" maxIdle="30" maxWait="10000" />
...
Jboss

Delete ~/liferay/jboss-tomcat-5.0.0/server/default/deploy/hsqldb-ds.xml.
Edit file ~/liferay/jboss-tomcat-5.0.0/server/default/deploy/liferay-ds.xml

<?xml version="1.0" encoding="UTF-8"?>
<datasources>

    <local-tx-datasource>
        <jndi-name>jdbc/LiferayPool</jndi-name>
        <use-java-context>false</use-java-context>
             <connection-url>jdbc:mysql://localhost:3306/lportal?useUnicode=true&amp;characterEncoding=UTF-8&amp;useFastDateParsing=false</connection-url>

        <driver-class>com.p6spy.engine.spy.P6SpyDriver</driver-class>

        <user-name>root</user-name>
        <password>root</password>
        <min-pool-size>5</min-pool-size>
    </local-tx-datasource>
    <local-tx-datasource>
        <jndi-name>DefaultDS</jndi-name>
        <connection-url>jdbc:mysql://localhost:3306/lportal?useUnicode=true&amp;characterEncoding=UTF-8&amp;useFastDateParsing=false</connection-url>
        <driver-class>com.p6spy.engine.spy.P6SpyDriver</driver-class>
        <user-name>root</user-name>
        <password>root</password>
        <metadata>
          <type-mapping>mySQL</type-mapping>
        </metadata>
    </local-tx-datasource>
</datasources>

Modify/Create ~/liferay/ext/ext-impl/src/portal-developer.properties


jdbc.default.jndi.name=jdbc/LiferayPool
Stop server
Deploy using "ant deploy" from ~/liferay/ext
Start server and confirm file "spy.log" appears whenever the server was run from. I commonly enter the home directory and from there start the server. That will start the server from the user home directory. We can then look into the queries from spy.log file, created in the home directory:


tail -f ~/spy.log
4. Customize liferay copying any property from ~/liferay/portal/portal-impl/src/portal.properties and pasting it into ~/liferay/ext/ext-impl/classes/portal-ext.properties. For example to remove the captcha for new registrations use this property:


captcha.max.challenges=0
Below is an important one that removes the chat portlet. You better do it ASAP as it hits performance in your local environment


layout.static.portlets.all=
Run "ant deploy-properties" from ~/liferay/ext and restart the server (Changes through the extension environment require a server restart). Try opening an account to verify the captcha is not there. Another file that can be included to modify default Liferay behaviors is system.properties.

5. Be sure you have now a user account you have created through registration "create account". Do not forget to change the automatic generated password from "my account"

6. There is an ant task deploy-fast to deploy anything below ext-web/docroot into the app server however only tomcat comes predefined there.

JBoss

Go ahead and add the below for that ant task at /Users/nurquiza/liferay/ext/ext-impl/build-parent.xml

        <if>
            <equals arg1="${app.server.type}" arg2="jboss-tomcat" />
            <then>
                <copy todir="${app.server.portal.dir}">
                    <fileset dir="docroot">
                        <include name="**/*.css" />
                        <include name="**/*.dtd" />
                        <include name="**/*.gif" />
                        <include name="**/*.jar" />
                        <include name="**/*.jpg" />
                        <include name="**/*.js" />
                        <include name="**/*.jsp" />
                        <include name="**/*.jspf" />
                        <include name="**/*.png" />
                        <include name="**/*.properties" />
                        <include name="**/*.tld" />
                        <include name="**/*.vm" />
                        <include name="**/*.xml" />
                    </fileset>
                </copy>
            </then>
        </if>




6. From Eclipse create a new java project from existing source pointing to the "ext" folder.
Note: I do not like that much running the server inside eclipse, I prefer just to connect directly to the server debug port.

Tomcat

Add the below line to ~/liferay/tomcat-6.0.18/bin/setenv.sh


JAVA_OPTS="$JAVA_OPTS -Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=n"
JBoss

Uncomment the below line from /Users/nurquiza/liferay/jboss-tomcat-5.0.0/bin/run.conf


JAVA_OPTS="$JAVA_OPTS -Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=n"

You will need to restart the server for it to start listening on debug port = 8787
From Eclipse go to Menu|Run|Debug Configuration then right click on "Remote Java Application" and select "New". Select as name "local-liferay" for example. Select port 8787 and then hit "Apply". You should be able to see all current running in the server threads from Menu|Window|Open Perspective|Debug.

7. Create a new java project using plugin folder.

8. Let us customize an existing core portlet. Note that many core portlet use just JSP and helper Java code which does not really respect the MVC pattern. For example the "Dictionary" portlet View code can be seen from the portal source code at ~/liferay/portal/portal-web/docroot/html/portlet/dictionary. Let us copy view.jsp to ~/liferay/ext/ext-web/docroot/html/portlet/dictionary


cp ~/liferay/portal/portal-web/docroot/html/portlet/dictionary/view.jsp ~/liferay/ext/ext-web/docroot/html/portlet/dictionary/
Edit the JSP, run "ant deploy-fast" from ~/liferay/ext/ext-web. Login and confirm changes are there.

9. Plugin SDK Development can be used to develop 5 kind of plugins. They are themes, layouts, portlets, hooks and web plugins. The first two I already introduced in the first tutorial and now I will show how to develop a simple JSP portlet:


cd ~liferay/plugin/portlets
./create.sh plugin_sdk_sample_1 "My first plugin SDK Portlet"
cd plugin_sdk_sample_1-portlet/
ant deploy
At this point portlet "My first plugin SDK Portlet" can be included in any page fo the portal using "Add Application" from the Liferay Dock. The new portlet can be customized changing the view.jsp and the JSPPortlet.java
To use Eclipse you will need to update the "Java Build Path" to point to the src folder of the new plugin and any new plugin you want to build/debug from Eclipse.

10. Extension Development is used to develop and extend existing portlets. Even though the recommended way to develop portlets is using plugin SDK the reality is it has limitations that the extension environment does not have. Let us create a simple JSP portlet called ext_sample_1.
Edit /Users/nurquiza/liferay/ext/ext-web/docroot/WEB-INF/portlet-ext.xml

<portlet>
        <portlet-name>ext_sample_1</portlet-name>
        <display-name>Extension Sample Portlet # 1</display-name>
        <portlet-class>com.liferay.util.bridges.jsp.JSPPortlet</portlet-class>
        <init-param>
            <name>view-jsp</name>
            <value>/html/portlet/ext/ext_sample_1/view.jsp</value>
        </init-param>
        <expiration-cache>0</expiration-cache>
        <supports>
            <mime-type>text/html</mime-type>
        </supports>
        <resource-bundle>com.liferay.portlet.StrutsResourceBundle</resource-bundle>
        <security-role-ref>
            <role-name>power-user</role-name>
        </security-role-ref>
        <security-role-ref>
            <role-name>user</role-name>
        </security-role-ref>
  </portlet>




Edit ~/liferay/ext/ext-web/docroot/WEB-INF/liferay-portlet-ext.xml


<portlet>
   <portlet-name>ext_sample_1</portlet-name>
</portlet>


Edit ~/liferay/ext/ext-web/docroot/WEB-INF/liferay-display.xml


<category name="category.example">
  <portlet id="ext_sample_1" />
</category>



Add file ~/liferay/ext/ext-web/docroot/html/portlet/ext/ext_sample_1/view.jsp with content:


<h1>Hello Liferay extension environment!</h1>

Edit /Users/nurquiza/liferay/ext/ext-impl/src/content/Language-ext.properties


javax.portlet.title.ext_sample_1=Extension Sample Portlet # 1
category.example=Tutorial
Let us deploy the portlet:


cd ~/liferay/ext/
ant deploy
You could also use "ant deploy" from ext-web and from ext-impl.
this takes time. In my local machine 3567 files are copied into ROOT.war and on top of that the server needs to be restarted.
Select "Add application" from Liferay dock. Portlet "Extension Sample Portlet # 1" should show up below "Tutorial" category
11. Let us create now a Struts portlet. Let us call it ext_sample_2:

Edit /Users/nurquiza/liferay/ext/ext-web/docroot/WEB-INF/portlet-ext.xml

<portlet>
                <portlet-name>ext_sample_2</portlet-name>
                <display-name>Extension Sample Portlet # 2 (Struts)</display-name>
                <portlet-class>com.liferay.portlet.StrutsPortlet</portlet-class>
                <init-param>
                    <name>view-action</name>
                    <value>/ext/ext_sample_2/view</value>
                </init-param>
                <expiration-cache>0</expiration-cache>
                <supports>
                    <mime-type>text/html</mime-type>
                </supports>
                <resource-bundle>com.liferay.portlet.StrutsResourceBundle</resource-bundle>
                <security-role-ref>
                    <role-name>power-user</role-name>
                </security-role-ref>
                <security-role-ref>
                    <role-name>user</role-name>
                </security-role-ref>
</portlet>




Edit ~/liferay/ext/ext-web/docroot/WEB-INF/liferay-portlet-ext.xml


<portlet>
   <portlet-name>ext_sample_2</portlet-name>
   <struts-path>ext/ext_sample_2</struts-path>
   <use-default-template>false</use-default-template>
</portlet>


Edit ~/liferay/ext/ext-web/docroot/WEB-INF/liferay-display.xml


<category name="category.tutorial">
  <portlet id="ext_sample_2" />
</category>



Edit ~/liferay/ext/ext-web/docroot/WEB-INF/struts-config.xml


<action path="/ext/ext_sample_2/view" forward="portlet.ext.ext_sample_2.view" />


Edit ~/liferay/ext/ext-web/docroot/WEB-INF/tiles-defs.xml


<definition name="portlet.ext.ext_sample_2" extends="portlet" />
<definition name="portlet.ext.ext_sample_2.view" extends="portlet.ext.ext_sample_2">
  <put name="portlet_content" value="/portlet/ext/ext_sample_2/view.jsp" />
</definition>


Create file ~/liferay/ext/ext-web/docroot/html/portlet/ext/ext_sample_2/init.jsp


<%@ include file="/html/portlet/init.jsp" %>
Create file ~/liferay/ext/ext-web/docroot/html/portlet/ext/ext_sample_2/view.jsp


<%@ include file="/html/portlet/ext/ext_sample_2/init.jsp" %>
<h1>Hello Liferay extension environment (From Struts)!</h1>

Edit /Users/nurquiza/liferay/ext/ext-impl/src/content/Language-ext.properties


javax.portlet.title.ext_sample_2=Extension Sample Portlet # 2 (Struts)
category.tutorial=Tutorial
Let us deploy the portlet:


cd ~/liferay/ext/
ant deploy
12. Any further development you do you will need the portal source so go ahead and download liferay-portal-src-5.2.3.zip and uncompress it to ~/liferay/portal, then create a Java project in eclipse pointing to that folder. All liferay sources are there so you can debug your projects even navigating through the core implementation. It is not a bad idea to point directly to the official SVN TAG repository so you are aware right away if the portal source is locally modified by accident. Remember, all changes must be done from the extension environment and never from the portal source.

13. Existing plugin projects can be hosted in their own project SVN paths, you just need to check those projects into the proper plugin-sdk folder. Do not keep all binary files on SVN. I still do not understand why Liferay does not support maven out of the box.

14. Extension projects most likely will be composed of subfolders of ext-impl, ext-service and ext-web. Try keeping those folders *only* on SVN to minimize download time per project. It is again a very bad idea to host binary files on SVN, especially JAR files. Maven should come to our rescue for this matter.

Thursday, October 08, 2009

Liferay For Designers

This is a tutorial for Web Designers in charge of developing Layouts and Themes for Liferay Portal.

To develop layouts or themes a local liferay instance is necessary. In addition we will need to install the extension development environment and the plugin SDK.

There is a lot of command prompt usage but you just need to copy and paste if you install the bundle I am using.

I am showing how to configure the jboss and tomcat bundles. I recommend using Tomcat (better and faster option).

I am using MAC OSX this time.

Tomcat

1. Set variables while editing file ~/.profile:

JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK/Home
PATH=$JAVA_HOME:$PATH

2. download bundle liferay-portal-tomcat-6.0-5.2.3.zip and uncompress as ~/liferay


3. download liferay-plugins-sdk-5.2.3.zip and uncompress as ~/liferay/plugin


4. download liferay-portal-ext-5.2.3.zip and uncompress as ~/liferay/ext


5. Customize the ext environment

echo 'app.server.parent.dir=${project.dir}/..' > ~/liferay/ext/app.server.`whoami`.properties

Deploy the extension environment into the application server:

cd ~/liferay/ext
ant deploy

6. Customize the plugin environment

echo 'app.server.dir=${project.dir}/../tomcat-6.0.18' > ~/liferay/plugin/build.`whoami`.properties
chmod +x ~/liferay/plugin/themes/create.sh

Run "ant deploy" from plugin folder.
7. Be sure the server is not running and delete the sample application data. If you run the server before you will need to drop the lportal db and recreate it as step 9 shows.

rm -fR ~/liferay/tomcat-6.0.18/webapps/sevencogs*
rm -fR ~/liferay/tomcat-6.0.18/webapps/sevencogswol*
rm -fR ~/liferay/tomcat-6.0.18/webapps/wol*

8. Install mysql/ from the .dmg image including "mysql start item", start the server:

sudo /Library/StartupItems/MySQLCOM/MySQLCOM start

9. Create the liferay DB:

/usr/local/mysql/bin/mysql -u root
drop database lportal;
create database lportal character set utf8;
UPDATE mysql.user SET Password=PASSWORD('root') WHERE User='root';
flush privileges;

10. Point the local server to the newly created mysql database. For that create/edit ~/liferay/ext/ext-impl/src/portal-ext.properties

jdbc.default.driverClassName=com.mysql.jdbc.Driver
jdbc.default.url=jdbc:mysql://localhost/lportal?useUnicode=true&characterEncoding=UTF-8&useFastDateParsing=false
jdbc.default.username=root
jdbc.default.password=root

Run "ant deploy-properties" from ext/ folder. At this point ~/liferay/tomcat-6.0.18/webapps/ROOT/WEB-INF/classes/portal-ext.properties will be a copy of the original. Do not edit this file, you must change the one from ext/ext-impl and then run "ant deploy-properties".
11. Run the server. In my case:

~/liferay/tomcat-6.0.18/bin/startup.sh

To stop the server use:

~/liferay/tomcat-6.0.18/bin/shutdown.sh

12. At this time you should be able to see all tables in local DB, to confirm that, run from mysql prompt:

/usr/local/mysql/bin/mysql -u root -proot
use lportal;
show tables;

13. After a while liferay server will finish creating all tables and will open a browser window showing up the welcome page of the portal. Use test@liferay.com / test for credential. You can check the server log using:

tail -f ~/liferay/tomcat-6.0.18/logs/catalina.out



14. Layout development: Follow my tutorial to create a custom layout.


15. Let us say you want to start from an existent theme. I am starting let us say from the "transparentia" theme. You download the theme WAR and uncompress it as "transparentia-theme". Then:

cd ~/liferay/plugin/themes
./create.sh my-project "My Project"
cp -r ~/Downloads/transparentia-theme/css ~/liferay/plugin/themes/my-project-theme/docroot/_diffs/
cp -r ~/Downloads/transparentia-theme/images ~/liferay/plugin/themes/my-project-theme/docroot/_diffs/
cp -r ~/Downloads/transparentia-theme/javascript ~/liferay/plugin/themes/my-project-theme/docroot/_diffs/
cp -r ~/Downloads/transparentia-theme/templates ~/liferay/plugin/themes/my-project-theme/docroot/_diffs/

You go ahead and edit docroot/WEB-INF/liferay-plugin-package.properties with your name as author etc
Deploy your theme:

ant deploy

16. Login to the Liferay local server and select "Manage Pages", click on "Look An Feel" and select the new created theme. See how the creative has changed. Now the theme provides exactly the same look and feel as the transparentia theme.


17. Do some customizations in docroot/_diffs/ (Note customizations are provided always from _diffs folder and never in the files below docroot), for example in css/custom.css you can change the body background color. Deploy again and check changes have been applied.


18. At least for this environment as far as I can tell changes cannot include comments inside any selector or they will be ignored. For example:

.portlet-title img {
display: none;
}

Commenting the display style will not result in the expected outcome. Just remove the class instead. Remember, you just add in _diffs those bits you need and nothing more. At least for this bundle the plugin appears to be unregistered and registered a couple of times which kind of delay getting the final result.


19. Go ahead and create a theme from scratch

cd ~/liferay/plugin/themes
./create.sh my-project2 "My Project 2"

and pick up css, javascript, images and templates folders from ~/liferay/tomcat-6.0.18/webapps/ROOT/html/themes/classic. That is the application server path for the classic Liferay theme.

cp -r ~/liferay/tomcat-6.0.18/webapps/ROOT/html/themes/classic/* ~/liferay/plugin/themes/my-project2-theme/docroot/_diffs/

Deploy the theme and test it is working.

20. Let us try to be a little more agile now. Open file ~/liferay/tomcat-6.0.18/bin/setenv.sh and add the below line at the end

JAVA_OPTS="-Dexternal-properties=portal-developer.properties $JAVA_OPTS"

Take a look at all settings that will take care after you restart the server from file ~/liferay/tomcat-6.0.18/webapps/ROOT/WEB-INF/classes/portal-developer.properties
For the changes to take effect restart the server.

21. Restart the server and make a modification in one of the files from _diffs folder, then copy the file straight to the application server, for example:

cp ~/liferay/plugin/themes/my-project2-theme/docroot/_diffs/css/custom.css ~/liferay/tomcat-6.0.18/webapps/my-project2-theme/css/


JBoss

1. Set variables while editing file ~/.profile:

JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK/Home
PATH=$JAVA_HOME:$PATH

2. download bundle liferay-portal-jboss-tomcat-5.0-5.2.3.zip and uncompress as ~/liferay


3. download liferay-plugins-sdk-5.2.3.zip and uncompress as ~/liferay/plugin


4. download liferay-portal-ext-5.2.3.zip and uncompress as ~/liferay/ext


5. Customize the ext environment

cd ~/liferay/ext
vi app.server.`whoami`.properties

Then use proper values for your local environment:

app.server.type=jboss-tomcat
app.server.parent.dir=${project.dir}/..

app.server.version=5.0
app.server.dir=${app.server.parent.dir}/jboss-tomcat-5.0.0
app.server.classes.global.dir=${app.server.instance.dir}/lib
app.server.instance.dir=${app.server.dir}/server/default
app.server.lib.global.dir=${app.server.instance.dir}/lib/ext

app.server.jboss-tomcat.version=5.0
app.server.jboss-tomcat.dir=${app.server.parent.dir}/jboss-tomcat-5.0.0
app.server.jboss-tomcat.classes.global.dir=${app.server.jboss-tomcat.instance.dir}/lib
app.server.jboss-tomcat.instance.dir=${app.server.jboss-tomcat.dir}/server/default
app.server.jboss-tomcat.lib.global.dir=${app.server.jboss-tomcat.instance.dir}/lib

Deploy the extension environment into the application server:

cd ~/liferay/ext
ant deploy

6. Customize the plugin environment

vi plugin/build.`whoami`.properties

Then use proper values for your local environment

app.server.dir=${project.dir}/../jboss-tomcat-5.0.0
app.server.lib.global.dir=${app.server.dir}/server/default/lib/ext
app.server.portal.dir=${app.server.dir}/server/default/deploy/ROOT.war
app.server.classes.portal.dir=${app.server.portal.dir}/WEB-INF/classes
app.server.lib.portal.dir=${app.server.portal.dir}/WEB-INF/lib

Run "ant deploy" from plugin folder.
7. Be sure the server is not running and delete the sample application data. If you run the server before you will need to drop the lportal db and recreate it as step 9 shows.

rm -fR ~/liferay/jboss-tomcat-5.0.0/server/default/deploy/sevencogs*
rm -fR ~/liferay/jboss-tomcat-5.0.0/server/default/deploy/sevencogswol*
rm -fR ~/liferay/jboss-tomcat-5.0.0/server/default/data/hypersonic/*
rm -fR ~/liferay/jboss-tomcat-5.0.0/server/default/deploy/wol*
rm -fR ~/liferay/jboss-tomcat-5.0.0//server/default/work/jboss.web/localhost/*

8. Install mysql/ from the .dmg image including "mysql start item", start the server:

sudo /Library/StartupItems/MySQLCOM/MySQLCOM start

9. Create the liferay DB:

/usr/local/mysql/bin/mysql -u root
drop database lportal;
create database lportal character set utf8;
UPDATE mysql.user SET Password=PASSWORD('root') WHERE User='root';

10. Point the local server to the newly created mysql database. For that create/edit ~/liferay/jboss-tomcat-5.0.0/server/default/deploy/ROOT.war/WEB-INF/classes/portal-ext.properties

jdbc.default.driverClassName=com.mysql.jdbc.Driver
jdbc.default.url=jdbc:mysql://localhost/lportal?useUnicode=true&characterEncoding=UTF-8&useFastDateParsing=false
jdbc.default.username=root
jdbc.default.password=root

11. Run the server. In my case:

~/liferay/jboss-tomcat-5.0.0/bin/run.sh

12. At this time you should be able to see all tables in local DB, to confirm that, run from mysql prompt:

use lportal;
show tables;

13. After a while liferay server will finish creating all tables and will open a browser window showing up the welcome page of the portal. Use test@liferay.com / test for credential.


14. Layout development: Follow my tutorial to create a custom layout.


15. Let us say you want to start from an existent theme. I am starting let us say from the "transparentia" theme. You download the theme WAR and uncompress it as "transparentia-theme". Then:

cd ~/liferay/plugin/themes
./create.sh my-project "My Project"
cp -r ~/Downloads/transparentia-theme/css ~/liferay/plugin/themes/my-project-theme/docroot/_diffs/
cp -r ~/Downloads/transparentia-theme/images ~/liferay/plugin/themes/my-project-theme/docroot/_diffs/
cp -r ~/Downloads/transparentia-theme/javascript ~/liferay/plugin/themes/my-project-theme/docroot/_diffs/
cp -r ~/Downloads/transparentia-theme/templates ~/liferay/plugin/themes/my-project-theme/docroot/_diffs/

You go ahead and edit docroot/WEB-INF/liferay-plugin-package.properties with your name as author etc
Deploy your theme:

ant deploy

16. Login to the Liferay local server and select "Manage Pages", click on "Look An Feel" and select the new created theme. See how the creative has changed. Now the theme provides exactly the same look and feel as the transparentia theme.


17. Do some customizations in docroot/_diffs/ (Note customizations are provided always from _diffs folder and never in the files below docroot), for example in css/custom.css you can change the body background color. Deploy again and check changes have been applied.


18. At least for this environment as far as I can tell changes cannot include comments inside any selector or they will be ignored. For example:

.portlet-title img {
display: none;
}

Commenting the display style will not result in the expected outcome. Just remove the class instead. Remember, you just add in _diffs those bits you need and nothing more. At least for this bundle the plugin appears to be unregistered and registered a couple of times which kind of delay getting the final result.


19. Go ahead and create a theme from scratch as explained before and pick up css, javascript, images and templates folders from ~/liferay/jboss-tomcat-5.0.0/server/default/deploy/ROOT.war/html/themes/classic. That is the application server path for the classic Liferay theme.


20. Let us try to be a little more agile now. Open file ~/liferay/jboss-tomcat-5.0.0/bin/run.sh and include the below line after one of the existing lines starting with "JAVA_OPTS=..."

JAVA_OPTS="-Dexternal-properties=portal-developer.properties $JAVA_OPTS"

Take a look at all settings that will take care after you restart the server from file ~/liferay/jboss-tomcat-5.0.0/server/default/deploy/ROOT.war/WEB-INF/classes/portal-developer.properties


21. Restart the server and make a modification in one of the files from _diffs folder, then copy the file straight to the application server, for example:

cp ~/Downloads/transparentia-theme/css/custom.css ~/liferay/jboss-tomcat-5.0.0/server/default/deploy/my-project-theme.war/css/


References:
http://www.liferay.com/web/guest/community/wiki/-/wiki/Main/Themes#section-Themes-DevelopingATheme

Wednesday, October 07, 2009

Liferay and Semantic WEB

By default most of what you get from Liferay will come full of tables still used for layout purposes.

After knowing about the existence of layouts I went ahead I gave it a try and voila I got rid of tables:

1. Assuming liferay was installed in ~/liferay:

cd ~/liferay/plugin/layouttpl
./create.sh 3-column-with-divs "3 Columns with divs"

2. Edit file ~/liferay/plugin/layouttpl/3-column-with-divs-layouttpl/docroot/3_column_with_divs.tpl

<div id="content-wrapper">
<div id="layout-grid">
<div id="first-div">
$processor.processColumn("column-1")
</div>
<div class="second-div">
$processor.processColumn("column-2")
</div>
<div class="third-div">
$processor.processColumn("column-3")
</div>
</div>
</div>


3. Install the layout:

cd ~/liferay/plugin/layouttpl/3-column-with-divs-layouttpl
ant

4. Now from the Liferay Server login with a proper account and select your newly created layout. Drop simple portlets like "Dictionary" and "Sign in".

5. Take a look at the source code to confirm no tables are doing any layout. If you get any tables they are coming from specific portlets.

Most of the back-end developers that are forced to do look and feel themselves still use tables and so do not expect many out of the box portlets with good separation of concerns even though in year 2009 it should be clear that tables are just to present tabular data.

Sunday, September 20, 2009

Maven to the rescue

I just finished a Maven Tutorial to be used from my upcoming Spring Web Flow Tutorial and I thought it is good time to talk a little bit about Maven.

I have being involved in several building process for many different languages. Specifically for Java I have used shell scripts, Ant and Maven.

At the beginning when I learned about the existence of a better tool to do my builds I went to forums trying to get opinions about Ant versus Maven. By that time the community was really divided.

Nowadays we know about Ant limitations and while Maven is not the only tool that allows to circumvent them, it is for sure one of those that solve main issues:

1. No need to store jar files on version control systems like SVN. This allows for extremely fast checkouts of projects.

2. Transitive dependency. Projects depend on jars that at the same time depends on others. Once you declare a dependency on a given jar you need not to worry about those dependencies that result from the new inclusion.

3. Version headaches. It is common to include a jar file and then dependencies of that jar file. It is very easy to get wrong on the versions of the dependencies of the included jar. Maven will pick the right dependency versions for you.

4. Maven is actually a project management tool and not just a build tool. There are hundreds of plugins available going from compilation to deployment issues and bringing project reports. It covers pretty much all you need from SDLC perspective.

5. Maven has extensive support and is used nowadays in sample projects embedded in the well known frameworks like Spring. It is also the min building tool in use from Jakarta and other important Open Source Software projects. Pretty much there is no big project out there that is lacking support for Maven.

6. Maven is IDE agnostic. Thanks to the plugin architecture your projects can be run from Eclise, Netbeans or IDEA. How the project is debugged and run in your local IDE is a concern of plugins and that allows to get cleaner version system repositories as there is no need to store specific to IDE files.

Mavenizing an existing project is easy. Well done Maven!

Tuesday, September 01, 2009

Wiki Tutorial

People get scared of Wiki syntax but actually with too little knowledge you can do a lot of editing. Below is what I would say is necessary and enough to edit WIKIs. At least MoinMoin and Google Code Wiki comply with this standards. So there we go:

1. Numbering
1.
1.
1.


2. Code block
{{{
}}}


3. Main header:
= Header Here =


4. Subheaders:
== Sub Header Here ==

5. Images:
attachment:image.jpg

6. Subpages index
[[FullSearch()]]

Thursday, August 27, 2009

JSON Forms

Forms in a WEB applications are always a need and as they get complicated frameworks come to your rescue. Sometimes though, you have not a framework to rely on.

JSON gives a clean solution that actually works with any language out there as it is a standard Object notation supported by APIs in all modern languages used in WEB programming. After all building a JSON parser is not that hard.

Here are simple steps to come up with your own JSON Form API:
1. Create a JSON Form Array per WEB Form listing the necessary form fields. Below is a prepopulated Person JSON Form:

[{"name":"first_name","minlength":"6","maxlength":"50","mandatory":true,"editable":true,"translation":"First Name","value":"","selected":"Foo"},{"name":"last_name","minlength":"6","maxlength":"50","mandatory":true,"editable":true,"translation":"Last Name","value":"","selected":"Bar"},{"name":"email","minlength":"6","maxlength":"50","mandatory":true,"editable":true,"translation":"Email","value":"","selected":"foo.bar@fakeemail.com"},{"name":"phone","minlength":"7","maxlength":"16","mandatory":true,"editable":true,"translation":"Phone","value":"","selected":"+01 305-222-4444"},
{"name":"address1","minlength":"1","maxlength":"100","mandatory":true,"editable":true,"translation":"Address 1","value":"","selected":"1509 Alton Road"},{"name":"address2","minlength":"1","maxlength":"100","mandatory":false,"editable":true,"translation":"Address 2","value":"","selected":"Suite 302"},{"name":"city","minlength":"3","maxlength":"50","mandatory":true,"editable":true,"translation":"City","value":"","selected":"Miami Beach"},{"name":"state","minlength":"1","maxlength":"50","mandatory":true,"editable":true,"translation":"State","value":"","selected":"FL"},{"name":"zip","minlength":"3","maxlength":"20","mandatory":true,"editable":true,"translation":"Zip","value":"","selected":"33139"},{"name":"country","minlength":"3","maxlength":"3","mandatory":true,"editable":false,"translation":"Country","value":["USA", "Canada"],"selected":"USA"},{"name":"age","minlength":"1","maxlength":"3","mandatory":true,"editable":true,"translation":"Product","value":"","type":"int","selected":"33","min":"21","max":"100"},{"name":"graduation_date","minlength":"6","maxlength":"10","mandatory":true,"editable":true,"translation":"Graduation Date","value":"","type":"date","selected":"1999-09-05","format":"yyyy-MM-dd"},{"name":"comments","minlength":"0","maxlength":"255","mandatory":false,"editable":true,"translation":"Comments","value":"","selected":""}]
{"name":"languages","minlength":"0","maxlength":"255","mandatory":false,"editable":true,"translation":"Languages","value":["English","Spanish","Russian"],"selected":"English","mutex":"false"}]


2. First time the form could be rendered with data using the "selected" field, others the form will come empty meaning "selected" will have no value. We can use "value" for a list of possible options, "mutex" to decide if we will accept multiple values, "format" for date types (we could actually change the format with the locale of the user), and so on.

3. When the form is submitted, pass the parameters to a Factory Form class that will append a validation error field (Array of errors) to the JSON Object. Things like "incorrect_length", "incorrect_type", "bigger number needed", "mandatory" are typical. On the other hand following conventions if the name of the field contains "email" then use regex to match if the email should be accepted or just use a library (Jakarta Commons Validator for Java offers an extensive amount of special data types). The same applies to zip codes and phones for example but commonly those follow special rules per country.

4. If special values are to be inserted from specific business rules do so in the JSON Object.

5. If there are validation errors render back the JSON Form explaining what the errors are (JQuery is great for this especially when using existing form plugins). Notice that with the same JSON Object you can provide both front end and middle tear validation. Ideally you should have a script that runs everytime the involved Database metadata changes, so the JSON Object is actually regenerated from time to time. That keeps the third tear (Backend) synchronized as well.

5. If there are no errors, proceed to persist the JSON Form Data in the Database. Again the persistence logic could be optionally provided on the fly using ORM techniques or even including a pre-populated by script Class that changes every time the model changes as well.

Wednesday, July 22, 2009

Mac ZIP / UNZIP, Microsoft Word 2007 and OpenOffice

Open Office reads Microsoft Word 2007 documents out of the box. Mac OSX comes with an unzip utility out of the box so it would make sense you receive on your MAC a zipped Word 2007 document and you will read it without any problems.

The reality is that at least in my latest Mac Book Pro (2009) when I double click the zip file it gets extracted in a folder full of XML files which Open Office will not recognize.

Of course from command prompt your old friend "unzip" will do nothing more than extracting your file but Finder will not. Somehow it is so smart that it will decompress the doc file as well (Yes the .doc you get from Microsoft 2007 is no more than a zipped bundle of XML files and more)

Now, if you are on a MAC you love GUI ;-) and so allow me to recommend http://www.edenwaith.com/products/guitar/ for uncompressing. It is a wrapper of command line utilities that do not go any smarter than what you need.

Wednesday, June 24, 2009

Windows Mobile, Outlook/Exchange and Google Contacts/Calendar Synchronization

There is no need to pay for any service or program to keep your Google calendar and contacts synchronized with those in your pocket-pc and your Windows Desktop OS.

1. Visit "Google Help › Mobile Help › Using Your Device › Windows Mobile Devices › Google Sync" and get instructions to set up your mobile device.
2. Visit "Google Help › Google Calendar Help › Access options › Sync › Sync with Microsoft Outlook › Google Calendar Sync: Overview" to download and install in your Windows Desktop OS "Google Calendar Sync"
3. Calendars are synchronized between all parts without problems. Contacts though is a different story. For contacts you will need to synchronize from the mobile device to Google and your Exchange Corporate Server, that way your ActiveSync will necessarily be the master as of the time of this writing not direct sync between Google and Exchange is still available.

From now on you can manage your contacts or calendar from either Microsoft Outlook, your mobile phone or Google Web interface.

Wednesday, March 25, 2009

Submitting Iphone Application to iTunes

Well this proved to be a little bit complicated but it shouldn't be if you carefully follow at least one time all the steps from IPhone developer Program ("Distribution|Prepare App" has all necessary instructions). Anyway here are some important and very basic steps to follow.

1. In XCode navigate to Products/${appname}.app. CTRL + click on top of it and select " Reveal in Finder"
2. In the Finder Window use CTRL + click on top of ${appname}.app and select "compress"
3. Open App Loader and select Menu | File | New and choose your application
4. Follow steps and point to the binary (zip created above)
5. Press Send.



The major problem I got was "application failed codesign verification" after trying to send the binary from App Loader.

Just be sure you follow careful all steps from IPhone developer Program ("Distribution|Prepare App" has them all). The error is very generic and anything you miss will stop your app from being sent to Apple. In my case I was working with the 'Project Info Window' (Menu | Project | Edit Project Settings) all the time, ignoring that many of the steps involve using the 'Target Info Window' (Menu | Project | Edit Active Target)

As a side note the application I was working on had a lot of local resources and being almost 40MB it took 15 minutes to get uploaded. It was not my Internet Connection for sure but just an Apple throughput limitation.

JQuery Tooltip for Html Select Boxes

Sometimes there is limited content width for options showing up in an HTML Select tag. Consider using in those cases JQuery Tooltip plugin:

<!-- Include jquery and tooltip plugin -->
<script src="jquery-1.2.6.js" type="text/javascript"></script>
<script src="jquery.tooltip.js" type="text/javascript"></script>

<!-- Forcing a narrow select box -->
<style type="text/css">
select{
width: 60;
}
option{
width: 60;
}
</style>

<!-- Include custom code to handle the tooltip -->
<script type="text/javascript">
//After loading the page insert the title attribute.
$(function(){
$("select option").attr( "title", "" );
$("select option").each(function(i){
this.title = this.text;
})
});

//Attach a tooltip to select elements
$("select").tooltip({
left: 25
});
</script>

<!-- When moving the mouse over the below options -->
<select>
<option>very long option text 1</option>
<option>very long option text 2</option>
<option>very long option text 3</option>
<option>very long option text 4</option>
<option>very long option text 5</option>
<option>very long option text 6</option>
</select>

Tuesday, March 10, 2009

Unicode Support in Web applications

UPDATE: While most of the points below are still true as of March 2014 the one thing you should do is to visit your whole stack for "Unicode" support. For example looking for tomcat you get this.

Regardless which programming language you use and what application server is involved, internationalization (i18n) support is about handling character encoding in a proper way. Below are guidelines to take into account for any combination of WEB Server/Language. Examples are shown for JBoss/Java though.

1. REQUEST ENCODING: Do not touch it in the server side. Do not use request.setCharacterEncoding() if you do not understand the implication. Only the client (browser) knows the supported encoding and it should be the only responsible for setting it. Most of the browsers use "ISO-8859-1" encoding when submitting data but for example using HTML FORM accept-charset attribute this encoding could be different. JBoss (4.0.4-GA at least) will not affect a GET request when request.setCharacterEncoding() is used but it will affect the POST parameters. It is clear that somehow the algorithm used to decode the query string is different than the one used for posted form data. By default, at least in my tests, request.getCharacterEncoding() returns always NULL.

2. REQUEST PARAMETERS PARSING: Browsers will encode "ISO-8859-1" even those characters not supported by that encoding. Be aware of your server limitations. A typical Servlet Container will try to interpret any request as "ISO-8859-1" and so corrupted bytes are going to end up in the String returned by ServletRequest.getParameter(). The good news is bytes can be read from that corrupted string and reencoded with the right encoding (the one you support in your application and I would say it should be 99% percent of the time UTF-8). To correct this issue wrap ServletRequest.getParameter() following http://www.adobe.com/devnet/server_archive/articles/internationalization_in_jrun.html advice. Notice however that it is assumed in that post that Browsers will always use "ISO-8859-1". A safer way is to find the encoding as below:
String requestEncoding = request.getCharacterEncoding();
if(requestEncoding == null || requestEncoding.trim().length() == 0) {
requestEncoding = "ISO-8859-1";
}


If you are using Tomcat you can avoid that all together with a a change in server.xml and the use of a CharacterEncodingFilter. These features should be available in other servers as well. So again use the below which will deal with GET requests:
<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" URIEncoding="UTF-8"/>
And the below which will deal with POST requests:
<!-- First fileter must be the Set Character Encoding Filter -->
    <filter>
        <filter-name>setCharacterFilter</filter-name>
        <filter-class>org.apache.catalina.filters.SetCharacterEncodingFilter</filter-class>
        <init-param>
            <param-name>encoding</param-name>
            <param-value>UTF-8</param-value>
            </init-param>
        <init-param>
            <param-name>ignore</param-name>
            <param-value>false</param-value>
            </init-param>
    </filter>
    <filter-mapping>
        <filter-name>setCharacterFilter</filter-name>
        <url-pattern>*</url-pattern>
    </filter-mapping>

3. PROGRAM MANIPULATION: Java has unicode support out of the box but not all classes are privileged to enjoy it. Be careful when working with Properties.class as UTF-8 is not allowed. If UTF-8 support is needed then it must be escaped in properties files. We need then to convert UTF-8 strings into ASCII Java Escaped Unicode. For example the two Asian characters string "中医" (URL Encoded version: %E4%B8%AD%E5%8C%BB) should be converted to \u00ef\u00bb\u00bf\u00e4\u00b8\u00ad\u00e5\u0152\u00bb removing all but the first "\u" before using Properties#load() . There is a project with source code available that can be used to work around
this issue (https://advancenative2asciitool.dev.java.net/). Below is a class that turns the Unicode characters into ASCII Java Escaped Unicode:
/**
* All credits for the "Advance Native2ASCII Tool" from https://advancenative2asciitool.dev.java.net/
*/

package com.bittime.util.design;

/**
* Please refer to http://en.wikipedia.org/wiki/ASCII for definition on valid/printable ASCII characters
*/
public class Unicode2ASCII {

public static String toHTML(String unicode)
{
String tc = unicode;
String output = "";

char[] ca = tc.toCharArray();
int x;

for( x = 0; x < a =" ca[x];">126){
output += "&#" + (int)a + ";" ;
}
else{
output += a ;
}
}

return output;
}

public static String toJAVA(String unicode)
{
String tc = unicode;
String output = "";

char[] ca = tc.toCharArray();
int x;

for( x = 0; x < a =" ca[x];">255){
String hexString = "0000" + Integer.toHexString((int)a);
hexString = hexString.substring(hexString.length() - 4);
output += "\\u" + hexString ;
}
else{
output += a ;
}
}

return output;
}
}


4. RESPONSE HEADERS: An HTTP header (Content-Type) is responsible to give the client (Browser) the encoding used in the response. A typical example is "Content-Type: text/html; charset=UTF-8". The response content type then must be set not only with the mime type you want but in case of any text mime type the proper charset must be specified as well.

From JSP for example:
<%@page contentType="text/html; charset=UTF-8" %>
From Java:
response.setContentType("text/html; charset=UTF-8");

There is no need to call response.setCharacterEncoding() if setContentType() is called passing both the mime type (text/html) and the encoding (charset). If response.setCharacterEncoding("UTF-8") is used then still setContentType("text/html")must be called. Refer to http://java.sun.com/javaee/5/docs/api/javax/servlet/ServletResponse.html#setCharacterEncoding(java.lang.String) for more explanation.

If you can identify if the request should be served as html in a Servlet Filter then it is a good idea to build a Unicode filter that will force the response to be UTF-8 for those requests:
public class UnicodeFilter implements Filter {


 public void doFilter(ServletRequest request, ServletResponse response,
   FilterChain chain) throws IOException, ServletException {
  response.setContentType("text/html; charset=UTF-8");
  chain.doFilter(request, response);
 }

 public void init(FilterConfig arg0) throws ServletException {
 }
 
 public void destroy() {
 }

}

5. RESPONSE CONTENT: Of course you must ensure all the body of the HTTP response uses the specified by RESPONSE HEADER encoding. This is not a big problem in Java but again there are subtle problems you need to be aware of. It is common for example to use JSP to specify markup. If the HTML included in the JSP needs non-ascii unicode characters then the encoding of the file must be the same as the one used in the responses AND as explained before an @page directive with contentType specified in it is required in absolutely all JSP pages. Add to the equation any file touched by building tools regardless if you use Maven, Ant, Make, Shell, manual, you-name-it methods. Any piece of information returned after the HTTP headers must be encoded as dictated by "Content-Type" header. So my advice is encode all your source files in UTF-8 if you need to support internationalization in your application. Finally, be sure your Server is running with default encoding set to UTF-8. Commonly this is not a problem in current versions of Linux where the default charset is precisely "UTF-8" but Windows uses "ISO-8859-1". In the case of Java Applications use the below when starting the server:
-Dfile.encoding=UTF-8
Particularly for JBoss Windows for example:
set JAVA_OPTS=-Dfile.encoding=UTF-8 %JAVA_OPTS%

You can always check from JSP the server encoding:
%=System.getProperty("file.encoding")%>

6. DEFAULT RESPONSE ENCODING: There are server specific configurations that will free the developer from setting the charset of the response. I do not recommend them since they break portability and might end up generating other problems related to internationalization.

7. BROWSER / OS language support: Firefox for example comes with support for all languages however Internet Explorer does not. Commonly WXP users running IE will need to install language support for Chinese and Thai, Font support in IE is tricky (Internet Options|General|Fonts|Latin Based;Times New Roman OR Arial;Courier New is a good selection, any other is not good for Unicode support), support for Assian languages must be selected manually (Control Panel | windows Regional and Language Options | Install files for 'East Asian Languages')

8. FILE ENCODING: If developers use Windows and Linux then default encoding in their Editors will be an issue. Linux uses UTF-8. Do not use non-ASCII characters in code if you are not able to force the use of UTF-8 from editors. If specific non ASCII code is needed for String comparison for example you can use escaped UTF-16 sequences for example in Java instead of ÀÁÂÃÄ use \u00c0\u00c1\u00c2\u00c3\u00c4

Here is an excellent Unicode table.

9. JDBC: If you are interacting with your database which is already storing UTF-8 successfully then it is time to tell your driver to use Unicode as well. Here is a typical URL for MySQL:
jdbc.url=jdbc:mysql://localhost:3306/myDB?Unicode=true&characterEncoding=UTF-8


10. FRAMEWORKS: Look for support of Unicode if you are using a specific Framework. For example in Spring org.springframework.web.filter.CharacterEncodingFilter will encode requests and responses.

Wednesday, March 04, 2009

Building Web Services Clients

When building Web Services (SOAP WS) clients in Java or any other language there are a couple of good tools to get familiar with before you drop your first line of code.

The two needed tools are a MockUp service that can be built from the WSDL of the remote web service and an HTTP Sniffer. While you can find similar tools written in a variety of languages the two that I am presenting hereare written in Java and thanks to that they run in any major OS out there by this time.

No matter what you use (DotNet client, RoR client, PHP Client, Spring-WS, AXIS2, AXIS, etc) the steps below should help you.

Prerequisites
1. SOAPUI
1. tcpmon

Use Cases
Writting use cases is important so you know every detail as to how your application should work. They are also a perfect initial documentation that even if outdated in the future at least gives other developers comming after you or even to yourself a reminder as to what the application does.

Building the Mockup Services (necessary to simulate Use Cases)
1. Start from understanding the services and ending points we will use from the remote service. In other words visit the WSDLs (for example https://domain/some/path/descriptor?wsdl)
2. Use SOAPUI to create a new project giving the WSDLs (File|New Project|Provide WSDL and a name|Check "Create Requests" and "Create MockService").
3. Locate from the Request group the remote method to invoke and double click on "Request 1". Complete the request parameters if any and click on the "Run" button
4. Right click on the root of the resulting entry on the right panel and click on "Generate MockService". Locate from the MockService group the remote method to invoke and double click on "Response1"
5. Copy the response text from the right pane of the request window into the right pane of the response window.
6. Repeat the above for any single method you want to Mock up (simulate)
7. Double click on the MockService group (or right click and select "Show MockService Editor"). Hit the "Run" button and the service will be available on port 8088
8. At this point from the Request window the address can be changed to point to the local Mock Service listening on port 8088. Hit the green button called "Submit request ...". The right pane should show up  whatever the Response window right pane contains

Using tcpmon

This is a java utility which comes with Axis (not Axis2). To run it use something like:

AXIS_LIB="/home/nurquiza/libraries/axis-1_4/lib"
CLASSPATH="$AXIS_LIB/wsdl4j-1.5.1.jar:$AXIS_LIB/axis.jar:$AXIS_LIB/jaxrpc.jar:$AXIS_LIB/commons-logging-1.0.4.jar:$AXIS_LIB/commons-discovery-0.2.jar:$AXIS_LIB/saaj.jar"
java -cp $CLASSPATH org.apache.axis.utils.tcpmon 8081 localhost 8088 &

With the commands above you get a window where you can see the TCP traffic going back and forth to port 8081. That port is proxing port 8088 meaning that requests to made to 8081 are actually forwarded to 8088.
Just go back to SOAPUI and click on any "Request 1" window address bar, pick the local address of the mockup service. Click again on the address bar and select "edit current", change the port number to 8081. After hitting the 'Run' button you can see the response on SOAPUI but you can also get the whole request and response including HTTP headers from tcpmon. Of course this is not that useful because you have all that information from SOAPUI already but when you start building your client you will not longer see your requests on SOAPUI.

Tuesday, March 03, 2009

DotNet Web Services From Axis2

I had to connect recently to some DotNet Web Services that were returning node names starting with an upper case (as a difference with Java where nodes start with lower case as Java class members do). I already posted a project about this but just wanted to conclude here.

This situation generates a problem at least forthe latest Axis2-1.4.1 which will be unable to dynamically deserialize the response (Assumming BeanUtil#deserialize() is used here). I have described this problem (http://markmail.org/message/2d2r7qlwnoeqka5m) in the mailing list (org.apache.ws.axis-dev).

For unmarshalling or deserializing XML into Java objects there are several Object-to-XML mapping (OXM) libraries: JAXB, JiBX, XMLBeans, Castor, XStream. Most of them are supported by Spring Web Services (Spring-WS) which provides a full stack to build and consume web services.

Another solution is going low level and use SAX (Simple API for XML), StAX (the streaming API for XML).

I have successfully worked around this issue using JAXB. The only reason that stopped me from using Sprint-WS was to stick to the old implementation code for the client (written using ADB stubs)

Using JAXB
1. Download JAXB and follow instructions from the site.
2. From the JAXB lib folder generate the classes from schema like in:
java -jar jaxb-xjc.jar -p com.nestorurquiza.axis2.samople.client.jaxb "https://mydomain/some/path/serviceShema.xsd" -d /path/to/project/client/jaxb/
3. You end up with generated classes that can be used from the client code to unmarshal the SOAP service response into Java objects. Below is a snippet as to how to get it working starting from an OMElement (Refer to my previous tutorial about Axis2 Dynamic Web Services). We are marshalling 'SampleObject' below. Note that SampleObject' is actually a class generated from the schema as explained before:
import org.apache.axiom.om.OMElement;
import javax.xml.bind.JAXBContext;
import javax.xml.bind.JAXBElement;
import javax.xml.bind.Unmarshaller;
import java.io.Reader;
import java.io.StringReader;
import javax.xml.transform.Source;
import javax.xml.transform.stream.StreamSource;
import com.nestorurquiza.axis2.sample.client.jaxb.SimpleObject;

...
//AXIOM stuff ...
OMElement result = sender.sendReceive(sampleObjectPayload);
OMElement firstElement = result.getFirstElement();
//The data element is a child of the first element for this example
OMElement dataElement = firstElement.getFirstElement();

//JAXB stuff ... Contextualize JAXB within the package holding the generated from schema sources
JAXBContext jc = JAXBContext.newInstance("com.nestorurquiza.axis2.sample.client.jaxb");
Unmarshaller unmarshaller = jc.createUnmarshaller();
Reader reader = new StringReader(dataElement.toString());
Source source = new StreamSource(reader);
//JAXB always unmarshals to a JAXBElement. Notice the below will work even if the XML node has a different name than the class.
//If the node and the class are named the same then just use SampleObject sampleObject = unmarshaller.unmarshal(source); instead.
JAXBElement root = unmarshaller.unmarshal(source, SampleObject.class);
SampleObject sampleObject = root.getValue();
//Let us retrieve one of the members from our 'SampleObject'. Note the use of getValue().
String sampleObjectMember = sampleObject.getSampleObjectMember().getValue();
//We can use our String member now
System.out.println("sampleObjectMember: " + sampleObjectMember);

...

Again, should the current Axis2 Axiom API work with node names starting with upper case we should have avoid all of the above JAXB code and just use:


SampleObject sampleObject = (SampleObject) BeanUtil.deserialize(SampleObject.class, dataElement, new DefaultObjectSupplier(), "SampleObject");
Dependencies
Here are the dependencies I successfully used:
        <dependency>
            <groupId>org.apache.axis2</groupId>
            <artifactId>axis2</artifactId>
            <version>1.4.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.axis2</groupId>
            <artifactId>axis2-adb</artifactId>
            <version>1.4.1</version>
        </dependency>
        <dependency>
            <groupId>com.sun.xml.ws</groupId>
            <artifactId>jaxws-tools</artifactId>
            <version>2.1.3</version>
        </dependency>

Axis2 Dynamic Binding Tutorial

My previous post was dealing with the dynamic consumption of a SOAP Web Service using Axis (JAX-RPC).

A web service can be dynamically consumed (Without the generation of static stubs) from Axis2 (JAX-WS) as well through the use of AXIOM.

I have written a tutorial explaining how to do it.

Again the advantage of going this path is just about solving issues when unexpected changes occur on the server side and there is no contract with the service provider to be informed of those changes before the release.

A simple extra complex type in the response will make the SOAP client to fail if it was generated using static stubs, hence the need for a dynamic binding solution.

Followers