Thursday, September 01, 2011

Upgrading Mediawiki

This one took a good chunk of time. More than anything the order was the important part:

  1. Install a fresh copy
  2. Modify LocalSettings.php to meet your goals
  3. Restore the previous db backup
  4. Run the below to update the tables so they work with the newest code
    $ cd /var/www/mediawiki/maintenance/
    $ php update.php
    

Monday, August 29, 2011

Transform XML with XSLT in Talend

Talend Open Studio is an excellent ETL tool that can be used beyond the typical database and CSV manipulation. XML processing for example is today all over the places in the Enterprise.

As a consequence XML transformations are a key skill for those folks doing data transformations.

Even though there are more efficient tools XSLT is a standard which is supported in Talend through the tXSLT component. You just need to provide your XML, XSL and output files and Talend will apply the transformation for you. Talend uses Saxon at the moment so you get the benefit of clear error messages when trying to build your XSL.

Of course XSLT might be a skill that even scares some people, however XSLT is not difficult at all and the more you work with it the better you get as with any other human skill. Do not try to avoid it, if you have XML to process and your ETL tool is Talend then do your homework and learn some XSL.

As there is no better way to teach than providing an example I decided to write this quick showcase that will pivot the data resulting from running an Advent Geneva RSL report (A SOAP service) which comes in the form of key value pairs into a tabular output. I will provide two responses: XML and HTML. The first is probably what you need in Talend while the second is probably what you need if you want to provide a quick add hoc HTML report page.

Here is the XML:
<?xml version='1.0' encoding='UTF-8'?>
<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:nsg="http://geneva.advent.com">
	<SOAP-ENV:Body id="_0">
		<reportResults xmlns="http://geneva.advent.com">
			<return xsi:type="nsg:reportResultsPortfolioStruct">
				<results xsi:type="nsg:reportResultsStruct">
					<portfolioName xsi:type="xsd:string">Fund1</portfolioName>
					<header xsi:type="nsg:reportResultsRecordStruct">
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">Head</name>
							<value xsi:type="xsd:string">Fund 1 Example</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">Head</name>
							<value xsi:type="xsd:string">DIVIDENDS RECEIVABLE AND PAYABLE</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">Head</name>
							<value xsi:type="xsd:string">FOR THE PERIOD INCEPTION TO July 31, 2011</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">Head</name>
							<value xsi:type="xsd:string"/>
						</field>
					</header>
					<record xsi:type="nsg:reportResultsRecordStruct">
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">CcyCode</name>
							<value xsi:type="xsd:string">BRL</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">Custodian</name>
							<value xsi:type="xsd:string">My Custodian</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">IDesc</name>
							<value xsi:type="xsd:string">My IDesc</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">TransID</name>
							<value xsi:type="xsd:string">10145834</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">TDate</name>
							<value xsi:type="xsd:string">March 15, 2011 12:00:00 am</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">SDate</name>
							<value xsi:type="xsd:string">December 31, 9999 11:59:59 pm</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">Qty</name>
							<value xsi:type="xsd:string">13528.013</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">PerShare</name>
							<value xsi:type="xsd:string">0.15100000</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">TaxRate</name>
							<value xsi:type="xsd:string"/>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">DivLocal</name>
							<value xsi:type="xsd:string">2042.73</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">TaxLocal</name>
							<value xsi:type="xsd:string">0.00</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">DivLocalNet</name>
							<value xsi:type="xsd:string">2042.73</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">DivBook</name>
							<value xsi:type="xsd:string">1225.76</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">TaxBook</name>
							<value xsi:type="xsd:string">0.00</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">DivBookNet</name>
							<value xsi:type="xsd:string">1225.76</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">UnrealFXGL</name>
							<value xsi:type="xsd:string">0.00</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">BookBal</name>
							<value xsi:type="xsd:string">1225.76</value>
						</field>
					</record>
					<record xsi:type="nsg:reportResultsRecordStruct">
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">CcyCode</name>
							<value xsi:type="xsd:string">USD</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">Custodian</name>
							<value xsi:type="xsd:string">My Custodian</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">IDesc</name>
							<value xsi:type="xsd:string">My IDesc</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">TransID</name>
							<value xsi:type="xsd:string">10756740</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">TDate</name>
							<value xsi:type="xsd:string">April 27, 2011 12:00:00 am</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">SDate</name>
							<value xsi:type="xsd:string">July 1, 2011 12:00:00 am</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">Qty</name>
							<value xsi:type="xsd:string">205212.046</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">PerShare</name>
							<value xsi:type="xsd:string">0.16918500</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">TaxRate</name>
							<value xsi:type="xsd:string"/>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">DivLocal</name>
							<value xsi:type="xsd:string">34718.80</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">TaxLocal</name>
							<value xsi:type="xsd:string">0.00</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">DivLocalNet</name>
							<value xsi:type="xsd:string">34718.80</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">DivBook</name>
							<value xsi:type="xsd:string">22153.39</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">TaxBook</name>
							<value xsi:type="xsd:string">0.00</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">DivBookNet</name>
							<value xsi:type="xsd:string">22153.39</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">UnrealFXGL</name>
							<value xsi:type="xsd:string">0.00</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">BookBal</name>
							<value xsi:type="xsd:string">23379.15</value>
						</field>
					</record>
					<addendumErrors xsi:type="nsg:reportResultsRecordStruct">
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">BegDesc</name>
							<value xsi:type="xsd:string"/>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">EndDesc</name>
							<value xsi:type="xsd:string">DIVIDENDS RECEIVABLE - CLOSING BALANCE</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">BegBal</name>
							<value xsi:type="xsd:string"/>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">EndBal</name>
							<value xsi:type="xsd:string">71643.79</value>
						</field>
					</addendumErrors>
					<addendumErrors xsi:type="nsg:reportResultsRecordStruct">
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">BegDesc</name>
							<value xsi:type="xsd:string"/>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">EndDesc</name>
							<value xsi:type="xsd:string">DIVIDENDS PAYABLE - CLOSING BALANCE</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">BegBal</name>
							<value xsi:type="xsd:string"/>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">EndBal</name>
							<value xsi:type="xsd:string">-20200.35</value>
						</field>
					</addendumErrors>
				</results>
			</return>
		</reportResults>
	</SOAP-ENV:Body>
</SOAP-ENV:Envelope>

If you open it with Firefox and you choose "View XPath" from a right click on the body of the page (contextual menu) you could try several XPATH expresions. Together with XSLT skill it comes XPATH which is just a way to address a node, an attribute or textual content in the XML. See below how I tested one of the XPATH using this tool. Pay attention to the namespace definition, I use simple letters to abbreviate more verbose prefixes.


Now that you have a quick tool for finding nodes in the XML document let us see the desired document structure. Here is a screenshot of what we would like to see in HTML:

Here is in the XML we would like to obtain for further processing in Talend:

Here is the XSL that will output HTML:
<?xml version="1.0"?>
<xsl:stylesheet version="1.0"
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns:e="http://schemas.xmlsoap.org/soap/envelope/"
  xmlns:g="http://geneva.advent.com">
    <xsl:output omit-xml-declaration="yes" indent="yes"/>

    <xsl:strip-space elements="*"/>

    <xsl:key name="kFieldNameByValue" match="/e:Envelope/e:Body/g:reportResults/g:return/g:results/g:record/g:field/g:name"
         use="."/>

    <xsl:variable name="vCols" select=
       "/e:Envelope/e:Body/g:reportResults/g:return/g:results/g:record/g:field/g:name
                [generate-id()
                =          
                 generate-id(key('kFieldNameByValue',.)[1])
                 ]"/>

    <xsl:template match="/">
             <table>
               <tr>
                 <xsl:apply-templates select="$vCols"/>
               </tr>

               <xsl:for-each select=
                 "/e:Envelope/e:Body/g:reportResults/g:return/g:results/g:record">                   
                 <tr>

                   <xsl:variable name="vPos" select="position()"/>

                   <xsl:for-each select="$vCols">
                     <td>
                       <xsl:value-of select=
                           "../../../g:record[$vPos]/g:field[g:name = current()]/g:value"/>
                     </td>
                   </xsl:for-each>

                 </tr>
              </xsl:for-each>
            </table>

    </xsl:template>

    <xsl:template match="/e:Envelope/e:Body/g:reportResults/g:return/g:results/g:record/g:field/g:name">
            <td>
              <xsl:value-of select="." />
            </td>   
    </xsl:template>
</xsl:stylesheet>

Here is the XSL that will output XML:
<?xml version="1.0"?>
<xsl:stylesheet version="1.0"
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns:e="http://schemas.xmlsoap.org/soap/envelope/"
  xmlns:g="http://geneva.advent.com" 
>
    <xsl:output omit-xml-declaration="no" indent="yes"/>

    <xsl:strip-space elements="*"/>

    <xsl:key name="kFieldNameByValue" match="/e:Envelope/e:Body/g:reportResults/g:return/g:results/g:record/g:field/g:name"
         use="."/>

    <xsl:variable name="vCols" select=
       "/e:Envelope/e:Body/g:reportResults/g:return/g:results/g:record/g:field/g:name
                [generate-id()
                =          
                 generate-id(key('kFieldNameByValue',.)[1])
                 ]"/>

    <xsl:template match="/">
	  	<xsl:element name="root">
               	<xsl:element name="header">
				  <xsl:apply-templates select="$vCols"/>
				</xsl:element>  

               <xsl:for-each select="/e:Envelope/e:Body/g:reportResults/g:return/g:results/g:record">                   
                 <xsl:element name="record">
                   <xsl:variable name="vPos" select="position()"/>
                   <xsl:for-each select="$vCols">
                       <xsl:element name="value"><xsl:value-of select="../../../g:record[$vPos]/g:field[g:name = current()]/g:value"/></xsl:element>
                   </xsl:for-each>
                 </xsl:element>
              </xsl:for-each>

        </xsl:element>
    </xsl:template>

    <xsl:template match="/e:Envelope/e:Body/g:reportResults/g:return/g:results/g:record/g:field/g:name">
            <xsl:element name="name"><xsl:value-of select="."/></xsl:element>
    </xsl:template>

</xsl:stylesheet>

A quick explanation of what the XSL code does:
1-6 Is just stating it is an XML document which uses a namespace prefixed "xsl" for the transaformation instructions, a soap namespace with prefix "e" and an Advent Geneva namespace with prefix "g". Note that I abreviated the original namespace suffixes for the last two.

7-9 How the final output should look like.

11-19 We use the Muenchian grouping that allows us to have a list of all possible column names.

21-37 XSLT is a functional language which works matching nodes and applying transformations to them. The logic can be affected using xsl:apply-template. We use xsl:element to create our custom nodes: First the headers which come out of the Muenchian key and later the records for which we again use the keys while addressing the correct record through an xsl:foreach nested loop.

39-41 The template responsible for generating the name nodes.

Saturday, August 27, 2011

Twitter RSS: Follow silently

I blog about the work that I do just because "the palest ink is better than the best memory"

I tweet about most of the stuff that I do or I am interested in just because I think for some people is just easier to get an SMS alike message with a title that makes you decide if the article might be of your interest.

I consume most of the information I am able to digest via RSS.

These three statements probably make me not a good social network user. I would love to have time to follow 2000 people, interact with them, engage in interesting conversations and so on but I don't believe that is possible for a first generation immigrant, father of three kids and with the hope to remain competitive in the fast paced World of IT.

The way I live and work push me for automation even when consuming information. RSS, Atom or whatever format for syndicated news is then the answer to my needs.

Syndicated feeds should be a mandatory option for any website or service providing information. If you are like me you will love this trick which allows you to silently consolidate in your News reader (I use Google Reader BTW) tweets from people you would like to follow.

With an example here is how you can consume my tweets from your favorite reader:

http://api.twitter.com/1/statuses/user_timeline.rss?screen_name=nestorurquiza

I make my statements public because I do care about what others think. If you consume them via RSS, email, native program or any other means that is secondary for me.

Thursday, August 25, 2011

TSQL Stored Procedure faster from SSMS than the application

No, I have no quick fix for this because SQL Server is not about magic and there is a reason why there are guys that work 100% of the time as SQL developers.

You have to ensure your queries are optimized and you know what query plan will be used when they are run. This is not about Java or .NET problems because your SQL Server Management Studio (SSMS) is "apparently" running the query 10 or 20 times faster than the application. It is indeed about your SQL code optimizations.

So if you have this problem then run the stored procedure using something like:
SET ARITHABORT OFF
EXEC sp_custom_procedure 'param1', 2, 'param3'
GO

Then run it again like:
SET ARITHABORT ON
EXEC sp_custom_procedure 'param1', 2, 'param3'
GO

You should get a slower response for the first and that will only tell you that your stored procedure is using a non efficient query plan. Now it is time to do your homework reading the previous link and make your stored procedure run with a custom well predefined fast and efficient query plan (of course as fast and efficient as you can)

Error creating bean with name 'liferayTransactionManager' defined in class path resource [META-INF/hibernate-spring.xml]

Every once in a while when I install Liferay in different servers I get something like the below:
15:39:20,176 ERROR [ContextLoader:215] Context initialization failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'transactionAdvice' defined in class path resource [META-INF/base-spring.xml]: Cannot resolve reference to bean 'liferayTransactionManager' while setting bean property 'transactionManager'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liferayTransactionManager' defined in class path resource [META-INF/hibernate-spring.xml]: Cannot resolve reference to bean 'liferayHibernateSessionFactory' while setting bean property 'sessionFactory'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liferayHibernateSessionFactory' defined in class path resource [META-INF/hibernate-spring.xml]: Invocation of init method failed; nested exception is org.hibernate.HibernateException: Hibernate Dialect must be explicitly set

This is an error that relates to a connectivity problem with your database. Be sure you see the *whole* stacktrace and look for a clue about connection/permission problems. Commonly you should be able to get it right if you follow these directions (Using MySQL here):

  1. Check what IP MySQL is listening to. Below it shows as listening in 192.168.0.6
    $ netstat -an|grep 3306
    tcp        0      0 192.168.0.6:3306        0.0.0.0:*               LISTEN     
    
  2. Try connecting with the same host and credentials Liferay is using from command line mysql client:
    $ mysql -u liferay -p -h 192.168.0.6
    Enter password: 
    ERROR 1130 (HY000): Host '192.168.0.6' is not allowed to connect to this MySQL server
    
  3. As the example above shows there is a permision problem so go ahead and correct it. In MySQL password must be specified per each granted user@host combination:
    $ mysql -u root -p
    ...
    mysql>  GRANT ALL ON lportal.* TO 'liferay'@'192.168.0.6';
    Query OK, 0 rows affected (0.00 sec)
    
  4. Review your /etc/hosts in the client to make sure if you are using a host it maps to the specified IP:
    $ vi /etc/hosts
    192.168.0.6  sqlHost
    

Tuesday, August 23, 2011

nullmailer smtp Failed 550 5.7.1 Unable to relay for user@myhost.myhost big log files

One of our servers was reporting low HDD resources and as usual I took a look at log files. Yes, again /var/log/mail.* were above 1GB, in some cases even 5GB.

This is the error that I saw in the traces:
Aug 23 21:56:33 myhost nullmailer[782]: Starting delivery: protocol: smtp host: mail.nestorurquiza.com file: 1314082502.18221
Aug 23 21:56:33 myhost nullmailer[27145]: smtp: Failed: 550 5.7.1 Unable to relay for nestorurquizaadmin@myhost.myhost
Aug 23 21:56:33 myhost nullmailer[782]: Sending failed:  Permanent error in sending the message

This was originated because in etc/hosts I had a line like:
192.168.1.23 myhost

And the SMTP server was not configured to relay to such a thing like myhost.myhost. So just changing it to an authorized domain should fix it right?
192.168.1.23 myhost.nestorurquiza.com

Well I kept getting the same error!!!

The reason was queued emails. I was getting from logs also a line like below:
Aug 23 22:26:47 myhost nullmailer[828]: Delivery complete, 42489 message(s) remain.

So deleting the queue made finally the trick:
sudo rm -fR  /var/spool/nullmailer/queue/
sudo mkdir  /var/spool/nullmailer/queue

nullmailer smtp Failed 550 5.7.1 Unable to relay for user@myhost.myhost big log files

One of our servers was reporting low HDD resources and as usual I took a look at log files. Yes, again /var/log/mail.* were above 1GB, in some cases even 5GB.

This is the error that I saw in the traces:
Aug 23 21:56:33 myhost nullmailer[782]: Starting delivery: protocol: smtp host: mail.nestorurquiza.com file: 1314082502.18221
Aug 23 21:56:33 myhost nullmailer[27145]: smtp: Failed: 550 5.7.1 Unable to relay for nestorurquizaadmin@myhost.myhost
Aug 23 21:56:33 myhost nullmailer[782]: Sending failed:  Permanent error in sending the message

This was originated because in etc/hosts I had a line like:
192.168.1.23 myhost

And the SMTP server was not configured to relay to such a thing like myhost.myhost. So just changing it to an authorized domain should fix it right?
192.168.1.23 myhost.nestorurquiza.com

Well I kept getting the same error!!!

The reason was queued emails. I was getting from logs also a line like below:
Aug 23 22:26:47 myhost nullmailer[828]: Delivery complete, 42489 message(s) remain.

So deleting the queue made finally the trick:
sudo rm -fR  /var/spool/nullmailer/queue/
sudo mkdir  /var/spool/nullmailer/queue

Followers