Saturday, December 19, 2015

Serving local files with a simple http file server built on NodeJS

Sometimes we want to share some files in a local network or we want to host quick local app demos for others to play with or we are forced to have files served by http to test tooling like it was my case this time.

I was playing with Git Patch Viewer chrome extension to see how specific patch files were shown but the extension wouldn't parse local file:// referenced resources.

I have published the most simple http file server I could build to address this issue. Probably you can share a better/shorter way?

Wednesday, December 02, 2015

On BI: Spring Tomcat Impersonation audit

The amount of intelligence you can pull from logs is unlimited. In a typical log like the below we see a session but we have no idea who is editing the employee. Furthermore the employee might be edited under impersonation.
2015-11-07 22:15:05,845 INFO [com.sample.web.filter.LoggingFilter doFilter] - 172.16.2.41 AD3A60A51885B74F2AC2B02F5BDD3AC0.node1 /employee/204187 192.168.0.43
We can easily see filtering by the sessionid if a user was impersonated and list both the real user and the impersonated user with the below awk script: We get now something like:
2015-11-07 22:15:05,845 192.168.0.43 AD3A60A51885B74F2AC2B02F5BDD3AC0.node1 impersonator@sample.com impersonated@sample.com /employee/204187
Note that if you are logging all params as part of the URL regardless if it is POST or GET you could be saving a lot of time. If you are using JSON payload most likely things will get a little more complicated as you might want to extract specific fields from the payload but overall you could extract a lot of business intelligence from logs just using the veteran awk.

Friday, November 20, 2015

Web site incorrectly rendering a font in italics at least in Windows Chrome browser

If your browser (I have seen it in Windows Chrome) incorrectly renders a specific font in italics you might be facing a corrupted font.

The first thing to do in order to resolve this issue is to determine the font being used. This is something you can find easily from chrome inspector (right click on an incorrectly rendered in italics word) which will reveal the current applied style including the font.

Once you know the font type then go to http://www.cssfontstack.com/ and select that font. Most likely it will be looking also in italics. See below how the whole page renders in italics:



This means that the particular font is corrupted as a result of a corrupted program installation or uninstallation.

To correct the issue just delete the font from C:\Windows\Fonts and bring it back again from a working machine (the file is a ttf which contains bold, italics, narrow and other combinations for example arial.ttf).

Saturday, November 14, 2015

On Protractor: End to End (e2e) tests should fail if there are javascript errors

End to End (e2e) tests should fail if there are javascript errors. They most likely won't cover all the application functionality, but many (if not most) of the regressions we encounter in production are related to javascript errors. Here is how to stop your continuous delivery pipeline if there is a single javascript error at runtime. Note that at a minimum smoke e2e tests should exist meaning that all of your views should be tested at least expecting to render html, not having unexpected logical errors and not having javascript errors. Webdriver allows us to get a hold of the browser console log. This means we can detect if there are javascript errors at runtime and make our tests fail. For more info see https://github.com/angular/protractor/blob/master/docs/faq.md#how-can-i-get-hold-of-the-browsers-console. Protractor makes our lifes easier though. All we need to do is to include the below configuration in protractor.conf.js:
...
exports.config = {
  ...
  plugins: [{
    path: 'node_modules/protractor-console-plugin',
    failOnWarning: true,
    failOnError: true
  }],
  ...
...
// To test the solution just force a JS error that does not stop your test from passing. // For example adding a simple script tag in your index.html:
  <script>
    console.undefinedMethod('will never run');
  </script>
After the test is run you get:
Plugin: /usr/local/lib/node_modules/protractor/plugins/console/index.js (teardown) Fail: Console output SEVERE: https://doma.in:port/path/ 72:17 Uncaught TypeError: console.undefinedMethod is not a function [launcher] 0 instance(s) of WebDriver still running [launcher] chrome #1 failed 1 test(s) [launcher] overall: 1 failed spec(s) [launcher] Process exited with error code 1

Tuesday, November 10, 2015

Find all javascript files except those below a directory

Very useful specially for those dealing with nodejs where we want to avoid searching for "node_modules"
find ./ -not -path "*node_modules*" -name "*.js"

Friday, October 30, 2015

Find out apache module version

It took me sometime to find out a way to determine the version number of any apache module. Basically the 'strings' command can extract the existing strings in the shared library and with grep we can use the regex "${mod_name}/[0-9]\.[0-9]" where mod_name is a variable containing the name of the module. I am omitting the real version numbers below for security reasons:
$ strings /usr/lib/apache2/modules/mod_pagespeed.so \
    | grep "mod_pagespeed/[0-9]\.[0-9]"
mod_pagespeed/w.x.y.z
$ strings /usr/lib/apache2/modules/mod_ssl.so \
    | grep "mod_ssl/[0-9]\.[0-9]"
mod_ssl/x.y.x

Wednesday, October 28, 2015

RabbitVCS needs restart in ubuntu 14.04 with lxde

It used to work out of the box with 12.04 and unity but now it needs restart after installation:
sudo add-apt-repository ppa:rabbitvcs/ppa
sudo apt-get -q -y --force-yes install rabbitvcs-core rabbitvcs-cli rabbitvcs-gedit rabbitvcs-nautilus3exit
sudo shutdown -r now
In addition in a different machine with update and upgrade it started showing up (no restart needed):
sudo apt-get -y update
sudo apt-get -y upgrade
In other cases just restarting nautilus made the trick:
killall nautilus
I even had to just start nautilus from command line to get it to work:
nautilus

Sunday, October 11, 2015

trying to overwrite ... which is also in package ... Sub-process /usr/bin/dpkg returned an error code (1)

Problem:
dpkg: error processing archive /var/cache/apt/archives/libffi-dev_3.1~rc1+r3.0.13-12_amd64.deb (--unpack):
 trying to overwrite '/usr/share/info/libffi.info.gz', which is also in package xrdp 0.9.0+master
Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
Processing triggers for doc-base (0.10.5) ...
Processing triggers for install-info (5.2.0.dfsg.1-2) ...
Errors were encountered while processing:
 /var/cache/apt/archives/libffi-dev_3.1~rc1+r3.0.13-12_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)
Solution:
$ sudo dpkg -i --force-overwrite /var/cache/apt/archives/libffi-dev_3.1~rc1+r3.0.13-12_amd64.deb

Thursday, October 08, 2015

Solution for mysql Warning: Using a password on the command line interface can be insecure

Mysql or mysqldump "Warning: Using a password on the command line interface can be insecure" should not be IMO in the stderr. The stderr is for errors and not for warnings. The correct way to handle these situations in the shell should be using a different handler (should I propose stdwarn or stdwarning?) which of course we do not currently have available.

As I have stated in Stackoverflow, the recommended way to solve this issue penalizes the ones that do the right thing to protect the ones that do the wrong thing. If the password is stored inside a script file it will not show up with ps or in any log. Putting the credentials in an external file does help the ones that would cron a command using the plain text password instead of variables which, but why helping those when they are doing the incorrect thing? In the meantime scripts that have been running for years now fail and we need to modify them just because this warning comes up in the stderr.

Since we are stuck with using the external credentials file here is a quick hack to please mysql commands. Basically we create the file on the fly:

Saturday, October 03, 2015

Ubuntu on Toshiba Satellite

It took me a while to install Ubuntu on a Toshiba Satellite originally shipping Windows 7. To completely wipe out Windows and use Ubuntu instead I followed the below steps.
  1. Turn off
  2. Insert Ubuntu DVD. In my case USB boot up wouldn't work
  3. Turn on keeping F2 key pressed (No Fn, just F2)). The BIOS/UEFI firmware screen shows up
  4. From Advanced/System Configuration select the "CSM boot" option
  5. From Main/Boot order put DVD/USB on top (before HDD)
  6. Save changes and exit
  7. Ubuntu DVD should take over allowing to wipe out everything from the hard disk and installing the new OS. If you run into problems you can try installing first using the "UEFI boot" option however you will still need to switch to "CSM boot" and reinstall Ubuntu because otherwise the OS won't come up but instead you will see the message "Reboot and Select Proper Boot Device"
  8. Linux rocks

Thursday, September 10, 2015

do-release-upgrade returning Checking for a new Ubuntu release No new release found

Not even trying to get to the development release was working:
Checking for a new Ubuntu release No new release found
Reason being:
$ cat /etc/update-manager/release-upgrades
# Default behavior for the release upgrader.

[DEFAULT]
# Default prompting behavior, valid options:
#
#  never  - Never check for a new release.
#  normal - Check to see if a new release is available.  If more than one new
#           release is found, the release upgrader will attempt to upgrade to
#           the release that immediately succeeds the currently-running
#           release.
#  lts    - Check to see if a new LTS release is available.  The upgrader
#           will attempt to upgrade to the first LTS release available after
#           the currently-running one.  Note that this option should not be
#           used if the currently-running release is not itself an LTS
#           release, since in that case the upgrader won't be able to
#           determine if a newer release is available.
prompt=never

Just replace never by normal or lts according to your needs.

Ubuntu 12.04 update fails with Unable to find expected entry 'utilities-1.4/binary-amd64/Packages'

We got the below error:
W: Failed to fetch http://repo.mysql.com/apt/ubuntu/dists/precise/Release Unable to find expected entry 'utilities-1.4/binary-amd64/Packages' in Release file (Wrong sources.list entry or malformed file) E: Some index files failed to download. They have been ignored, or old ones used instead.
To resolve it we had to install mysql-apt-config selecting the option "apply"
sudo apt-get install mysql-apt-config

Wednesday, August 05, 2015

Use 'set -x' in to debug docker multiline statements or docker invoked bash scripts

Some people new to Docker are also new to bash which is a must-have skill for *nix sysadmin. Perhaps the most important statement for bash debugging is 'set -x'. If you have multiline statement, use it to understand where exactly the whole RUN command is failing. The classical example involves entering a directory and issuing there several commands. Without 'set -x' you will be lost about why and in some cases even what specific command failed. Example:
RUN set -x \
      cd /tmp \
      ...
Another example involves calling an external script. In that case you just need to make sure the flag is included through sed, for example let us say you use the 'bash -e' in your shebang (aren't you?). To change the shebang on the fly is as easy as:
RUN sed -i 's/bash -e/bash -ex/g' /some/bash/script.sh

Wednesday, June 17, 2015

Scatter Diagrams from any two columns in Excel 2010

In Excel 2010 to plot a scatter diagram out of two columns that appear in any position and any order follow this steps:

  1. Click on an empty cell; select Menu | Insert | Scatter | Select first
  2. Right click on the chart area: click “Select Data” | Add legend entries (series) | Pick X and Y values | Pick a name for the series for example “Comparison of size and rental cost of apartments” or in general “Comparison of X and Y" | click OK
  3. From "Chart Tools | Design | Chart Layout" pick the first one (layout 1) which adds the axis labels
  4. Remove the label on the right which contains the name of the series. This is redundant as the title already states the same
  5. Click on each axis title labels to select it, then click again inside it to change it to the real name of Y and X

Thursday, May 07, 2015

Talend OutOfMemoryError: Java heap space because of many files in a directory

I have blogged in the past about how to debug OutOfMemoryError in Talend jobs.

There is at least one official Talend component that would be generating these errors when we point to a specific directory containing a really large amount of files. The reason is that some code generates an array of strings containing the file names which clearly will not scale. The way I figured this out was following the steps in that previous post. From Eclipse Memory Analyzer I saw the cause for high memory consumption was an array of strings which matched file names.

Of course it is a bad practice to use a root directory to store all files, one should use a temporary directory per run. So the solution is actually simple. Nevertheless keeping such array of strings is just a waste of resources so that should be avoided as well.

The bottom line is that just automatically increasing memory when a JVM code throws OutOfMemoryError is not an option. Instead the engineer should investigate and get to the bottom of why processes are inefficient. Failure to do so will only postpone the inevitable because simply underperforming jobs won't scale. In the case of Talend as in any java application the JVM provides the tools to understand what happened when a memory leak originated a crash.

Saturday, May 02, 2015

Fastest idempotent way to install nodejs in linux or MAC OSX

Simply run the below one-liner from my plain old bash (POB) recipe as shown below:
export NODE_VERSION=18.19.0; curl -sL https://raw.githubusercontent.com/nestoru/pob-recipes/master/common/nodejs.sh | sudo bash -s $NODE_VERSION $USER
Note that you might need to run 'npm rebuild' in your project(s) if you run into errors like:
Error: The module '/app/node_modules/node-expat/build/Release/node_expat.node'
was compiled against a different Node.js version using
NODE_MODULE_VERSION 64. This version of Node.js requires
NODE_MODULE_VERSION 83. Please try re-compiling or re-installing

Thursday, April 30, 2015

Fastest idempotent way to install nodejs in Ubuntu

Originally I created a gist tailored at Ubuntu however a fastest way is just to use a Plain Old Bash script to install the binaries as presented here. This will work in any linux and MAC OSX.

Friday, April 17, 2015

run cygwin sshd under SYSTEM user - The CYGWIN sshd service on Local Computer started and then stopped. Some services stop automatically if they are not in use by other services or programs

I got this error when trying to switch cygwin sshd to run as windows SYSTEM user. This is a need if you want to allow cygwin to interact with graphical applications.
The CYGWIN sshd service on Local Computer started and then stopped. Some services stop automatically if they are not in use by other services or programs
The first thing to do is to look into the sshd logs:
$ tail -10 /var/log/sshd.log
/var/empty must be owned by root and not group or world-writable.
The reason for this error is the fact that SYSTEM user (look upper case) is not the owner of /var/empty. The exact reason why this error happens is explained here. So the solution is simple:
$ chown SYSTEM /var/empty
Even though I went pass this issue I later realized there was no way to run desktop applications remotely using this method as explained here and here.

Monday, April 13, 2015

dockerized mediawiki 1.25 upgrade

The latest version of mediawiki comes with VisualEditor which marks the beginning of real wiki visual edition thanks to the use of node parsoid project. This is a radical change when we compare with the former CKEditor which would add a lot of html tags to your wiki markup. With Visual Editor we get plain wiki markup that can be edited by those who do enjoy the keyboard more than the mouse ;-)

Here is a partial docker guideline that can be used to install this latest version.

Friday, March 27, 2015

When my Apple MAC notes disappeared ...

Several posts on the web suggest to open the different files you see in ~/Library/Containers/com.apple.Notes/Data/Library/Notes/ renaming them to use the ".html" extension, however I was able to recover them after deleting the ".storedata-wal" and ".storedata-shm" files.

Before proceeding please make a copy of the whole Notes directory. The file(s) with extension ".storedata" is/are what you want to keep and in fact these are sqlite files so you can open them, see the schema and inspect tables with the sqlite3 command.

BTW this happened to me after I started playing with iCloud. My iPhone and MAC notes were not synchronizing anymore and after finding that the iPhone notes were the ones in iCloud I started playing with iTunes and Notes accounts' settings to end up losing my MAC notes and getting those from the iPhone locally. Worth to be said that Notes act funky in terms of synchronization and when they do going to accounts, unchecking and checking back Notes in the MAC solves the issue.

Saturday, March 14, 2015

Docker - izing real life production services in Ubuntu

I plan to publish several real life production showcases on docker usage. Instead of using the classical posts on this blog I will go this time with github documentation as I will be posting some working code in there.

If you are new to docker you might check it out as there is an initial section to get to speed with docker. If you are well versed with docker please do so as well, to get it right as usual I need your criticism.

Tuesday, March 10, 2015

host command returns correct IP but SSH still goes somewhere else!

When /etc/hosts overrides DNS strange things might happen like in this case today when someone would ssh into a remote machine but end up still in his own host.

You better run ping and host and if there are not the same then look for the domain in question inside /etc/hosts.

Friday, February 06, 2015

Setting DISPLAY variable to avoid No protocol specified Error: Can't open display: :# Selenium UnknownError: unknown error: Chrome failed to start: exited abnormally

If Selenium server cannot open the current display the client will fail with an error similar to "UnknownError: unknown error: Chrome failed to start: exited abnormally". This is a common issue many people ask about when they try to either run X applications (like tests against selenium in a remote server) remotely or when they are logged in via VNC, XRDP etc. Basically the DISPLAY variable is assigned to the local terminal console and not really to the remote session. You need to find the proper DISPLAY variable for your remote session in order to resolve this problem.

Setting DISPLAY variable will avoid the typical problem "No protocol specified Error: Can't open display: :#" where # is usually 0.

If this variable is set incorrectly your session will have no access to run X applications in the visible display. Perhaps the fastest way to find out if your DISPLAY is correctly set is to run the xclock application. The commands below should be self explanatory about how it fails and how it should be corrected at least in Debian/Ubuntu: So all you really need to do in order to make sure your selenium server spawns the chrome browser successfully is to export the DISPLAY variable to the correct value before starting the selenium server. Keep that in mind specially after restarting.

Followers