SFTP with password from script

The first question I have is if you have access to SFTP client and server? If you do then do not use password, instead use public key authentication.

Then now back to real world. You need to connect to an external SFTP server, the owner does not want to provide public key authentication. A second scenario is a client who wants to connect to your SFTP and they do not want to use certificates either. These two scenarios are driven by bad decisions but those are decisions that are not for you to take nor to fight. You make it clear in a polite email "Please be advise that using passwords is less secure than using public key authentication since it is more vulnerable to brute force attacks". I do not want to engage into the eternal discussion about where to keep the password or the key, so let us move on.

You could use expect and bash to resolve this issue. Remoto-IT uses expect for example to command remote servers. It works perfectly and definitely you could use the same way expect to interact with SFTP.

But there is a great program called lftp which is available in most Unix/Linux distros and if not you can always compile from sources. Here is an example of how to install it and use it in ubuntu:

# Ubuntu installation
$ sudo apt-get install lftp
# Save the list directory command to a file
$ echo dir > commands.sftp
# Run all commands from a file
$ cat commands.sftp | lftp -u myUser,myPassword sftp://sftp.sample.com
#Run a command directly from #linuxoneliner
$ echo dir | lftp -u -u myUser,myPassword sftp://sftp.sample.com

Mac OSX brew MD5 mismatch

I was trying to install aacgain today when I got:

$ brew install aacgain
==> Downloading http://altosdesign.com/aacgain/alvarez/aacgain-1.8.tar.bz2
######################################################################## 100.0%
Error: MD5 mismatch
Expected: 61ce9e648fa1773adb3d4b3c84c6e4ca
Got: 18461da7c93ef44001051ea8aa07d34c
Archive: /Library/Caches/Homebrew/aacgain-1.8.tar.bz2
(To retry an incomplete download, remove the file above.)

The below commands allowed me to install aacgain without issue. Basically you just need to update brew and then delete the cached file.

$ brew update
$ rm /Library/Caches/Homebrew/aacgain-1.8.tar.bz2

Generate private keys and SSL certificate requests in batch or unattended way

When you have to generate multiple certificates for a lot of domains you better get smart at it ;-) Below is a bash script to generate a password-less key (needless to say you need this for servers but at the same time you must protect them from unauthorized access) and a certificate request:

#!/bin/bash -e
# gencert.sh
# @author: Nestor Urquiza

USAGE="Usage: `basename $0`       "

if [ $# -ne "7" ]
then
  echo $USAGE
  exit 1
fi

countryCode=$1
state=$2
city=$3
company=$4
organizationalUnitName=$5
domain=$6
email=$7

openssl req -nodes -newkey rsa:2048 -keyout ${domain}.key -out  ${domain}.csr -batch -subj "/C=$countryCode/ST=$state/L=$city/O=$company/OU=$organizationalUnitName/CN=$domain/emailAddress=$email"

Here is how you would use it:

export domain=domain.sample.com && ./gencert.sh "US" "CA" "San Francisco" "Domain Sample LLC" "Operations" "$domain" "@sample.com"

brew Warning: Your Xcode is outdated even though I had last version

Yup. After installing XCode and after installing the Command Line Tools (Following Downloads/Components/Command Line Tools/Install) I still got issues with brew:

$ brew install aacgain
Warning: Your Xcode (4.2) is outdated
Please install Xcode 4.6.2.
...

The issue was related to a miss configuration:

$ xcode-select -print-path
/Volumes/Xcode/Xcode.app/Contents/Developer

And corrected when pointing to the right path:

$ sudo xcode-select --switch /Applications/Xcode.app 

Create a zip file out of certain files in the file system

I had to generate several certificates today which I did from a script. To just pack them all I used the below which can be used to zip respecting any number of files respecting their directory structure.

find . -name "*.csr" -exec zip -g certs.zip {} +

Ubuntu The following packages have unmet dependencies: $package1 : Depends: $package2 but it is not going to be installed

I got today the below from recipes that are supposed to configure a brand new desktop:

$ sudo apt-get -q -y -f build-dep virtualbox-ose-guest-utils virtualbox-ose-guest-x11 virtualbox-ose-guest-dkms
...
The following packages have unmet dependencies:
 virtualbox-ose-guest-x11 : Depends: virtualbox-guest-x11 but it is not going to be installed
...

If you add virtualbox-guest-x11 to the installation then you get something else:

$ sudo apt-get -q -y -f install virtualbox-ose-guest-utils virtualbox-guest-x11 virtualbox-ose-guest-x11 virtualbox-ose-guest-dkms
...
The following packages have unmet dependencies:
 virtualbox-guest-x11 : Depends: xorg-video-abi-11
                        Depends: xserver-xorg-core (>= 2:1.10.99.901)
E: Unable to correct problems, you have held broken packages.
...

Just running the below will take care of installing the package and all its dependencies, however be prepared to wait a really long time before all packages are built and installed. You probably do not want to proceed this route and just find absolutely all dependencies by hand. It will depend on your needs:

$ sudo apt-get -q -y -f build-dep virtualbox-ose-guest-utils virtualbox-guest-x11 virtualbox-ose-guest-x11 virtualbox-ose-guest-dkms

The bottom line is that after using build-dep you can now proceed without errors with your install:

$ sudo apt-get -q -y -f install virtualbox-ose-guest-utils virtualbox-guest-x11 virtualbox-ose-guest-x11 virtualbox-ose-guest-dkms

Pre-populate Talend custom component schema

I run into an issue today where the tFileInputCSVFilter custom component was outputting empty lines instead of real records. The issue was related to using READONLY="true" in PARAMETER FIELD="SCHEMA_TYPE". Setting it to true or just removing it to fall into the default value did the trick. Leaving it to true will make the schema non editable. Going the extra mile I set the TABLE node inside the filter and reject schemas to make sure the "line" (which represents a whole line from the input) was available automatically. For example for the filter (I did the same for main and reject even though main is configured as having no outputs):

...
     <PARAMETER NAME="SCHEMA_FILTER" FIELD="SCHEMA_TYPE" NUM_ROW="1" CONTEXT="FILTER">
       <TABLE READONLY="false">
         <COLUMN NAME="line" TYPE="id_String" READONLY="false"/>
       </TABLE>
     </PARAMETER>
...

Do not forget to set the component schema to auto propagate using COMPONENT SCHEMA_AUTO_PROPAGATE ="true".

Bottom line I have found that the schema must be editable so in Talend custom components in order for them to be pre-populate or even available to following components. Failure to do so will result in empty lines as the schema will be missing and so there would be no place to flush the output.