Tuesday, June 25, 2013

Cleaning up artifactory OSS

Artifactory OSS lacks IMO a clean way to aid on cleaning up unneeded artifacts. But there is also an issue with the algorithm to pick when cleaning artifacts. You cannot go just by which artifacts are old as some old artifacts are still needed and in use.

I have concluded that it is better to share with the team a wiki page (it can be also shared on any SCM like SVN, the important thing is that multiple people can edit the file in a collaborative way) composed of current artifacts and leave the team decides which ones should stay. They just need to remove those they want to keep. After they confirm the job is done you can go ahead and delete them. Below is a step by step procedure to get this done:
  1. Running the below command will list all files from all repos in a file called artifactsToDelete. It uses the current date as "to" to make sure all artifacts are included no matter how old they are, then it extracts from the resulting json string (which contains no new lines === unformatted) the artifact urls. It then removes /api/storage from the resulting URL artifact so we can have the real URL that can be used for deletion purposes.
     curl -XGET -u user:password "http://artifactory.sample.com/api/search/creation?to=$[$(date --date "`date`" +%s)*1000]" | sed 's/uri":"\([^"]*\)"/\n\1\n/g' | grep -o '^http.*/' |  sed 's/api.storage.//g' | sort | uniq > artifactsToDelete
    You could add something like the below to remove those artifacts corresponding to a given package/path:
    grep -v "com/sample"
    Or make sure the list includes only those from a certain package/path:
    grep "com/sample"
  2. Offer the list to the team on a wiki page or as SCM resource. The team should delete the entries they are interested on keeping.
  3. Update artifactsToDelete with the filtered content, create a script to delete them and run it:
    $ vi ./deleteArtifacts.sh 
    #!/bin/bash -ex
    for URL in `cat artifactsToDelete`
      curl -XDELETE -u user:password $URL
    $ chmod +x ./deleteArtifacts.sh 
    $ ./deleteArtifacts.sh 
  4. After it runs go to the Maintenance admin page and run the storage garbage collection.

Friday, June 21, 2013

Solaris stty: : Invalid argument when running from cron

In Solaris you might have noticed the below definition in ~/.cshrc and ~/.bashrc:
stty erase ^H
This originates the below error when you try to run a scheduled command (non interactive mode):
stty: : Invalid argument
To solve this issue use the below instead:
test -t 0 && stty erase ^H
You use stty to help with some keyboard issues. Oversimplifying, the issue here is that cron runs with no keyboard attached to the session.

Friday, June 14, 2013

real time port utilization in solaris

I learned TCP/IP using Solaris snoop. Many years after it is proven to be still powerful. I had this issue today where I could not find from where certain requests were coming to one of our services, which is filling log files with socket write error failures, every time a TCP connection is open and suddenly closed. Actually finding the culprit is quite easy with snoop you just provide 'port $port' option and you will see all TCP traffic to and from the local port.
snoop -t a port 9000 > /tmp/snoop.log &
Continue working on something else and inspect snoop.log until you find the culprit. Do not forget to kill the background process though ;-)

Thursday, June 13, 2013

Is your web server vulnerable to BEAST, CRIME, NKOTB?

I have been advocating for deprecating browsers for a while specially because of security reasons but also from an economic perspective.

Is your server vulnerable to SSL attacks like BEAST and CRIME? Check it out from command line or online.

My recommendation is anyway to upgrade your OS and completely remove support for older browsers. That will give you peace of mind about exploits to vulnerabilities in the front end. It is very common to see really old Linux servers, for which automated package managers can't solve vulnerabilities as the OS version is not longer supported.

Unfortunately support for TLS 1.2 is poor so for now a perfect 100% protection is impossible without banning most major browsers.

Compare ( diff ) content from two different remote servers

To get an idea of different files in directories just use ssh+rsync.
ssh -t user1@server1 rsync -nvrc --delete /path/to/dir/to/compare/ user2@server2:/path/to/dir/to/compare/
To check differences in a particular file use ssh+diff.
$ ssh -t user1@server1 'ssh -t user2@server2 cat /path/to/file | diff - /path/to/file'

Thursday, June 06, 2013

svn from cron - svn: Can't read stdin: End of file found

Another "it works from command line but not from cron":
ATTENTION!  Your password for authentication realm:   Subversion repository can only be stored to disk unencrypted!  You are advised to configure your system so that Subversion can store passwords encrypted, if possible.  See the documentation for details. You can avoid future appearances of this warning by setting the value of the 'store-plaintext-passwords' option to either 'yes' or 'no' in '/root/.subversion/servers'. ----------------------------------------------------------------------- Store password unencrypted (yes/no)? svn: Can't read stdin: End of file found
The thing to remember: Commands running from cron will run in a different environment so do not assume your local variables like $HOME, $PATH will be available. In this particular case subversion will store if accepted the password in ~/.subversion directory only after entering "yes". You need instead complete unattended run, so you need to use flag --no-auth-cache like in:
svn co http://subversion.sample.com/my/repo  --username 'myUser' --password 'myPass' --no-auth-cache

Wednesday, June 05, 2013

A shift calendar with Perl

You are told, provide me the availability from/to date for the team members that would be on call to provide support in weekdays. So you create a file with the initials (max three letters) of the team members:
$ cat initials.txt
And then just use this script to generate the below output:
$ ./shift-cal.pl 12 2013 initials.txt

December 2013

Su       Mo       Tu       We       Th       Fr       Sa
         2-JP     3-PP     4-NUR    5-JA     6-FOO             
         9-BAR    10-JP    11-PP    12-NUR   13-JA             
         16-FOO   17-BAR   18-JP    19-PP    20-NUR            
         23-JA    24-FOO   25-BAR   26-JP    27-PP             
         30-NUR   31-JA                                        
In Ubuntu here is how to install the dependencies:
sudo apt-get install libcalendar-simple-perl