Monday, February 24, 2014

Parsing JSON from bash

The easiest way I have found so far to parse JSON from bash is being to use JSON.sh. To illustrate its usage let us pick a JIRA search JSON response for the below query which is looking for all the issues reported for the current user
https://jira.atlassian.com/rest/api/2/search?jql=reporter%20%3D%20currentUser()
Let us save the result as search.json and run the below:
$ cat  ~/Downloads/search.json | ./JSON.sh -l > /tmp/search.json.all.leaf
Note how the complete structure is shown, up to the leaf nodes so you will need to use some filtering (grep) to extract the needed results:
$ cat /tmp/search.json.all.leaf
["expand"] "schema,names"
["startAt"] 0
["maxResults"] 50
["total"] 3
["issues",0,"expand"] "editmeta,renderedFields,transitions,changelog,operations"
["issues",0,"id"] "317538"
["issues",0,"self"] "https://jira.atlassian.com/rest/api/2/issue/317538"
["issues",0,"key"] "JRA-36973"
["issues",0,"fields","progress","progress"] 0
["issues",0,"fields","progress","total"] 0
["issues",0,"fields","summary"] "Setting Personal WIP or WIP limits per user"
["issues",0,"fields","issuetype","self"] "https://jira.atlassian.com/rest/api/2/issuetype/2"
["issues",0,"fields","issuetype","id"] "2"
["issues",0,"fields","issuetype","description"] "A new feature of the product, which has yet to be developed."
["issues",0,"fields","issuetype","iconUrl"] "https://jira.atlassian.com/images/icons/issuetypes/newfeature.png"
["issues",0,"fields","issuetype","name"] "New Feature"
["issues",0,"fields","issuetype","subtask"] false
["issues",0,"fields","customfield_10610"] null
["issues",0,"fields","timespent"] null
["issues",0,"fields","customfield_10230"] null
["issues",0,"fields","customfield_14532"] null
["issues",0,"fields","reporter","self"] "https://jira.atlassian.com/rest/api/2/user?username=nurquiza"
["issues",0,"fields","reporter","name"] "nurquiza"
["issues",0,"fields","reporter","emailAddress"] "me@sample.com"
["issues",0,"fields","reporter","avatarUrls","16x16"] "https://jira.atlassian.com/secure/useravatar?size=xsmall&avatarId=10612"
["issues",0,"fields","reporter","avatarUrls","24x24"] "https://jira.atlassian.com/secure/useravatar?size=small&avatarId=10612"
["issues",0,"fields","reporter","avatarUrls","32x32"] "https://jira.atlassian.com/secure/useravatar?size=medium&avatarId=10612"
["issues",0,"fields","reporter","avatarUrls","48x48"] "https://jira.atlassian.com/secure/useravatar?avatarId=10612"
["issues",0,"fields","reporter","displayName"] "Nestor Urquiza"
["issues",0,"fields","reporter","active"] true
["issues",0,"fields","customfield_13231"] null
["issues",0,"fields","updated"] "2014-02-12T13:52:06.000+0000"
["issues",0,"fields","created"] "2014-02-12T13:52:06.000+0000"
["issues",0,"fields","customfield_10180"] null
["issues",0,"fields","description"] "Hello JIRA team,\r\n\r\nI would like to propose an enhancement for JIRA that will IMO benefit all agile/lean teams out there. \r\n\r\n=========\r\nNarrative\r\n=========\r\n\r\nAS a team member\r\n\r\nI WANT jira to stop me from going above my defined WIP limit\r\n\r\nSO that I can stop starting and start finalizing\r\n\r\n================================================\r\nScenarios (examples based on classes of services aka CoS)\r\n================================================\r\n\r\nGIVEN an attempt to assign an issue to a user\r\n\r\nWHEN the issue is expedite and there is one current expedite issue assigned\r\n\r\nTHEN do not accept the assignment\r\n\r\nWHEN the issue is non expedite (fixed delivery date, standard or intangible) and there are two current non expedite issues assigned\r\n\r\nTHEN do not accept the assignment\r\n\r\n==========================================================\r\nImplementation proposal (applicable to any kind of scenarios including the above)\r\n==========================================================\r\n\r\nNote how the two WHEN can be expressed as a JQL filter, an operator and a result count. Let us call them Personal WIP rules:\r\n\r\nfilter, operator, count\r\n\r\nPWIP rule1: CoS = \"expedite\", <=, 1\r\n\r\nPWIP rule2: CoS in (\"fixed delivery date\", \"standard\", \"intangible\"), <=, 2\r\n\r\nThese rules should be assigned to user roles.\r\n\r\nSuppose some user (the originator) attempts to assign a candidate issue to another user or self (the target). The system will count the current tickets assigned based on the filter rules for the target user, will apply the filter rules for the candidate issue and if a filter condition is matched then the count will increase for that filter. The rules will trigger and only if the user can accept the ticket the operation will be successful.\r\n\r\nIn fact the target user should not even be presented as a candidate to resolve the ticket. At a minimum, as stated above, the validation of the submission should fail. The result should be an error like \"Could not perform this action. Personal WIP limit reached\"\r\n\r\nThanks!\r\n\r\n- Nestor"
["issues",0,"fields","priority","self"] "https://jira.atlassian.com/rest/api/2/priority/4"
["issues",0,"fields","priority","iconUrl"] "https://jira.atlassian.com/images/icons/priorities/minor.png"
["issues",0,"fields","priority","name"] "Minor"
["issues",0,"fields","priority","id"] "4"
["issues",0,"fields","customfield_13430"] null
["issues",0,"fields","customfield_13230"] null
["issues",0,"fields","customfield_14430"] null
["issues",0,"fields","customfield_12630"] null
["issues",0,"fields","status","self"] "https://jira.atlassian.com/rest/api/2/status/10034"
["issues",0,"fields","status","description"] "This issue is newly created and is awaiting review from Product Management."
["issues",0,"fields","status","iconUrl"] "https://jira.atlassian.com/images/icons/statuses/generic.png"
["issues",0,"fields","status","name"] "New"
["issues",0,"fields","status","id"] "10034"
["issues",0,"fields","status","statusCategory","self"] "https://jira.atlassian.com/rest/api/2/statuscategory/2"
["issues",0,"fields","status","statusCategory","id"] 2
["issues",0,"fields","status","statusCategory","key"] "new"
["issues",0,"fields","status","statusCategory","colorName"] "blue-gray"
["issues",0,"fields","status","statusCategory","name"] "New"
["issues",0,"fields","customfield_10575"] null
["issues",0,"fields","workratio"] -1
["issues",0,"fields","customfield_12832"] null
["issues",0,"fields","customfield_12531"] "Not Started"
["issues",0,"fields","customfield_12833"] null
["issues",0,"fields","project","self"] "https://jira.atlassian.com/rest/api/2/project/10240"
["issues",0,"fields","project","id"] "10240"
["issues",0,"fields","project","key"] "JRA"
["issues",0,"fields","project","name"] "JIRA"
["issues",0,"fields","project","avatarUrls","16x16"] "https://jira.atlassian.com/secure/projectavatar?size=xsmall&pid=10240&avatarId=17294"
["issues",0,"fields","project","avatarUrls","24x24"] "https://jira.atlassian.com/secure/projectavatar?size=small&pid=10240&avatarId=17294"
["issues",0,"fields","project","avatarUrls","32x32"] "https://jira.atlassian.com/secure/projectavatar?size=medium&pid=10240&avatarId=17294"
["issues",0,"fields","project","avatarUrls","48x48"] "https://jira.atlassian.com/secure/projectavatar?pid=10240&avatarId=17294"
["issues",0,"fields","project","projectCategory","self"] "https://jira.atlassian.com/rest/api/2/projectCategory/10031"
["issues",0,"fields","project","projectCategory","id"] "10031"
["issues",0,"fields","project","projectCategory","description"] ""
["issues",0,"fields","project","projectCategory","name"] "Atlassian Products"
["issues",0,"fields","customfield_12931"] null
["issues",0,"fields","environment"] null
["issues",0,"fields","customfield_10723"] null
["issues",0,"fields","aggregateprogress","progress"] 0
["issues",0,"fields","aggregateprogress","total"] 0
["issues",0,"fields","lastViewed"] "2014-02-24T18:14:38.173+0000"
["issues",0,"fields","timeoriginalestimate"] null
["issues",0,"fields","customfield_10150",0] "nurquiza(nurquiza)"
["issues",0,"fields","customfield_11436"] "230142"
["issues",0,"fields","customfield_11435"] "230142"
["issues",0,"fields","customfield_11437"] "229812"
["issues",0,"fields","votes","self"] "https://jira.atlassian.com/rest/api/2/issue/JRA-36973/votes"
["issues",0,"fields","votes","votes"] 10
["issues",0,"fields","votes","hasVoted"] false
["issues",0,"fields","resolution"] null
["issues",0,"fields","customfield_10680"] null
["issues",0,"fields","resolutiondate"] null
["issues",0,"fields","creator","self"] "https://jira.atlassian.com/rest/api/2/user?username=nurquiza"
["issues",0,"fields","creator","name"] "nurquiza"
["issues",0,"fields","creator","emailAddress"] "me@sample.com"
["issues",0,"fields","creator","avatarUrls","16x16"] "https://jira.atlassian.com/secure/useravatar?size=xsmall&avatarId=10612"
["issues",0,"fields","creator","avatarUrls","24x24"] "https://jira.atlassian.com/secure/useravatar?size=small&avatarId=10612"
["issues",0,"fields","creator","avatarUrls","32x32"] "https://jira.atlassian.com/secure/useravatar?size=medium&avatarId=10612"
["issues",0,"fields","creator","avatarUrls","48x48"] "https://jira.atlassian.com/secure/useravatar?avatarId=10612"
["issues",0,"fields","creator","displayName"] "Nestor Urquiza"
["issues",0,"fields","creator","active"] true
["issues",0,"fields","aggregatetimeoriginalestimate"] null
["issues",0,"fields","customfield_12930"] null
["issues",0,"fields","customfield_10161"] "true"
["issues",0,"fields","customfield_12430"] null
["issues",0,"fields","customfield_10160"] "1036800"
["issues",0,"fields","customfield_12431"] null
["issues",0,"fields","customfield_12432"] null
["issues",0,"fields","customfield_12433"] null
["issues",0,"fields","customfield_12434"] null
["issues",0,"fields","customfield_12831"] null
["issues",0,"fields","customfield_12435"] null
["issues",0,"fields","customfield_12830"] null
["issues",0,"fields","customfield_12730"] null
["issues",0,"fields","watches","self"] "https://jira.atlassian.com/rest/api/2/issue/JRA-36973/watchers"
["issues",0,"fields","watches","watchCount"] 3
["issues",0,"fields","watches","isWatching"] true
["issues",0,"fields","customfield_11631"] null
["issues",0,"fields","customfield_11130"] "244682"
["issues",0,"fields","customfield_10651"] null
["issues",0,"fields","customfield_10650"] null
["issues",0,"fields","customfield_11230"] null
["issues",0,"fields","customfield_10653"] null
["issues",0,"fields","assignee"] null
["issues",0,"fields","customfield_14130"] null
["issues",0,"fields","aggregatetimeestimate"] null
["issues",0,"fields","customfield_14330"] null
["issues",0,"fields","customfield_12131"] null
["issues",0,"fields","timeestimate"] null
["issues",0,"fields","customfield_11531"] null
["issues",0,"fields","customfield_14334"] null
["issues",0,"fields","customfield_11433"] "238684"
["issues",0,"fields","customfield_14333"] null
["issues",0,"fields","customfield_11434"] "229880"
["issues",0,"fields","customfield_14332"] null
["issues",0,"fields","customfield_11431"] "244122"
["issues",0,"fields","customfield_11930"] null
["issues",0,"fields","customfield_14331"] null
["issues",0,"fields","customfield_12130"] null
["issues",0,"fields","aggregatetimespent"] null
["issues",0,"fields","customfield_14335"] null
["issues",1,"expand"] "editmeta,renderedFields,transitions,changelog,operations"
["issues",1,"id"] "315032"
["issues",1,"self"] "https://jira.atlassian.com/rest/api/2/issue/315032"
["issues",1,"key"] "JRA-36716"
["issues",1,"fields","progress","progress"] 7200
["issues",1,"fields","progress","total"] 7200
["issues",1,"fields","progress","percent"] 100
["issues",1,"fields","summary"] "worklog expand not working for REST search API"
["issues",1,"fields","issuetype","self"] "https://jira.atlassian.com/rest/api/2/issuetype/5"
["issues",1,"fields","issuetype","id"] "5"
["issues",1,"fields","issuetype","description"] "Issue that is likely specific to the installation"
["issues",1,"fields","issuetype","iconUrl"] "https://jira.atlassian.com/images/icons/status_generic.gif"
["issues",1,"fields","issuetype","name"] "Support Request"
["issues",1,"fields","issuetype","subtask"] false
["issues",1,"fields","customfield_10610"] null
["issues",1,"fields","timespent"] 7200
["issues",1,"fields","customfield_10230"] null
["issues",1,"fields","customfield_14532"] null
["issues",1,"fields","reporter","self"] "https://jira.atlassian.com/rest/api/2/user?username=nurquiza"
["issues",1,"fields","reporter","name"] "nurquiza"
["issues",1,"fields","reporter","emailAddress"] "me@sample.com"
["issues",1,"fields","reporter","avatarUrls","16x16"] "https://jira.atlassian.com/secure/useravatar?size=xsmall&avatarId=10612"
["issues",1,"fields","reporter","avatarUrls","24x24"] "https://jira.atlassian.com/secure/useravatar?size=small&avatarId=10612"
["issues",1,"fields","reporter","avatarUrls","32x32"] "https://jira.atlassian.com/secure/useravatar?size=medium&avatarId=10612"
["issues",1,"fields","reporter","avatarUrls","48x48"] "https://jira.atlassian.com/secure/useravatar?avatarId=10612"
["issues",1,"fields","reporter","displayName"] "Nestor Urquiza"
["issues",1,"fields","reporter","active"] true
["issues",1,"fields","customfield_13231"] "1_*:*_1_*:*_474593564_*|*_5_*:*_1_*:*_0"
["issues",1,"fields","updated"] "2014-01-30T18:01:05.000+0000"
["issues",1,"fields","created"] "2014-01-24T17:39:24.000+0000"
["issues",1,"fields","customfield_10180"] null
["issues",1,"fields","description"] "Hello,\r\n\r\nWorklog details for a specific issue can be obtained from /rest/api/latest/issue/59022/worklog however it can't be obtained from the search API. The call /rest/api/2/search?jql=createdDate > \"2014/1/1\"&expand=worklog does not work as I would have expected. No worklog information will be returned.\r\n\r\nPlease my apologies if this is to be considered a feature request instead in which case feel free to change it to bug.\r\n\r\nIt would be great if JIRA REST API would return more detailed information for all bugs contained in a JQL. The \"expand\" parameter works with custom fields but I can't figure how to expand the worklog. Getting the results of the JQL and iterating one issue at a time to get the worklog details is overkilling.\r\n\r\nI know you can do this with paid plugins but such functionality I believe should be provided out of the box.\r\n\r\nThanks,\r\n\r\n- Nestor"
["issues",1,"fields","priority","self"] "https://jira.atlassian.com/rest/api/2/priority/4"
["issues",1,"fields","priority","iconUrl"] "https://jira.atlassian.com/images/icons/priorities/minor.png"
["issues",1,"fields","priority","name"] "Minor"
["issues",1,"fields","priority","id"] "4"
["issues",1,"fields","customfield_13430"] null
["issues",1,"fields","customfield_13230"] "2014-01-30 05:26:14.142"
["issues",1,"fields","customfield_14430"] null
["issues",1,"fields","customfield_10571"] null
["issues",1,"fields","customfield_12630"] null
["issues",1,"fields","status","self"] "https://jira.atlassian.com/rest/api/2/status/5"
["issues",1,"fields","status","description"] "A resolution has been taken, and it is awaiting verification by reporter. From here issues are either reopened, or are closed."
["issues",1,"fields","status","iconUrl"] "https://jira.atlassian.com/images/icons/statuses/resolved.png"
["issues",1,"fields","status","name"] "Resolved"
["issues",1,"fields","status","id"] "5"
["issues",1,"fields","status","statusCategory","self"] "https://jira.atlassian.com/rest/api/2/statuscategory/3"
["issues",1,"fields","status","statusCategory","id"] 3
["issues",1,"fields","status","statusCategory","key"] "done"
["issues",1,"fields","status","statusCategory","colorName"] "green"
["issues",1,"fields","status","statusCategory","name"] "Complete"
["issues",1,"fields","customfield_10575"] null
["issues",1,"fields","workratio"] -1
["issues",1,"fields","customfield_12832"] null
["issues",1,"fields","customfield_12833"] null
["issues",1,"fields","customfield_12531"] "Not Started"
["issues",1,"fields","project","self"] "https://jira.atlassian.com/rest/api/2/project/10240"
["issues",1,"fields","project","id"] "10240"
["issues",1,"fields","project","key"] "JRA"
["issues",1,"fields","project","name"] "JIRA"
["issues",1,"fields","project","avatarUrls","16x16"] "https://jira.atlassian.com/secure/projectavatar?size=xsmall&pid=10240&avatarId=17294"
["issues",1,"fields","project","avatarUrls","24x24"] "https://jira.atlassian.com/secure/projectavatar?size=small&pid=10240&avatarId=17294"
["issues",1,"fields","project","avatarUrls","32x32"] "https://jira.atlassian.com/secure/projectavatar?size=medium&pid=10240&avatarId=17294"
["issues",1,"fields","project","avatarUrls","48x48"] "https://jira.atlassian.com/secure/projectavatar?pid=10240&avatarId=17294"
["issues",1,"fields","project","projectCategory","self"] "https://jira.atlassian.com/rest/api/2/projectCategory/10031"
["issues",1,"fields","project","projectCategory","id"] "10031"
["issues",1,"fields","project","projectCategory","description"] ""
["issues",1,"fields","project","projectCategory","name"] "Atlassian Products"
["issues",1,"fields","customfield_12931"] null
["issues",1,"fields","environment"] null
["issues",1,"fields","customfield_10723"] null
["issues",1,"fields","aggregateprogress","progress"] 7200
["issues",1,"fields","aggregateprogress","total"] 7200
["issues",1,"fields","aggregateprogress","percent"] 100
["issues",1,"fields","lastViewed"] "2014-01-30T18:01:06.346+0000"
["issues",1,"fields","components",0,"self"] "https://jira.atlassian.com/rest/api/2/component/13170"
["issues",1,"fields","components",0,"id"] "13170"
["issues",1,"fields","components",0,"name"] "Remote API (REST)"
["issues",1,"fields","components",0,"description"] "Issues affecting JIRA's REST API"
["issues",1,"fields","timeoriginalestimate"] null
["issues",1,"fields","customfield_10150",0] "nurquiza(nurquiza)"
["issues",1,"fields","customfield_10150",1] "ohernandez@atlassian.com(ohernandez@atlassian.com)"
["issues",1,"fields","customfield_11436"] "227731"
["issues",1,"fields","customfield_11435"] "227731"
["issues",1,"fields","customfield_11437"] "227401"
["issues",1,"fields","votes","self"] "https://jira.atlassian.com/rest/api/2/issue/JRA-36716/votes"
["issues",1,"fields","votes","votes"] 8
["issues",1,"fields","votes","hasVoted"] false
["issues",1,"fields","customfield_10401"] null
["issues",1,"fields","customfield_10680"] null
["issues",1,"fields","resolution","self"] "https://jira.atlassian.com/rest/api/2/resolution/9"
["issues",1,"fields","resolution","id"] "9"
["issues",1,"fields","resolution","description"] "Support query has been answered, or issue confirmed"
["issues",1,"fields","resolution","name"] "Answered"
["issues",1,"fields","resolutiondate"] "2014-01-30T05:29:18.000+0000"
["issues",1,"fields","creator","self"] "https://jira.atlassian.com/rest/api/2/user?username=nurquiza"
["issues",1,"fields","creator","name"] "nurquiza"
["issues",1,"fields","creator","emailAddress"] "me@sample.com"
["issues",1,"fields","creator","avatarUrls","16x16"] "https://jira.atlassian.com/secure/useravatar?size=xsmall&avatarId=10612"
["issues",1,"fields","creator","avatarUrls","24x24"] "https://jira.atlassian.com/secure/useravatar?size=small&avatarId=10612"
["issues",1,"fields","creator","avatarUrls","32x32"] "https://jira.atlassian.com/secure/useravatar?size=medium&avatarId=10612"
["issues",1,"fields","creator","avatarUrls","48x48"] "https://jira.atlassian.com/secure/useravatar?avatarId=10612"
["issues",1,"fields","creator","displayName"] "Nestor Urquiza"
["issues",1,"fields","creator","active"] true
["issues",1,"fields","aggregatetimeoriginalestimate"] null
["issues",1,"fields","customfield_12930"] null
["issues",1,"fields","customfield_12430"] null
["issues",1,"fields","customfield_10161"] "true"
["issues",1,"fields","customfield_12431"] null
["issues",1,"fields","customfield_10160"] "2160000"
["issues",1,"fields","customfield_12432"] null
["issues",1,"fields","customfield_12433"] null
["issues",1,"fields","customfield_12434"] null
["issues",1,"fields","customfield_12831"] null
["issues",1,"fields","customfield_12435"] null
["issues",1,"fields","customfield_12830"] null
["issues",1,"fields","duedate"] null
["issues",1,"fields","customfield_12730"] null
["issues",1,"fields","watches","self"] "https://jira.atlassian.com/rest/api/2/issue/JRA-36716/watchers"
["issues",1,"fields","watches","watchCount"] 4
["issues",1,"fields","watches","isWatching"] true
["issues",1,"fields","customfield_10162"] "sample.com"
["issues",1,"fields","customfield_11631"] null
["issues",1,"fields","customfield_11130"] "242403"
["issues",1,"fields","customfield_10651"] null
["issues",1,"fields","customfield_10650"] null
["issues",1,"fields","customfield_11230"] null
["issues",1,"fields","customfield_10653"] null
["issues",1,"fields","assignee"] null
["issues",1,"fields","customfield_10200"] "
\r\nSupport Requests are no longer handled by this system.

\r\nAs part of our ongoing effort to streamline support and make it a fairer system we are trying to centralise all support.
\r\nPlease create all Support Requests on our Support System\"\". If you have not used our support service before you will need to sign up for a new account.

\r\n\r\nIf you have been directed here by any documentation, please let us know in your support ticket and we will update the documentation.
\r\nSorry for any inconvenience.\r\n

\r\nCheers,
\r\nAtlassian Support Manager\r\n

\r\n\r\n" ["issues",1,"fields","customfield_14130"] null ["issues",1,"fields","aggregatetimeestimate"] 0 ["issues",1,"fields","customfield_12131"] null ["issues",1,"fields","customfield_14330"] null ["issues",1,"fields","timeestimate"] 0 ["issues",1,"fields","customfield_11531"] null ["issues",1,"fields","customfield_14334"] null ["issues",1,"fields","customfield_11433"] "236313" ["issues",1,"fields","customfield_14333"] null ["issues",1,"fields","customfield_11434"] "227469" ["issues",1,"fields","customfield_14332"] null ["issues",1,"fields","customfield_11930"] null ["issues",1,"fields","customfield_11431"] "242073" ["issues",1,"fields","customfield_14331"] null ["issues",1,"fields","customfield_12130"] null ["issues",1,"fields","aggregatetimespent"] 7200 ["issues",1,"fields","customfield_14335"] null ["issues",2,"expand"] "editmeta,renderedFields,transitions,changelog,operations" ["issues",2,"id"] "308150" ["issues",2,"self"] "https://jira.atlassian.com/rest/api/2/issue/308150" ["issues",2,"key"] "GHS-10114" ["issues",2,"fields","progress","progress"] 0 ["issues",2,"fields","progress","total"] 0 ["issues",2,"fields","summary"] "As a Lean Project Coordinator I would like a horizontal scrollbar on JIRA Agile Kanban Board" ["issues",2,"fields","issuetype","self"] "https://jira.atlassian.com/rest/api/2/issuetype/16" ["issues",2,"fields","issuetype","id"] "16" ["issues",2,"fields","issuetype","description"] "" ["issues",2,"fields","issuetype","iconUrl"] "https://jira.atlassian.com/images/icons/issuetypes/sales.png" ["issues",2,"fields","issuetype","name"] "Story" ["issues",2,"fields","issuetype","subtask"] false ["issues",2,"fields","timespent"] null ["issues",2,"fields","customfield_10230"] null ["issues",2,"fields","customfield_14532"] null ["issues",2,"fields","reporter","self"] "https://jira.atlassian.com/rest/api/2/user?username=nurquiza" ["issues",2,"fields","reporter","name"] "nurquiza" ["issues",2,"fields","reporter","emailAddress"] "me@sample.com" ["issues",2,"fields","reporter","avatarUrls","16x16"] "https://jira.atlassian.com/secure/useravatar?size=xsmall&avatarId=10612" ["issues",2,"fields","reporter","avatarUrls","24x24"] "https://jira.atlassian.com/secure/useravatar?size=small&avatarId=10612" ["issues",2,"fields","reporter","avatarUrls","32x32"] "https://jira.atlassian.com/secure/useravatar?size=medium&avatarId=10612" ["issues",2,"fields","reporter","avatarUrls","48x48"] "https://jira.atlassian.com/secure/useravatar?avatarId=10612" ["issues",2,"fields","reporter","displayName"] "Nestor Urquiza" ["issues",2,"fields","reporter","active"] true ["issues",2,"fields","customfield_13231"] "1_*:*_1_*:*_1381035808_*|*_5_*:*_1_*:*_0" ["issues",2,"fields","updated"] "2013-12-12T02:51:23.000+0000" ["issues",2,"fields","created"] "2013-11-26T03:14:08.000+0000" ["issues",2,"fields","customfield_10180"] null ["issues",2,"fields","description"] "As a Lean Project Coordinator \r\nI would like a horizontal scrollbar on JIRA Agile Kanban Board\r\nSo that the number of columns (value stream stages) can grow to any number\r\n\r\nI believe the original request for the Scrum rapid board was fair as much as this one.\r\n\r\nIdeally JIRA Agile should support a configuration that allows fixed board or scrollable board.\r\n\r\nIn the meanwhile we can use one of the hacks I described in http://thinkinginsoftware.blogspot.com/2013/11/jira-agile-extended-kanban-board.html which involves modifying the style after the page is rendered via a script that can be run from a browser plugin like https://github.com/nestoru/tampermonkey-jira-extended-rapidboard\r\n" ["issues",2,"fields","priority","self"] "https://jira.atlassian.com/rest/api/2/priority/2" ["issues",2,"fields","priority","iconUrl"] "https://jira.atlassian.com/images/icons/priorities/critical.png" ["issues",2,"fields","priority","name"] "Critical" ["issues",2,"fields","priority","id"] "2" ["issues",2,"fields","customfield_13430"] null ["issues",2,"fields","issuelinks",0,"id"] "118494" ["issues",2,"fields","issuelinks",0,"self"] "https://jira.atlassian.com/rest/api/2/issueLink/118494" ["issues",2,"fields","issuelinks",0,"type","id"] "10030" ["issues",2,"fields","issuelinks",0,"type","name"] "Cloners" ["issues",2,"fields","issuelinks",0,"type","inward"] "was cloned as" ["issues",2,"fields","issuelinks",0,"type","outward"] "is cloned from" ["issues",2,"fields","issuelinks",0,"type","self"] "https://jira.atlassian.com/rest/api/2/issueLinkType/10030" ["issues",2,"fields","issuelinks",0,"outwardIssue","id"] "248309" ["issues",2,"fields","issuelinks",0,"outwardIssue","key"] "GHS-6951" ["issues",2,"fields","issuelinks",0,"outwardIssue","self"] "https://jira.atlassian.com/rest/api/2/issue/248309" ["issues",2,"fields","issuelinks",0,"outwardIssue","fields","summary"] "As a Scrum Master I would like a horizontal scrollbar on greenhopper Rapid Board" ["issues",2,"fields","issuelinks",0,"outwardIssue","fields","status","self"] "https://jira.atlassian.com/rest/api/2/status/5" ["issues",2,"fields","issuelinks",0,"outwardIssue","fields","status","description"] "A resolution has been taken, and it is awaiting verification by reporter. From here issues are either reopened, or are closed." ["issues",2,"fields","issuelinks",0,"outwardIssue","fields","status","iconUrl"] "https://jira.atlassian.com/images/icons/statuses/resolved.png" ["issues",2,"fields","issuelinks",0,"outwardIssue","fields","status","name"] "Resolved" ["issues",2,"fields","issuelinks",0,"outwardIssue","fields","status","id"] "5" ["issues",2,"fields","issuelinks",0,"outwardIssue","fields","status","statusCategory","self"] "https://jira.atlassian.com/rest/api/2/statuscategory/3" ["issues",2,"fields","issuelinks",0,"outwardIssue","fields","status","statusCategory","id"] 3 ["issues",2,"fields","issuelinks",0,"outwardIssue","fields","status","statusCategory","key"] "done" ["issues",2,"fields","issuelinks",0,"outwardIssue","fields","status","statusCategory","colorName"] "green" ["issues",2,"fields","issuelinks",0,"outwardIssue","fields","status","statusCategory","name"] "Complete" ["issues",2,"fields","issuelinks",0,"outwardIssue","fields","priority","self"] "https://jira.atlassian.com/rest/api/2/priority/3" ["issues",2,"fields","issuelinks",0,"outwardIssue","fields","priority","iconUrl"] "https://jira.atlassian.com/images/icons/priorities/major.png" ["issues",2,"fields","issuelinks",0,"outwardIssue","fields","priority","name"] "Major" ["issues",2,"fields","issuelinks",0,"outwardIssue","fields","priority","id"] "3" ["issues",2,"fields","issuelinks",0,"outwardIssue","fields","issuetype","self"] "https://jira.atlassian.com/rest/api/2/issuetype/16" ["issues",2,"fields","issuelinks",0,"outwardIssue","fields","issuetype","id"] "16" ["issues",2,"fields","issuelinks",0,"outwardIssue","fields","issuetype","description"] "" ["issues",2,"fields","issuelinks",0,"outwardIssue","fields","issuetype","iconUrl"] "https://jira.atlassian.com/images/icons/issuetypes/sales.png" ["issues",2,"fields","issuelinks",0,"outwardIssue","fields","issuetype","name"] "Story" ["issues",2,"fields","issuelinks",0,"outwardIssue","fields","issuetype","subtask"] false ["issues",2,"fields","issuelinks",1,"id"] "119776" ["issues",2,"fields","issuelinks",1,"self"] "https://jira.atlassian.com/rest/api/2/issueLink/119776" ["issues",2,"fields","issuelinks",1,"type","id"] "10001" ["issues",2,"fields","issuelinks",1,"type","name"] "Duplicate" ["issues",2,"fields","issuelinks",1,"type","inward"] "is duplicated by" ["issues",2,"fields","issuelinks",1,"type","outward"] "duplicates" ["issues",2,"fields","issuelinks",1,"type","self"] "https://jira.atlassian.com/rest/api/2/issueLinkType/10001" ["issues",2,"fields","issuelinks",1,"outwardIssue","id"] "248309" ["issues",2,"fields","issuelinks",1,"outwardIssue","key"] "GHS-6951" ["issues",2,"fields","issuelinks",1,"outwardIssue","self"] "https://jira.atlassian.com/rest/api/2/issue/248309" ["issues",2,"fields","issuelinks",1,"outwardIssue","fields","summary"] "As a Scrum Master I would like a horizontal scrollbar on greenhopper Rapid Board" ["issues",2,"fields","issuelinks",1,"outwardIssue","fields","status","self"] "https://jira.atlassian.com/rest/api/2/status/5" ["issues",2,"fields","issuelinks",1,"outwardIssue","fields","status","description"] "A resolution has been taken, and it is awaiting verification by reporter. From here issues are either reopened, or are closed." ["issues",2,"fields","issuelinks",1,"outwardIssue","fields","status","iconUrl"] "https://jira.atlassian.com/images/icons/statuses/resolved.png" ["issues",2,"fields","issuelinks",1,"outwardIssue","fields","status","name"] "Resolved" ["issues",2,"fields","issuelinks",1,"outwardIssue","fields","status","id"] "5" ["issues",2,"fields","issuelinks",1,"outwardIssue","fields","status","statusCategory","self"] "https://jira.atlassian.com/rest/api/2/statuscategory/3" ["issues",2,"fields","issuelinks",1,"outwardIssue","fields","status","statusCategory","id"] 3 ["issues",2,"fields","issuelinks",1,"outwardIssue","fields","status","statusCategory","key"] "done" ["issues",2,"fields","issuelinks",1,"outwardIssue","fields","status","statusCategory","colorName"] "green" ["issues",2,"fields","issuelinks",1,"outwardIssue","fields","status","statusCategory","name"] "Complete" ["issues",2,"fields","issuelinks",1,"outwardIssue","fields","priority","self"] "https://jira.atlassian.com/rest/api/2/priority/3" ["issues",2,"fields","issuelinks",1,"outwardIssue","fields","priority","iconUrl"] "https://jira.atlassian.com/images/icons/priorities/major.png" ["issues",2,"fields","issuelinks",1,"outwardIssue","fields","priority","name"] "Major" ["issues",2,"fields","issuelinks",1,"outwardIssue","fields","priority","id"] "3" ["issues",2,"fields","issuelinks",1,"outwardIssue","fields","issuetype","self"] "https://jira.atlassian.com/rest/api/2/issuetype/16" ["issues",2,"fields","issuelinks",1,"outwardIssue","fields","issuetype","id"] "16" ["issues",2,"fields","issuelinks",1,"outwardIssue","fields","issuetype","description"] "" ["issues",2,"fields","issuelinks",1,"outwardIssue","fields","issuetype","iconUrl"] "https://jira.atlassian.com/images/icons/issuetypes/sales.png" ["issues",2,"fields","issuelinks",1,"outwardIssue","fields","issuetype","name"] "Story" ["issues",2,"fields","issuelinks",1,"outwardIssue","fields","issuetype","subtask"] false ["issues",2,"fields","customfield_13230"] null ["issues",2,"fields","customfield_10654"] null ["issues",2,"fields","customfield_14430"] null ["issues",2,"fields","status","self"] "https://jira.atlassian.com/rest/api/2/status/5" ["issues",2,"fields","status","description"] "A resolution has been taken, and it is awaiting verification by reporter. From here issues are either reopened, or are closed." ["issues",2,"fields","status","iconUrl"] "https://jira.atlassian.com/images/icons/statuses/resolved.png" ["issues",2,"fields","status","name"] "Resolved" ["issues",2,"fields","status","id"] "5" ["issues",2,"fields","status","statusCategory","self"] "https://jira.atlassian.com/rest/api/2/statuscategory/3" ["issues",2,"fields","status","statusCategory","id"] 3 ["issues",2,"fields","status","statusCategory","key"] "done" ["issues",2,"fields","status","statusCategory","colorName"] "green" ["issues",2,"fields","status","statusCategory","name"] "Complete" ["issues",2,"fields","customfield_10575"] null ["issues",2,"fields","customfield_10621"] null ["issues",2,"fields","workratio"] -1 ["issues",2,"fields","customfield_10530"] null ["issues",2,"fields","customfield_12832"] null ["issues",2,"fields","customfield_12833"] null ["issues",2,"fields","customfield_12531"] "Not Started" ["issues",2,"fields","project","self"] "https://jira.atlassian.com/rest/api/2/project/12200" ["issues",2,"fields","project","id"] "12200" ["issues",2,"fields","project","key"] "GHS" ["issues",2,"fields","project","name"] "JIRA Agile (formerly GreenHopper)" ["issues",2,"fields","project","avatarUrls","16x16"] "https://jira.atlassian.com/secure/projectavatar?size=xsmall&pid=12200&avatarId=41220" ["issues",2,"fields","project","avatarUrls","24x24"] "https://jira.atlassian.com/secure/projectavatar?size=small&pid=12200&avatarId=41220" ["issues",2,"fields","project","avatarUrls","32x32"] "https://jira.atlassian.com/secure/projectavatar?size=medium&pid=12200&avatarId=41220" ["issues",2,"fields","project","avatarUrls","48x48"] "https://jira.atlassian.com/secure/projectavatar?pid=12200&avatarId=41220" ["issues",2,"fields","project","projectCategory","self"] "https://jira.atlassian.com/rest/api/2/projectCategory/10050" ["issues",2,"fields","project","projectCategory","id"] "10050" ["issues",2,"fields","project","projectCategory","description"] "" ["issues",2,"fields","project","projectCategory","name"] "Atlassian Add-ons" ["issues",2,"fields","customfield_12931"] null ["issues",2,"fields","environment"] null ["issues",2,"fields","customfield_10723"] null ["issues",2,"fields","customfield_10630"] null ["issues",2,"fields","aggregateprogress","progress"] 0 ["issues",2,"fields","aggregateprogress","total"] 0 ["issues",2,"fields","lastViewed"] "2014-02-17T13:42:22.920+0000" ["issues",2,"fields","timeoriginalestimate"] null ["issues",2,"fields","customfield_10150",0] "nurquiza(nurquiza)" ["issues",2,"fields","customfield_11436"] "221048" ["issues",2,"fields","customfield_11435"] "221048" ["issues",2,"fields","customfield_11437"] "220737" ["issues",2,"fields","votes","self"] "https://jira.atlassian.com/rest/api/2/issue/GHS-10114/votes" ["issues",2,"fields","votes","votes"] 7 ["issues",2,"fields","votes","hasVoted"] false ["issues",2,"fields","customfield_10681"] null ["issues",2,"fields","customfield_10680"] null ["issues",2,"fields","resolution","self"] "https://jira.atlassian.com/rest/api/2/resolution/3" ["issues",2,"fields","resolution","id"] "3" ["issues",2,"fields","resolution","description"] "The problem is a duplicate of an existing issue." ["issues",2,"fields","resolution","name"] "Duplicate" ["issues",2,"fields","resolutiondate"] "2013-12-12T02:51:23.000+0000" ["issues",2,"fields","creator","self"] "https://jira.atlassian.com/rest/api/2/user?username=nurquiza" ["issues",2,"fields","creator","name"] "nurquiza" ["issues",2,"fields","creator","emailAddress"] "me@sample.com" ["issues",2,"fields","creator","avatarUrls","16x16"] "https://jira.atlassian.com/secure/useravatar?size=xsmall&avatarId=10612" ["issues",2,"fields","creator","avatarUrls","24x24"] "https://jira.atlassian.com/secure/useravatar?size=small&avatarId=10612" ["issues",2,"fields","creator","avatarUrls","32x32"] "https://jira.atlassian.com/secure/useravatar?size=medium&avatarId=10612" ["issues",2,"fields","creator","avatarUrls","48x48"] "https://jira.atlassian.com/secure/useravatar?avatarId=10612" ["issues",2,"fields","creator","displayName"] "Nestor Urquiza" ["issues",2,"fields","creator","active"] true ["issues",2,"fields","aggregatetimeoriginalestimate"] null ["issues",2,"fields","customfield_12430"] null ["issues",2,"fields","customfield_10161"] "true" ["issues",2,"fields","customfield_12431"] null ["issues",2,"fields","customfield_10160"] "7776000" ["issues",2,"fields","customfield_12432"] null ["issues",2,"fields","customfield_12433"] null ["issues",2,"fields","customfield_12434"] null ["issues",2,"fields","customfield_12831"] null ["issues",2,"fields","customfield_12435"] null ["issues",2,"fields","customfield_12830"] null ["issues",2,"fields","duedate"] null ["issues",2,"fields","customfield_12730"] null ["issues",2,"fields","watches","self"] "https://jira.atlassian.com/rest/api/2/issue/GHS-10114/watchers" ["issues",2,"fields","watches","watchCount"] 10 ["issues",2,"fields","watches","isWatching"] true ["issues",2,"fields","customfield_11631"] null ["issues",2,"fields","customfield_11130"] "236098" ["issues",2,"fields","customfield_10651"] null ["issues",2,"fields","customfield_10650"] null ["issues",2,"fields","customfield_11230"] null ["issues",2,"fields","customfield_10653"] null ["issues",2,"fields","assignee"] null ["issues",2,"fields","customfield_14130"] null ["issues",2,"fields","aggregatetimeestimate"] null ["issues",2,"fields","customfield_10510"] "tkotecki(tkotecki)" ["issues",2,"fields","versions",0,"self"] "https://jira.atlassian.com/rest/api/2/version/29118" ["issues",2,"fields","versions",0,"id"] "29118" ["issues",2,"fields","versions",0,"name"] "6.1" ["issues",2,"fields","versions",0,"archived"] false ["issues",2,"fields","versions",0,"released"] true ["issues",2,"fields","versions",0,"releaseDate"] "2012-12-07" ["issues",2,"fields","customfield_12131"] null ["issues",2,"fields","customfield_14330"] null ["issues",2,"fields","timeestimate"] null ["issues",2,"fields","customfield_11531"] null ["issues",2,"fields","customfield_14334"] "com.atlassian.servicedesk.sla.model.SLAValue@f9c070b" ["issues",2,"fields","customfield_11433"] "228358" ["issues",2,"fields","customfield_14333"] "com.atlassian.servicedesk.sla.model.SLAValue@4c926b2e" ["issues",2,"fields","customfield_14230"] null ["issues",2,"fields","customfield_11434"] "220801" ["issues",2,"fields","customfield_14332"] "com.atlassian.servicedesk.sla.model.SLAValue@20bac048" ["issues",2,"fields","customfield_11930"] null ["issues",2,"fields","customfield_11431"] "236698" ["issues",2,"fields","customfield_14331"] null ["issues",2,"fields","customfield_12130"] null ["issues",2,"fields","aggregatetimespent"] null ["issues",2,"fields","customfield_14335"] "com.atlassian.servicedesk.sla.model.SLAValue@44a5c7c1"

Saturday, February 22, 2014

Java Mail API traces

As usual "it depends". According to http://www.oracle.com/technetwork/java/faq-135477.html#debug:
Q: How do I debug my application that uses JavaMail APIs? A: Turn on session debugging by invoking the method setDebug(true) on the Session object in your code. That will cause debug information to be printed to the console, including a protocol trace. If you passed the System properties to the Session when you created it, you can simply run your program with java -Dmail.debug=true ... If you think that you found a bug in JavaMail, send us this trace along with a test case that reproduces the problem, the platform you are using, the version of the JDK you are using, and the name and version of the mail servers (IMAP, SMTP) that you are using.
That means it depends on your application details. For example if you are using Spring Framework you probably need something like the below and nothing else:
...
    <bean id="defaultMailSender" class="org.springframework.mail.javamail.JavaMailSenderImpl" >...
        <property name="javaMailProperties">...
            <props>...
                <prop key="mail.debug">true</prop>...
...

Friday, February 21, 2014

linux one liner to mount a windows share ( CIFS )

sudo mount -t cifs //domain/path/to/dir /mnt/dir -o 'username=user,password=password,domain=domain,rw,file_mode=0777,dir_mode=0777'

Solaris: where is the sudoers file?

Some would say /etc/opt/csw/sudoers, others /usr/local/etc/sudoers and I have even found it in /opt/csw/etc/sudoer. In fact I have seen people referring to /opt/sfw/etc/sudoers. The reality is that it depends on the environment.

The first thing you need to know is where is sudo command using a regular user:
$ which sudo
/opt/csw/bin/sudo
Now that we know where the bin is we need to locate the sudoers file for which you will run the below as root:
# truss /opt/csw/bin/sudo ls 2>&1 | grep sudoers
You will get something like:
stat64("/opt/csw/libexec/sudoers.so", 0x08047C70) = 0 xstat(2, "/opt/csw/libexec/sudoers.so", 0x08047090) = 0 resolvepath("/opt/csw/libexec/sudoers.so", "/opt/csw/libexec/sudoers.so", 1023) = 27 open("/opt/csw/libexec/sudoers.so", O_RDONLY) = 4 stat64("/opt/csw/libexec/sudoers.so", 0x08047C70) = 0 lstat64("/etc/opt/csw/sudoers", 0x08047BD0) = 0 open64("/etc/opt/csw/sudoers", O_RDONLY) = 4 openat(-3041965, "/etc/opt/csw/sudoers.d", O_RDONLY|O_NDELAY|O_LARGEFILE) = 5
So in this case the file is /etc/opt/csw/sudoers.

If you want to install sudo take a look at this post.

Thursday, February 20, 2014

Unix: file names containing non printing characters

In Unix a filename can contain non printing characters. Most likely the name will be the result of a mistake but in any case you might want to rename it, remove it, look at its content etc. So let us illustrate this issue. First let us build any filename including the "del" non printable character. So type "test", then press and release "CTRL-v", and finally press and release the "del" key. You will notice something like the below:
$ echo hello > test^?
$ ls test*
test
$ ls -b test*
test\177
$ ls -q test*
test?
To see the content you can take advantage of wild card support where the "?" means any single character matching just use:
$ cat test?
hello
To move it, remove it etc just use wildcards again. Note that "^?" means I have inserted a printable character as explained above:
$ mv test? test2^?
$ rm test2?
hello
You can also use the regular pipes and commands to come up with other ways to go around the issue like the below:
$ cat $(ls | grep test)
hello
$ rm $(ls | grep test)

Tuesday, February 18, 2014

libreoffice - terminate called after throwing an instance of 'com::sun::star::uno::RuntimeException'

The below works from command line:
$ libreoffice --headless --help
However from a nodejs server it was not working but instead failing with:
terminate called after throwing an instance of 'com::sun::star::uno::RuntimeException'
Every time a command does not work from an environment but it works from the command line you need to look at the environment. So in running the 'env' command and comparing the results from such environment and command line we should be able to determine what is different and what needs to be changed.

In this case we were caught by a HOME variable forced to an invalid path. However in reviewing the upstart script I realized we could be doing so much better. Upstart scripts should use no sudo nor using su but instead start-stop-daemo:
$ vi /etc/init/shell-server.conf
...
script
    #export HOME="/root"
    #exec sudo -u admin /usr/local/bin/node /opt/nodejs/shell-server.js
    #exec su -c "/usr/local/bin/node /opt/nodejs/shell-server.js" dev
    exec start-stop-daemon --start -c dev --exec /usr/local/bin/node /opt/nodejs/shell-server.js
end script
...

Monday, February 17, 2014

Simple bind failed: domain:port Root exception is javax.net.ssl.SSLHandshakeException ... PKIX path building failed

I have to explain this every so often. The below error is typical when you use self signed certificates and have not added them to the java keystore:
simple bind failed: domain:port Root exception is javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target]
A good practice is to add as alias the domain(if not also the port) for which you have added the authorization. That way you can easily find out if a given cert is missing, also you can see if the cert actually corresponds to the intended domain:
$ /opt/jdk/bin/keytool -list -v -keystore  $JAVA_HOME/jre/lib/security/cacerts|grep sample.com
Enter keystore password:  ********
Alias name: service1.sample.com
Owner: CN=service1.sample.com, OU=IT, O=Sample, L=Miami, ST=Florida, C=US
Issuer: CN=service1.sample.com, OU=IT, O=Sample, L=Miami, ST=Florida, C=US
Alias name: service2.sample.com
Owner: CN=service2.sample.com, OU=Domain Control Validated - RapidSSL(R), OU=See www.rapidssl.com/resources/cps (c)11, OU=GT52990491, O=service2.sample.com, C=US, SERIALNUMBER=mXKLdrX4UaRs2o6aqLNK3Rakjog/7peB
  DNSName: service2.sample.com
Alias name: service3.sample.com
Owner: EMAILADDRESS=admin@sample.com, CN=service3.sample.com, OU=Operations, O="Kaufman Rossin Fund Services, LLC", L=Miami, ST=FL, C=US
Issuer: EMAILADDRESS=admin@sample.com, CN=service3.sample.com, OU=Operations, O="Kaufman Rossin Fund Services, LLC", L=Miami, ST=FL, C=US

Friday, February 14, 2014

Excel to CSV in Linux servers - Best option is libreoffice so far

So far libreoffice looks like the best alternative to convert excel to csv in linux servers. Installation ( for ubuntu/debian )
$ sudo apt-get install libreoffice-calc
Here is how to convert a file and put the result in the /tmp directory
$ libreoffice --headless --convert-to csv /tmp/foo.xls --outdir /tmp/ 
convert /tmp/foo.xls -> /tmp/foo.csv using Text - txt - csv (StarCalc)
It works for XLS and XLSX as well.

Thursday, February 13, 2014

WIP limit versus batch size, fair comparison?

I thought I rather keep my statements in my blog for future reference. So here are they.

IMO WIP limit and batch size are orthogonal concepts.

They are both important and should be addressed in parallel. In practice I have identified two types of WIP limits so far like I posted in http://thinkinginsoftware.blogspot.com/2014/01/kanban-wip-limit-how-to.html

You need column WIP limits to balance demand versus throughput (queues and buffers).

You need Personal WIP limits because of the poor performance of most of human beings with multitasking.

You need small batches because otherwise predictability will suffer and so you will fail on delivering the promise of the Just In Time system.

Small batching is difficult without controlling variability, which is hard to control for most complex software projects because of the nature of knowledge work.

Balancing demand versus throughput is a necessary step, otherwise you will find yourself "guesstimating" forever. The agile team will try hard to make smaller batches naturally. However variability will impact their ability to do so. As variability is slowly controlled the small batches can flow through the system at a constant pace without the need to create new issues and going beyond the WIP limits, especially the Personal WIP .

So the idea that it is OK to have multiple small tickets assigned to the same responsible person is not good. The limited WIP is needed to guarantee continuous delivery. Otherwise the team will find out that the small issue coming out of an epic might transform into an epic itself. At that point we do not want to open 10 new issues out of that one and get them assigned to the only four developers in our small agile team or put them in a queue that will never get burnt. We need to negotiate the smallest possible part that can be done and which will provide value (MMF). Most likely bunch of those 10 potential tickets can be reviewed as brand new specifications. The team will happily continue delivering value at a predictable pace. The team and stakeholders will be happy with the outcome of the combination of small increments of code AND limitted WIP.

Sunday, February 09, 2014

CPU wait due to process blocked by IO : a story for VMware tools in Ubuntu guest

When the CPU waits, there is a problem. The server will be slow causing other issues. We were receiving high CPU wait percentages alerts from monit and we found the culprit to be VMWare tools:
$ while true; do date; ps auxf | awk '{if($8=="D") print $0;}'; sleep 1; done
Sun Feb  9 08:59:15 EST 2014
root      1233  0.0  0.0  87576   980 ?        D     2013  79:20 /usr/sbin/vmtoolsd
Sun Feb  9 08:59:16 EST 2014
root      1233  0.0  0.0  87576   980 ?        D     2013  79:20 /usr/sbin/vmtoolsd
Sun Feb  9 08:59:17 EST 2014
...
Clearly this process has been running for a while. I went ahead and traced the pid:
$ sudo strace -p1233
Process 1233 attached - interrupt to quit
But for minutes it stayed there, no traces. In a similar box I was able to see activity:
$ ps -ef|grep vm
root      1240     1  0  2013 ?        01:08:34 /usr/sbin/vmtoolsd
$ sudo strace -p1240
Process 1240 attached - interrupt to quit
restart_syscall(<... resuming interrupted call ...>) = 0
times({tms_utime=306343, tms_stime=105067, tms_cutime=4, tms_cstime=0}) = 2260068445
times({tms_utime=306343, tms_stime=105067, tms_cutime=4, tms_cstime=0}) = 2260068445
poll([{fd=4, events=POLLIN}, {fd=13, events=POLLIN}, {fd=13, events=POLLIN}, {fd=13, events=POLLIN}, {fd=13, events=POLLIN}, {fd=13, events=POLLIN}], 6, 100) = 0 (Timeout)
My first conclusion could have been that there is a deadlock in the below version of VMware Tools:
$ vmware-toolbox-cmd -v
8.6.5.16159 (build-821615)
$ sudo vmware-toolbox-cmd upgrade status
VMware Tools are up-to-date.
After a restart:
$ sudo service vmware-tools restart
vmware-tools stop/waiting
vmware-tools start/running
I got actually a new instance but the old instance was still there:
$ ps auxf|grep vm
root      1233  0.0  0.0  87576   980 ?        D     2013  79:20 /usr/sbin/vmtoolsd
root       953  0.1  0.0  87576  3976 ?        S    09:59   0:00 /usr/sbin/vmtoolsd
The old process won't get killed with SIGTERM nor SIGKILL so I had to restart the box. If this ever repeats it should be probably a good idea to notify vmware of a possible bug.

Saturday, February 08, 2014

Linux: timeout that slow command

Sometimes you need to deal with frozen applications which stay alive longer than expected consuming resources unnecessarily. In linux you just use the timeout command in those situations. For example, given the below command which takes 30 seconds to finish:
$ cat /tmp/slowCommand 
sleep 30
Here is how you make sure it does not take more than 5 seconds:
$ timeout 5 /tmp/slowCommand 

Saturday, February 01, 2014

Parse bash long options arguments

After reading a lot about this issue here is a proposal to parse the typical --longoption type of arguments from bash .
#!/bin/bash -e
# parse-long-options-arguments.sh

# Set a constant to hold the number of mandatory arguments
declare -r mandatory=3

basename=`basename $0`

# Use HEREDOC for multi line output. Note the backstick in a line after the closing EOF. You can include any type of quotation inside HEREDOC.
usage_text=`cat <<EOF
usage: $basename 
  --option1  Option #1
  --option2  Option #2
  --option3  Option #3
EOF
`
# Use the HEREDOC defined variable from a function that exits with a non zero code indicating failure
usage() {
 echo "$usage_text";
 exit 1;
}

# Loop through arguments thanks to shell exapansion builting "a colon". Avoid repetition using declare
argc=0
while :
do
        case $1 in
                --option1|\
                --option2|\
                --option3) var=${1:2}; shift; val=$1; shift; declare "$var=$val"; argc=$[$argc + 1];;
                *) if [[ ! -z $1 ]]; then usage; fi; break;;
        esac
done

# Check that mandatory parameters have been supplied
if [ $argc -ne $mandatory ]; then
  usage;
fi

# Sample output to illustrate this actually works
echo  "option1=$option1 option2=$option2 option3=$option3"
Here are some examples after invoking the script:
$ ./parse-long-options-arguments.sh
usage: parse-long-options-arguments.sh 
  --option1  Option #1
  --option2  Option #2
  --option3  Option #3
$ ./parse-long-options-arguments.sh --option1 1 --option2 2 --option3 3
option1=1 option2=2 option3=3
$ ./parse-long-options-arguments.sh --option1 1 --option2 "my option 2" --option3 3
option1=1 option2=my option 2 option3=3

Followers