Wednesday, September 21, 2011

Shell processes from Java and the infamous OutOfMemory

If you run shell processes from a Java Application Server you can really easy run out of heap memory because Java will invoke a fork() system call which will duplicate the parent memory (current JVM memory in use) to be able to run the child (the command you are trying to run). Apparently this is supposed to be corrected in version 8. Is it?

Here is a workaround for this issue. Basically it relies on a war file containing a servlet that accepts regular get parameters like "cmd" containing the complete command to run. It returns a JSON response with the stderr and stdout content. The war file will need to be deployed in an application server with really small footprint to prevent the child process from originating again an OutOfMemory.

Note that I on purpose do not use any special jar files because that way I keep memory usage to a minimum. Do not use a JSON parser for this, just build the response string yourself. Simpler is better ;-)

import java.io.*;
import javax.servlet.*;
import javax.servlet.http.*;

public class ShellServlet extends HttpServlet {
    public void doGet(HttpServletRequest request, HttpServletResponse response)
            throws ServletException, IOException {
        String cmd = request.getParameter("cmd");
        StringBuilder stdout = new StringBuilder();
        StringBuilder stderr = new StringBuilder();
        
        
        
        if( cmd != null && cmd.length() > 0 ) {
            try {
                Process process = new ProcessBuilder(cmd.split(" ")).start();
                
                InputStream is = process.getInputStream();
                InputStreamReader isr = new InputStreamReader(is);
                BufferedReader br = new BufferedReader(isr);
                String line;
                while ((line = br.readLine()) != null) {
                    stdout.append(line);
                }
                
                InputStream errorInputStream = process.getErrorStream();
                isr = new InputStreamReader(errorInputStream);
                br = new BufferedReader(isr);
                while ((line = br.readLine()) != null) {
                    stderr.append(line);
                }
            } catch (Exception e) {
                stderr.append(e.getMessage());
            }
            
        }
        
        StringBuilder sb = new StringBuilder();
        sb.append("{");
        sb.append("'stdout':'" + stdout + "'");
        sb.append(",");
        sb.append("'stderr':'" + stderr + "'");
        sb.append(",");
        sb.append("'cmd':'" + cmd + "'");
        sb.append("}");
        PrintWriter out = response.getWriter();
        out.println(sb);
    }
}

For a request like:
http://localhost:8088/shell-service/?cmd=ls%20-al%20notifier.png

You will get (provided notifier.png file exist in the working directory) something like:
{'stdout':'-rw-r--r--  1 nestor  staff  886 Oct 18  2010 notifier.png','stderr':'','cmd':'ls -al notifier.png'}

Probably your best bet in Java would be Jetty because it is really lightweight. Simply download the zip file, uncompress it and follow the below steps:
$ cd $JETTY_HOME
$ vi etc/jetty.xml #change port to 8088
$ java -jar start.jar & 

Edit: A better alternative for a server to run local commands would be to build a lighter nodejs service like I explain here.

No comments:

Followers