Imagine а person, who blocks (do nothing) after sending an email until he/she gets a response. An executive director would need a hundred blocking assistants in order to do his daily job. Quite expensive, though.
What happens, if a recipient is on vacation and cannot reply? Or answer after 3 days?
The director would need to set a timeout - to block for maximum 1 day, then report an error if there is no reply.
Crazy in the real world, but quite real in software.
Non-blocking Java Evolution
The funny situation described above is well known in Java world. In the old releases (<1.4) one thread must block and wait for socket response thus resulting in 100 threads waiting for 100 internet clients. At that time it was really challenging to implement smart client connection managers. It is far away from optimal to block threads waiting for remote responses, especially if the responses are slow. Threads are not ordinary Java objects - they consume a lot of memory and are bound to native resources.
With introduction of NIO package as part of Java 1.4, the low-level blocking problem was solved. The NIO functionality is far away from the normal web application programming, since web applications are using higher-level communication frameworks. A web application must return the whole response in doGet() method and the processing thread remains blocked.
Now, after introduction of Java 7 and servlets 3.0 the game changed completely - it is possible to code non-blocking web applications in Java. It is possible to receive a request in doGet(), schedule some job and return without response. The processing thread is free to handle new requests. Later, the scheduled job could answer to the client.
Several server-side Javascript frameworks (NodeJS,Vert.x) emerged recently, offering non-blocking API for web development. The point of this article is to show that there is no need to change the language in order to get your application non-blocking.
Yes – it is possible in Java now.
Example Application
Let's introduce an example web application which illustrates the situation. The application reads a XML from a legacy backend service, transforms it to JSON and return it to the client. The backend legacy service is "slow" and takes let's say 40 ms to respond. If the web server is configured to have X processing threads, than X parallel requests would block the server completely for 40 ms, and no other requests would be processed in that time.
No free processing threads, incoming requests are queued- this is a typical situation when thread bound request processing tries to handle huge number of I/O bound tasks.
MyBackendServlet.java
import org.json.JSONObject;
import org.json.XML;
import org.apache.http.HttpHost;
import org.apache.http.HttpResponse;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpGet;
HttpHost httpHost = new HttpHost("my_backend", 8080);
HttpClient httpClient; //init() method
protected void doGet(final HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException
{ HttpGet httpGet = new HttpGet("/backend/service"); //now block the thread to wait for http response HttpResponse backendResponse = httpClient.execute(host, httpGet); try { String backendXml = readResponseAsString( backendResponse ); JSONObject json = XML.toJSONObject(backendXml); response.getWriter().print(json); } catch(Exception x) {....}
}
Migrate to Non-Blocking Model
Now we will improve that application. First, we do not want to block and wait for backend response in the processing thread. Instead, we want to send a request to backend and register a callback to handle the backend response when it appears.
There is an asynchronous HTTP client from Apache ( http://hc.apache.org/httpcomponents-asyncclient-dev/index.html ) which can be used to unblock our example application.Very important - implementing asynchronous logic has no sense if not combined with asynchronous support in servlet 3.0. So our servlet is marked as asynchronous - class annotation is @WebServlet(urlPatterns={"/myservlet"}, asyncSupported=true)
MyAsyncBackendServlet.java
import org.json.JSONObject;
import org.json.XML;
import org.apache.http.HttpHost;
import org.apache.http.HttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.nio.client.HttpAsyncClient;
import org.apache.http.concurrent.FutureCallback;
import javax.servlet.AsyncContext;
HttpHost httpHost = new HttpHost("my_backend", 8080);
HttpAsyncClient httpAsyncClient; //init() method
protected void doGet(final HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException
{ final AsyncContext async = request.startAsync(request, response); HttpGet httpGet = new HttpGet("/backend/service"); Future<HttpResponse> fut = httpAsyncClient.execute(host, httpGet, new FutureCallback<HttpResponse>() { @Override public void completed(HttpResponse backendResponse) { try { String backendXml = readResponseAsString( backendResponse ); JSONObject json = XML.toJSONObject(backendXml); ServletResponse response =async.getResponse(); response.getWriter().print(json); async.complete(); } catch(Exception x) {....} }); ........... }
}
Let's have a detailed look on thread executions in order to understand better what happens in reality.
HTTP Processing Thread (Web Server)
1. Receives a HTTP request
2. Calls MyBackendServlet.doGet(..)
3. Calls HttpAsyncClient.execute(..)
4. HttpAsyncClient sends a HTTP request to backend and returns
5. Returns from doGet()
6. Released back to pool and can handle other incoming requests
40 ms later ......
I/O Dispatcher Thread ( Apache Async HTTP client )
1. Receives response from the backend service
2. Calls the FutureCallback which listens for that event
3. Gets the async context from web container and writes response to the client
4. Released back to pool
The benefits of asynchronous web programming are more obvious when we consider several HTTP requests executed in doGet(). Let's extend our example with calling three backend services instead of one. Of course, it makes sense to send all HTTP requests in parallel. The challange is again to avoid threads blocked on waiting. Let's see sample code.
MyAsyncMultiBackendServlet.java
import java.util.concurrent.CountDownLatch;
import org.json.JSONObject;
import org.json.XML;
import org.apache.http.HttpHost;
import org.apache.http.HttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.nio.client.HttpAsyncClient;
import org.apache.http.concurrent.FutureCallback;
import javax.servlet.AsyncContext;
protected void doGet(final HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException
{ final AsyncContext asyncContext = request.startAsync(request, response); HttpGet httpGet = new HttpGet("/backend/service"); HttpGet httpGet2 = new HttpGet("/backend/service2"); HttpGet httpGet3 = new HttpGet("/backend/service3"); final ArrayList<String> resultList = new ArrayList<String>(3); final CountDownLatch latch = new CountDownLatch(3); FutureCallback<HttpResponse> callback = new FutureCallback<HttpResponse>() { @Override public void completed(HttpResponse backendResponse) { String backendXml = readResponseAsString( backendResponse ); resultList.add(backendXml); latch.countDown(); if( latch.getCount()==0 ) { processResultList( resultList, asyncContext ); } } }; Future<HttpResponse> r = httpClient.execute(host, httpGet, callback); Future<HttpResponse> r2 = httpClient.execute(host, httpGet2, callback); Future<HttpResponse> r3 = httpClient.execute(host, httpGet3, callback);
}
private void processResultList(ArrayList<String> resultList, AsyncContext asyncContext)
{
//send to client via asyncContext
}
Asynchronous Database Processing
Well, here we cannot talk about non-blocking processing, since there is no non-blocking JDBC interface (except one special MySQL async connector -http://code.google.com/p/async-mysql-connector/). It is not very clear if non-blocking JDBC would be of big benefit for web application programming. A database connection is mapped to a database session in the database and the price of a blocked thread is neglectable compare to both connection + session. In some cases it is useful to execute SQL commands asynchronously - especially, if we trigger long-running database procedures.
In the example below we define a dedicated thread pool for executing long-running database tasks
MyAsyncDatabaseServlet.java
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.sql.Connection;
import javax.servlet.AsyncContext;
import javax.sql.DataSource;
import org.json.JSONObject;
import org.json.XML;
ExecutorService databaseProcedureExecutor = Executors.newFixedThreadPool(20);
DataSource dataSource;
protected void doGet(final HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException
{ final AsyncContext async = request.startAsync(request, response); databaseProcedureExecutor .submit(new Callable<Object>(){ public Object call() throws Exception { Connection conn = dataSource.getConnection(); CallableStatement cst = conn.prepareCall("call doSomeAnalytics()"); cst.execute(); async.getResponse().getWriter().print("Database procedure executed"); async.complete(); return this; } });
}
Other Non-Blocking Aspects
Major feature in Java 7 is introducing to file NIO - a new file system API with support for asynchronous file operations and directory events. For more information see AsynchronousFileChannel (http://docs.oracle.com/javase/7/docs/api/java/nio/channels/AsynchronousFileChannel.html) and WatchService (http://docs.oracle.com/javase/7/docs/api/java/nio/file/WatchService.html).
Asynchronous style web services are not new, they could be used exactly in the same way as the asynchronous HTTP calls above.
Thanks to servlets 3.0, it is possible to develop non-blocking and highly scalable web applications. Still we have to be somehow careful with non-blocking style programming.
From one side, it improves scalability, from the other side there is a certain CPU price for threads switching (context switches).
If the processing threads just schedule background tasks and exits, than there is a risk to schedule too much background tasks and overload the server. In the last example with the long-running database tasks the asynchronous thread pool executor is limited to 20 threads. If an unlimited thread pool is used, than it is not difficult to schedule thousands of threads, which leads to sever overload (too much threads) and database overload (too much tasks).
Too much asynchronous style makes code difficult to read, which affects supportability. Typically it makes sense for remote communication calls and long-running database procedures.