Async HTTP Client 1.2.0

Async HTTP Client 1.2.0

Intro

About month ago we released async-http-client 1.1.0.
During this month we received lots of contributions and user feedback.

We took all of this great feedback into consideration,

and here it is: lots of bug fixes and new features in async-http-client are ready to be released.

You can get it from Maven Central near you.

Highlights

To cut the long story short here are highlights of few new features.

• JDK based provider
  
  This release support a new AsyncHttpProvider based on JDK's URLConnection.
  This provider can be used in environments that aren't supporting NIO.
  AsyncHttpClient client = new AsyncHttpClient(new JDKAsyncHttpProvider(config));
  

• Authentication
  
  The library now support 3 authentication scheme: Basic, Digest and NTLM.
  

  RequestBuilder r = client.prepareGet(url)
             .setRealm((new Realm.RealmBuilder())
                        .setScheme(AuthScheme)
                        .setPrincipal(user)
                        .setPassword(admin)
                        .setUsePreemptiveAuth(true).build());

  AuthScheme can be BASIC, DIGEST or NTLM.

• SSL Tunneling support
  
  This release also adds support for SSL Tunneling.
  

  AsyncHttpClientConfig.Builder b = new AsyncHttpClientConfig.Builder();
  b.setFollowRedirects(true);
  
  ProxyServer ps = new ProxyServer(ProxyServer.Protocol.HTTPS, "127.0.0.1", port1);
  b.setProxyServer(ps);
  
  AsyncHttpClientConfig config = b.build();
  AsyncHttpClient asyncHttpClient = new AsyncHttpClient(config);
  
  RequestBuilder rb = new RequestBuilder("GET").setUrl(url);
  Future<Response> responseFuture = asyncHttpClient.executeRequest(rb.build(), new AsyncCompletionHandlerBase() {
  
  public void onThrowable(Throwable t) {
      log.debug(t);
  }
  
  @Override
  public Response onCompleted(Response response) throws Exception {
      return response;
  }
  });

• WebDav
  
  WebDav is now supported and can be used with two interfaces.
  • Low level
    

    AsyncHttpClient c = new AsyncHttpClient();
    
    Request mkcolRequest = new RequestBuilder("MKCOL").setUrl("http://127.0.0.1:8080/folder1").build();
    Response response =  c.executeRequest(mkcolRequest).get();
    
    Request putRequest = new RequestBuilder("PUT").setUrl("http://127.0.0.1:8080/folder1/Test.txt").setBody("this is a test").build();
    response =  c.executeRequest(putRequest).get();
    
    Request propFindRequest = new RequestBuilder("PROPFIND").setUrl("http://127.0.0.1:8080/folder1/Test.txt").build();
    response =  c.executeRequest(propFindRequest).get();
    
    // Check the response.
    response.getStatusCode(); //Will return 207
    response.getResponseBody().contains("<status>HTTP/1.1 200 OK</status>"));

  • High level
    
    You can also use a special AsyncHandler which construct a response
    based on the webdav response's code.
    

    AsyncHttpClient c = new AsyncHttpClient();
    
    Request mkcolRequest = new RequestBuilder("MKCOL").setUrl("http://127.0.0.1:8080/folder1").build();
    Response response =  c.executeRequest(mkcolRequest).get();
    assertEquals(response.getStatusCode(), 201);
    
    Request propFindRequest = new RequestBuilder("PROPFIND").setUrl("http://127.0.0.1:8080/folder1/").build();
    WebDavResponse webDavResponse =  c.executeRequest(propFindRequest, new WebDavCompletionHandlerBase<WebDavResponse>() {
    
    @Override
    public WebDavResponse onCompleted(WebDavResponse response) throws Exception {
        return response;
    }
    }).get();
    
    webDavResponse.getStatusCode(); // will return 200.

• Zero copy
  
  Sometimes you need to just download some content, with out
  copying/inspecting the bytes in memory, just download and store.
  

  File file = new File(url.toURI());
  
  File tmp = new File(System.getProperty("java.io.tmpdir") + File.separator + "zeroCopy.txt");
  final FileOutputStream stream = new FileOutputStream(tmp);
  // Upload a file using setBody(File);
  Future<Response> f = client.preparePost("http://127.0.0.1:" + port1 + "/").setBody(file).execute(new AsyncHandler<Response>() {
  public void onThrowable(Throwable t) {}
  
  public STATE onBodyPartReceived(HttpResponseBodyPart bodyPart) throws Exception {
      bodyPart.writeTo(stream);
      // Copy the bytes directly to a File. In our case it is an FileInputStream
          return STATE.CONTINUE;
  }
  
  public STATE onStatusReceived(HttpResponseStatus responseStatus) throws Exception {
      return STATE.CONTINUE;
  }
  
  public STATE onHeadersReceived(HttpResponseHeaders headers) throws Exception {
      return STATE.CONTINUE;
  }
  
  public Response onCompleted() throws Exception {
      return null;
  }
  });
  
  Response resp = f.get();
  stream.close();

• Automatic HTTPS redirects
  
  Another new feature is we do support redirect from http to HTTPS automatically:
  

  AsyncHttpClientConfig cg = new AsyncHttpClientConfig.Builder().setFollowRedirects(true).build();
  AsyncHttpClient c = new AsyncHttpClient(cg);
  
  Response response = c.preparePost(url).execute().get();

  If the server redirect us to https, we will follow the new url
  without any issue.

Changelog

• Bug fixes
  
  • Wrong import statement cause OSGi to fail
    
  • Parameter encoding issue
    Test case by Martin Traverso.
  • Connection Pool: Make sure the channel is still open
    
  • NettyAsyncProvider optimization: avoid invoking onStatusReceived and onHeadersReceived multiple time per transaction
    
  • Do not set a default Content-Type
    
  • fix redirect loop issue
    Patch provided by Jenny Loomis.
  • NettyAsyncHttpProvider.construct incorrectly appends default port to host header
    
  • NettyAsyncHttpProvider.construct allows empty Request-URIs in Request-Line
    
  • The current reaper may close connection event if they are active
    
  • Escape from redirect loops & allow more urls
    Patch provided by Jenny Loomis.
  • Http to Https redirect broken
    
  • Mixing secure and not secure connections on a single client causes issues
    
  • Support custom HTTP methods
    
  • Configurable Connection Pool
    
  • NettyAsyncHttpProvider.construct should default to using Accept: / header
    
  • AsyncHttpClientConfig.Builder leaks executors
    
  • onCompleted may get called twice
    
  • Race condition when the remote server close the connection and ConnectionsPool is used
    
  • Issue HTTPS tunneling when ProxyServer being set on the AsyncHttpConfig
    
  • Infinite redirects on some server because ":80" is appended to host name
    
  • Use full url (including host : port ) for request thru proxy
    
  • Channel must be closed when idle timeout fires
    
  • Per request follow redirects
    Patch provided by Erwan Loisant.
  • MaximumConnectionsTotal doesn't work
    
  • Fix Idle timeout
    Patch provided by felixtrepanier.

• New features
  
  • Add security support to ProxyServer
    
  • Add support for zero-copy file upload
    
  • Progress callbacks
    
  • Add CONNECT support
    
  • Add request timeout per request
    
  • Add support for zero-copy bytes to ResponseBodyPart
    
  • Make Response getter more robust, add helper
    
  • Add support for preemptive authentication configuration
    
  • Expose ByteBuffer in HttpResponseBodyPart to reduce byte copy
    
  • Add support for AsyncProviderConfig object that can be used to configure proprietary properties
    
  • SSL failure leaks channel
    Test case and a fix provided by Michael Pilquist.
  • Repackage the providers/* to providers/netty
    
  • Compile with 1.6+ and run with 1.5+
    
  • Add NTLM authentication support
    
  • Add WebDav support
    
  • Add support for a JDK Provider
    

Asynchronous HTTP Client for Clojure and Twitter Streaming API

http.async.client & Twitter Streaming API

http.async.client is a Clojure library based on Asynchronous Http Client for Java which runs on top of Netty Project.

So much for introduction, lets see some sample code that uses Twitter streaming api.

Importing:

(ns twitter-sample
 (:require [http.async.client :as c]
           [org.danlarkin.json :as j]))

Settings and helper to print twits as they fly by:

(def u "username")
(def p "password")

(defn print-user-and-text [s]
 (let [twit (j/decode-from-str s)
       user (:screen_name (:user twit))
       text (:text twit)]
   (println user "=>" text)))

Lets read statuses/sample:

(doseq [twit-str (c/string
                 (c/stream-seq :get "http://stream.twitter.com/1/statuses/sample.json"
                               :auth {:user u :password p}))]
 (print-user-and-text twit-str))

Or statuses/filter:

(doseq [twit-str (c/string
                  (c/stream-seq :post "http://stream.twitter.com/1/statuses/filter.json"
                                :body {"track" "basketball,football,baseball,footy,soccer"}
                                :auth {:user u :password p}))]
 (print-user-and-text twit-str))

Those two samples show also how to consume HTTP Streams as clojure.core/seq, a new feature that was introduced in v0.2.0.

You can also handle sequences via callbacks.

Requesting HTTP resources that are not streams is even simpler:

(let [response (c/GET "http://github.com/neotyk/http.async.client/")]
 (c/await response)
 (c/string response))

There is extensive documentation available on the github pages of the project, you can also find Changelog and API.

Big thanks go to Amsterdam Clojurians, this library received so much love and help from this group.
Kudos to Jeanfrancois Arcand for starting async-http-client recently reaching the 1.1.0 release.

I would love to hear from you what you think of this library.