Async HTTP Client 1.2.0

Async HTTP Client 1.2.0

Intro

About month ago we released async-http-client 1.1.0.
During this month we received lots of contributions and user feedback.

We took all of this great feedback into consideration,

and here it is: lots of bug fixes and new features in async-http-client are ready to be released.

You can get it from Maven Central near you.

Highlights

To cut the long story short here are highlights of few new features.

• JDK based provider
  
  This release support a new AsyncHttpProvider based on JDK's URLConnection.
  This provider can be used in environments that aren't supporting NIO.
  AsyncHttpClient client = new AsyncHttpClient(new JDKAsyncHttpProvider(config));
  

• Authentication
  
  The library now support 3 authentication scheme: Basic, Digest and NTLM.
  

  RequestBuilder r = client.prepareGet(url)
             .setRealm((new Realm.RealmBuilder())
                        .setScheme(AuthScheme)
                        .setPrincipal(user)
                        .setPassword(admin)
                        .setUsePreemptiveAuth(true).build());

  AuthScheme can be BASIC, DIGEST or NTLM.

• SSL Tunneling support
  
  This release also adds support for SSL Tunneling.
  

  AsyncHttpClientConfig.Builder b = new AsyncHttpClientConfig.Builder();
  b.setFollowRedirects(true);
  
  ProxyServer ps = new ProxyServer(ProxyServer.Protocol.HTTPS, "127.0.0.1", port1);
  b.setProxyServer(ps);
  
  AsyncHttpClientConfig config = b.build();
  AsyncHttpClient asyncHttpClient = new AsyncHttpClient(config);
  
  RequestBuilder rb = new RequestBuilder("GET").setUrl(url);
  Future<Response> responseFuture = asyncHttpClient.executeRequest(rb.build(), new AsyncCompletionHandlerBase() {
  
  public void onThrowable(Throwable t) {
      log.debug(t);
  }
  
  @Override
  public Response onCompleted(Response response) throws Exception {
      return response;
  }
  });

• WebDav
  
  WebDav is now supported and can be used with two interfaces.
  • Low level
    

    AsyncHttpClient c = new AsyncHttpClient();
    
    Request mkcolRequest = new RequestBuilder("MKCOL").setUrl("http://127.0.0.1:8080/folder1").build();
    Response response =  c.executeRequest(mkcolRequest).get();
    
    Request putRequest = new RequestBuilder("PUT").setUrl("http://127.0.0.1:8080/folder1/Test.txt").setBody("this is a test").build();
    response =  c.executeRequest(putRequest).get();
    
    Request propFindRequest = new RequestBuilder("PROPFIND").setUrl("http://127.0.0.1:8080/folder1/Test.txt").build();
    response =  c.executeRequest(propFindRequest).get();
    
    // Check the response.
    response.getStatusCode(); //Will return 207
    response.getResponseBody().contains("<status>HTTP/1.1 200 OK</status>"));

  • High level
    
    You can also use a special AsyncHandler which construct a response
    based on the webdav response's code.
    

    AsyncHttpClient c = new AsyncHttpClient();
    
    Request mkcolRequest = new RequestBuilder("MKCOL").setUrl("http://127.0.0.1:8080/folder1").build();
    Response response =  c.executeRequest(mkcolRequest).get();
    assertEquals(response.getStatusCode(), 201);
    
    Request propFindRequest = new RequestBuilder("PROPFIND").setUrl("http://127.0.0.1:8080/folder1/").build();
    WebDavResponse webDavResponse =  c.executeRequest(propFindRequest, new WebDavCompletionHandlerBase<WebDavResponse>() {
    
    @Override
    public WebDavResponse onCompleted(WebDavResponse response) throws Exception {
        return response;
    }
    }).get();
    
    webDavResponse.getStatusCode(); // will return 200.

• Zero copy
  
  Sometimes you need to just download some content, with out
  copying/inspecting the bytes in memory, just download and store.
  

  File file = new File(url.toURI());
  
  File tmp = new File(System.getProperty("java.io.tmpdir") + File.separator + "zeroCopy.txt");
  final FileOutputStream stream = new FileOutputStream(tmp);
  // Upload a file using setBody(File);
  Future<Response> f = client.preparePost("http://127.0.0.1:" + port1 + "/").setBody(file).execute(new AsyncHandler<Response>() {
  public void onThrowable(Throwable t) {}
  
  public STATE onBodyPartReceived(HttpResponseBodyPart bodyPart) throws Exception {
      bodyPart.writeTo(stream);
      // Copy the bytes directly to a File. In our case it is an FileInputStream
          return STATE.CONTINUE;
  }
  
  public STATE onStatusReceived(HttpResponseStatus responseStatus) throws Exception {
      return STATE.CONTINUE;
  }
  
  public STATE onHeadersReceived(HttpResponseHeaders headers) throws Exception {
      return STATE.CONTINUE;
  }
  
  public Response onCompleted() throws Exception {
      return null;
  }
  });
  
  Response resp = f.get();
  stream.close();

• Automatic HTTPS redirects
  
  Another new feature is we do support redirect from http to HTTPS automatically:
  

  AsyncHttpClientConfig cg = new AsyncHttpClientConfig.Builder().setFollowRedirects(true).build();
  AsyncHttpClient c = new AsyncHttpClient(cg);
  
  Response response = c.preparePost(url).execute().get();

  If the server redirect us to https, we will follow the new url
  without any issue.

Changelog

• Bug fixes
  
  • Wrong import statement cause OSGi to fail
    
  • Parameter encoding issue
    Test case by Martin Traverso.
  • Connection Pool: Make sure the channel is still open
    
  • NettyAsyncProvider optimization: avoid invoking onStatusReceived and onHeadersReceived multiple time per transaction
    
  • Do not set a default Content-Type
    
  • fix redirect loop issue
    Patch provided by Jenny Loomis.
  • NettyAsyncHttpProvider.construct incorrectly appends default port to host header
    
  • NettyAsyncHttpProvider.construct allows empty Request-URIs in Request-Line
    
  • The current reaper may close connection event if they are active
    
  • Escape from redirect loops & allow more urls
    Patch provided by Jenny Loomis.
  • Http to Https redirect broken
    
  • Mixing secure and not secure connections on a single client causes issues
    
  • Support custom HTTP methods
    
  • Configurable Connection Pool
    
  • NettyAsyncHttpProvider.construct should default to using Accept: / header
    
  • AsyncHttpClientConfig.Builder leaks executors
    
  • onCompleted may get called twice
    
  • Race condition when the remote server close the connection and ConnectionsPool is used
    
  • Issue HTTPS tunneling when ProxyServer being set on the AsyncHttpConfig
    
  • Infinite redirects on some server because ":80" is appended to host name
    
  • Use full url (including host : port ) for request thru proxy
    
  • Channel must be closed when idle timeout fires
    
  • Per request follow redirects
    Patch provided by Erwan Loisant.
  • MaximumConnectionsTotal doesn't work
    
  • Fix Idle timeout
    Patch provided by felixtrepanier.

• New features
  
  • Add security support to ProxyServer
    
  • Add support for zero-copy file upload
    
  • Progress callbacks
    
  • Add CONNECT support
    
  • Add request timeout per request
    
  • Add support for zero-copy bytes to ResponseBodyPart
    
  • Make Response getter more robust, add helper
    
  • Add support for preemptive authentication configuration
    
  • Expose ByteBuffer in HttpResponseBodyPart to reduce byte copy
    
  • Add support for AsyncProviderConfig object that can be used to configure proprietary properties
    
  • SSL failure leaks channel
    Test case and a fix provided by Michael Pilquist.
  • Repackage the providers/* to providers/netty
    
  • Compile with 1.6+ and run with 1.5+
    
  • Add NTLM authentication support
    
  • Add WebDav support
    
  • Add support for a JDK Provider
    

Asynchronous HTTP Client for Clojure and Twitter Streaming API

http.async.client & Twitter Streaming API

http.async.client is a Clojure library based on Asynchronous Http Client for Java which runs on top of Netty Project.

So much for introduction, lets see some sample code that uses Twitter streaming api.

Importing:

(ns twitter-sample
 (:require [http.async.client :as c]
           [org.danlarkin.json :as j]))

Settings and helper to print twits as they fly by:

(def u "username")
(def p "password")

(defn print-user-and-text [s]
 (let [twit (j/decode-from-str s)
       user (:screen_name (:user twit))
       text (:text twit)]
   (println user "=>" text)))

Lets read statuses/sample:

(doseq [twit-str (c/string
                 (c/stream-seq :get "http://stream.twitter.com/1/statuses/sample.json"
                               :auth {:user u :password p}))]
 (print-user-and-text twit-str))

Or statuses/filter:

(doseq [twit-str (c/string
                  (c/stream-seq :post "http://stream.twitter.com/1/statuses/filter.json"
                                :body {"track" "basketball,football,baseball,footy,soccer"}
                                :auth {:user u :password p}))]
 (print-user-and-text twit-str))

Those two samples show also how to consume HTTP Streams as clojure.core/seq, a new feature that was introduced in v0.2.0.

You can also handle sequences via callbacks.

Requesting HTTP resources that are not streams is even simpler:

(let [response (c/GET "http://github.com/neotyk/http.async.client/")]
 (c/await response)
 (c/string response))

There is extensive documentation available on the github pages of the project, you can also find Changelog and API.

Big thanks go to Amsterdam Clojurians, this library received so much love and help from this group.
Kudos to Jeanfrancois Arcand for starting async-http-client recently reaching the 1.1.0 release.

I would love to hear from you what you think of this library.

clojure.core/lazy-seq makes sense

clojure.core/lazy-seq makes sense

Few days ago on #clojure bartj asked about an infamous wiki entry.
I remember that while ago, when I tried to understand laziness in Clojure I stumbled upon this entry as well,
and it caused me only confusion, as it is totally outdated.
Let's try to shed some light on it.

Not lazy seq

When you construct a seq in non lazy way you should expect that all elements are realized at creation time.

(let [a (cons 1 (cons 2 nil))] (first a))
=> 1

Nothing special here. Let's introduce side effects to see what happens.

(let [a (cons
 (do (println "one") 1)
 (cons
  (do (println "two") 2)
  nil))]
(first a))
: one
: two
=> 1

As you expected all elements has been realized while constructing a, though we only needed the first element.
And that's exactly where laziness kicks in.

lazy-seq

First lets construct lazy seq with no side effects.

(let [a (lazy-seq (cons 1
           (lazy-seq (cons 2 nil))))]
(first a))
=> 1

So what is new? You have to type more and effect is the same. Why?
Adding side effects will help understanding it.

(let [a (lazy-seq (cons
           (do (println "one") 1)
           (lazy-seq (cons
                      (do (println "two") 2) nil))))]
(first a))
: one
=> 1

Now you should be able to see the benefit. Result is the same, except the whole sequence hasn't been realized,
except for the first element, as it was the only element that was needed.

Laziness is a very powerful tool:

• you can construct infinite sequences,
• have side effects create elements on demand, if seq construction is heavy, you might want to only issue creation of elements you'll use,
• it has same interface as non-lazy seq, code using it does not have to care about it.

To sum it up

(let [a (cons
 (do (println "one") 1)
 (cons
  (do (println "two") 2) nil))])
: one
: two
=> nil

We didn't need even single element but the whole seq was realized.

(let [a (lazy-seq (cons
           (do (println "one") 1)
           (lazy-seq (cons
                           (do (println "two") 2) nil))))])
=> nil

If not even a single element is needed, than not even a single element will be realized.
So there you have it clojure.core/lazy-seq explained.

clojure.core/str makes sense now

clojure.core/str makes sense

While reading excellent Programming Clojure by Stuart Halloway, instead just reading I started looking into how things are done by Clojure itself.
Yesterday it took me some time before I understood how things are done by clojure.core/str. This post is documentation of my findings.
Following modification of said function was very helpful in understanding how it works:
:tag metadata key is a symbol naming a class or a Class object that indicates the Java type of the object in the var, or its return value if the object is a fn..
Next are three parameters body pairs.

([] "")

No arguments (arity 0) will result in "" (empty string)

([#^Object x]
(if (nil? x) "" (. x (toString))))

One argument call (arity 1) will call java.lang.Object.toString method.
#^Object is a Type Hint.
if should not need much explanation.

([x & ys]
((fn [#^StringBuilder sb more]
(if more
(recur (. sb (append (str (first more)))) (next more))
(str sb)))
(new StringBuilder #^String (str x)) ys)))

This is where it starts to be interesting.
First unnamed function is defined that accepts StringBuilder sb and more params.
If more is true, that is not nil or false than call recursively self with new parameters

• sb with appended result of calling str with first element of more,
  
• more with elements after first.

If more is false, that in our context will happen only when more is nil, call sb's toString.
Next just defined function is getting called with new StringBuilder constructed from x as sb and ys as more.

It took me some time to realize that ys are turning into more.

So there you have it clojure.core/str explained.