Thursday, October 7, 2010

Async HTTP Client 1.2.0

Async HTTP Client 1.2.0

Intro

About month ago we released async-http-client 1.1.0.
During this month we received lots of contributions and user feedback.
We took all of this great feedback into consideration,
and here it is: lots of bug fixes and new features in async-http-client are ready to be released.
You can get it from Maven Central near you.

Highlights

To cut the long story short here are highlights of few new features.
  • JDK based provider

    This release support a new AsyncHttpProvider based on JDK's URLConnection.
    This provider can be used in environments that aren't supporting NIO.
    AsyncHttpClient client = new AsyncHttpClient(new JDKAsyncHttpProvider(config));
  • Authentication

    The library now support 3 authentication scheme: Basic, Digest and NTLM.
    RequestBuilder r = client.prepareGet(url)
    .setRealm((new Realm.RealmBuilder())
    .setScheme(AuthScheme)
    .setPrincipal(user)
    .setPassword(admin)
    .setUsePreemptiveAuth(true).build());
    AuthScheme can be BASIC, DIGEST or NTLM.
  • SSL Tunneling support

    This release also adds support for SSL Tunneling.
    AsyncHttpClientConfig.Builder b = new AsyncHttpClientConfig.Builder();
    b.setFollowRedirects(true);

    ProxyServer ps = new ProxyServer(ProxyServer.Protocol.HTTPS, "127.0.0.1", port1);
    b.setProxyServer(ps);

    AsyncHttpClientConfig config = b.build();
    AsyncHttpClient asyncHttpClient = new AsyncHttpClient(config);

    RequestBuilder rb = new RequestBuilder("GET").setUrl(url);
    Future<Response> responseFuture = asyncHttpClient.executeRequest(rb.build(), new AsyncCompletionHandlerBase() {

    public void onThrowable(Throwable t) {
    log.debug(t);
    }

    @Override
    public Response onCompleted(Response response) throws Exception {
    return response;
    }
    });
  • WebDav

    WebDav is now supported and can be used with two interfaces.
    • Low level
      AsyncHttpClient c = new AsyncHttpClient();

      Request mkcolRequest = new RequestBuilder("MKCOL").setUrl("http://127.0.0.1:8080/folder1").build();
      Response response = c.executeRequest(mkcolRequest).get();

      Request putRequest = new RequestBuilder("PUT").setUrl("http://127.0.0.1:8080/folder1/Test.txt").setBody("this is a test").build();
      response = c.executeRequest(putRequest).get();

      Request propFindRequest = new RequestBuilder("PROPFIND").setUrl("http://127.0.0.1:8080/folder1/Test.txt").build();
      response = c.executeRequest(propFindRequest).get();

      // Check the response.
      response.getStatusCode(); //Will return 207
      response.getResponseBody().contains("<status>HTTP/1.1 200 OK</status>"));
    • High level

      You can also use a special AsyncHandler which construct a response
      based on the webdav response's code.
      AsyncHttpClient c = new AsyncHttpClient();

      Request mkcolRequest = new RequestBuilder("MKCOL").setUrl("http://127.0.0.1:8080/folder1").build();
      Response response = c.executeRequest(mkcolRequest).get();
      assertEquals(response.getStatusCode(), 201);

      Request propFindRequest = new RequestBuilder("PROPFIND").setUrl("http://127.0.0.1:8080/folder1/").build();
      WebDavResponse webDavResponse = c.executeRequest(propFindRequest, new WebDavCompletionHandlerBase<WebDavResponse>() {

      @Override
      public WebDavResponse onCompleted(WebDavResponse response) throws Exception {
      return response;
      }
      }).get();

      webDavResponse.getStatusCode(); // will return 200.
  • Zero copy

    Sometimes you need to just download some content, with out
    copying/inspecting the bytes in memory, just download and store.
    File file = new File(url.toURI());

    File tmp = new File(System.getProperty("java.io.tmpdir") + File.separator + "zeroCopy.txt");
    final FileOutputStream stream = new FileOutputStream(tmp);
    // Upload a file using setBody(File);
    Future<Response> f = client.preparePost("http://127.0.0.1:" + port1 + "/").setBody(file).execute(new AsyncHandler<Response>() {
    public void onThrowable(Throwable t) {}

    public STATE onBodyPartReceived(HttpResponseBodyPart bodyPart) throws Exception {
    bodyPart.writeTo(stream);
    // Copy the bytes directly to a File. In our case it is an FileInputStream
    return STATE.CONTINUE;
    }

    public STATE onStatusReceived(HttpResponseStatus responseStatus) throws Exception {
    return STATE.CONTINUE;
    }

    public STATE onHeadersReceived(HttpResponseHeaders headers) throws Exception {
    return STATE.CONTINUE;
    }

    public Response onCompleted() throws Exception {
    return null;
    }
    });

    Response resp = f.get();
    stream.close();
  • Automatic HTTPS redirects

    Another new feature is we do support redirect from http to HTTPS automatically:
    AsyncHttpClientConfig cg = new AsyncHttpClientConfig.Builder().setFollowRedirects(true).build();
    AsyncHttpClient c = new AsyncHttpClient(cg);

    Response response = c.preparePost(url).execute().get();
    If the server redirect us to https, we will follow the new url
    without any issue.

Changelog

Friday, September 10, 2010

Asynchronous HTTP Client for Clojure and Twitter Streaming API

http.async.client & Twitter Streaming API

http.async.client is a Clojure library based on Asynchronous Http Client for Java which runs on top of Netty Project.

So much for introduction, lets see some sample code that uses Twitter streaming api.

Importing:

(ns twitter-sample
(:require [http.async.client :as c]
[org.danlarkin.json :as j]))

Settings and helper to print twits as they fly by:

(def u "username")
(def p "password")

(defn print-user-and-text [s]
(let [twit (j/decode-from-str s)
user (:screen_name (:user twit))
text (:text twit)]
(println user "=>" text)))

Lets read statuses/sample:

(doseq [twit-str (c/string
(c/stream-seq :get "http://stream.twitter.com/1/statuses/sample.json"
:auth {:user u :password p}))]
(print-user-and-text twit-str))

Or statuses/filter:

(doseq [twit-str (c/string
(c/stream-seq :post "http://stream.twitter.com/1/statuses/filter.json"
:body {"track" "basketball,football,baseball,footy,soccer"}
:auth {:user u :password p}))]
(print-user-and-text twit-str))

Those two samples show also how to consume HTTP Streams as clojure.core/seq, a new feature that was introduced in v0.2.0.

You can also handle sequences via callbacks.

Requesting HTTP resources that are not streams is even simpler:

(let [response (c/GET "http://github.com/neotyk/http.async.client/")]
(c/await response)
(c/string response))

There is extensive documentation available on the github pages of the project, you can also find Changelog and API.

Big thanks go to Amsterdam Clojurians, this library received so much love and help from this group.
Kudos to Jeanfrancois Arcand for starting async-http-client recently reaching the 1.1.0 release.

I would love to hear from you what you think of this library.

Monday, June 7, 2010

clojure.core/lazy-seq makes sense

clojure.core/lazy-seq makes sense

Few days ago on #clojure bartj asked about an infamous wiki entry.
I remember that while ago, when I tried to understand laziness in Clojure I stumbled upon this entry as well,
and it caused me only confusion, as it is totally outdated.
Let's try to shed some light on it.

Not lazy seq

When you construct a seq in non lazy way you should expect that all elements are realized at creation time.
(let [a (cons 1 (cons 2 nil))] (first a))
=> 1
Nothing special here. Let's introduce side effects to see what happens.
(let [a (cons
(do (println "one") 1)
(cons
(do (println "two") 2)
nil))]
(first a))
: one
: two

=> 1
As you expected all elements has been realized while constructing a, though we only needed the first element.
And that's exactly where laziness kicks in.

lazy-seq

First lets construct lazy seq with no side effects.
(let [a (lazy-seq (cons 1
(lazy-seq (cons 2 nil))))]
(first a))
=> 1
So what is new? You have to type more and effect is the same. Why?
Adding side effects will help understanding it.
(let [a (lazy-seq (cons
(do (println "one") 1)
(lazy-seq (cons
(do (println "two") 2) nil))))]
(first a))
: one
=> 1
Now you should be able to see the benefit. Result is the same, except the whole sequence hasn't been realized,
except for the first element, as it was the only element that was needed.

Laziness is a very powerful tool:
  • you can construct infinite sequences,
  • have side effects create elements on demand, if seq construction is heavy, you might want to only issue creation of elements you'll use,
  • it has same interface as non-lazy seq, code using it does not have to care about it.

To sum it up

(let [a (cons
(do (println "one") 1)
(cons
(do (println "two") 2) nil))])
: one
: two

=> nil
We didn't need even single element but the whole seq was realized.
(let [a (lazy-seq (cons
(do (println "one") 1)
(lazy-seq (cons
(do (println "two") 2) nil))))])
=> nil
If not even a single element is needed, than not even a single element will be realized.
So there you have it clojure.core/lazy-seq explained.

Monday, January 11, 2010

clojure.core/str makes sense now

clojure.core/str makes sense

While reading excellent Programming Clojure by Stuart Halloway, instead just reading I started looking into how things are done by Clojure itself.
Yesterday it took me some time before I understood how things are done by clojure.core/str. This post is documentation of my findings.
Following modification of said function was very helpful in understanding how it works:
:tag metadata key is a symbol naming a class or a Class object that indicates the Java type of the object in the var, or its return value if the object is a fn..
Next are three parameters body pairs.
([] "")
No arguments (arity 0) will result in "" (empty string)

([#^Object x]
(if (nil? x) "" (. x (toString))))
One argument call (arity 1) will call java.lang.Object.toString method.
#^Object is a Type Hint.
if should not need much explanation.

([x & ys]
((fn [#^StringBuilder sb more]
(if more
(recur (. sb (append (str (first more)))) (next more))
(str sb)))
(new StringBuilder #^String (str x)) ys)))
This is where it starts to be interesting.
First unnamed function is defined that accepts StringBuilder sb and more params.
If more is true, that is not nil or false than call recursively self with new parameters
  • sb with appended result of calling str with first element of more,
  • more with elements after first.
If more is false, that in our context will happen only when more is nil, call sb's toString.
Next just defined function is getting called with new StringBuilder constructed from x as sb and ys as more.

It took me some time to realize that ys are turning into more.

So there you have it clojure.core/str explained.