SPCoast
Railroading on the Southern Pacific Coast

Java NBD Server

From SPCoast

Jump to: navigation, search

One of the basic components of Cloud Computing is a Network Block Device - its "disks" or "volumes" are the foundation upon which much of the rest of the infrastructure is built. As part of an effort to better understand the NBD protocols used in Linux, and to gather performance and usage data, I decided to build two versions of the Linux Network Block Device (NBD) server in Java using Apache Maven (a project management and build tool), one using Apache Mina and the other using JBoss' Netty. Both Mina and Netty are non-blocking IO and thread infrastructures written by Trustin Lee, one as part of the Apache Directory Service, and the other as a fork/follow-on after he moved on to JBoss. In a couple of week's worth of "jump in over my head and learn to swim", I managed to come up with working servers that pass all the tests I've thrown at tem. I'm a firm believer in starting with "slow, obvious and correct code" and using real world tests to avoid wasting time on a premature optimization quest...

Here's what I did:

Get and install Maven, use it to create a sample Application called "NBDServer" Edit the pom.xml file to add dependencies for mina-core, slf4j-api, slf4j-simple and log4j Create a set of Mina classes: Encoders and Decoders for both Requests and Responses, as well as the Requests and Responses themselves, a "factory" to create instances of the codecs, a Handler to implement the business logic of the protocol, and a Server to run the whole thing.

Repeat with Netty instead of Mina. Spend a good part of a week wrapping your brain around the differences between the two "related - but - different" frameworks. Spend the rest of it proving that simple things are really obvious in hindsight, but opaque as hell before then :-)

The source for both of these implementations is available under the Apache open source license:

The two are (not surprisingly) very similar - the main differences being that Apache's version 2.x and JBoss' version 3.x have chosen to go off in different directions from a common ancestor. Mina's documentation is patchy and out of date - many things have changed in their 2.x release, but much of the docs and examples are still firmly in the 1.x world. Netty's docs and examples are superb, but somewhat trivial; going from either ones examples to your own stack takes several leaps of faith, not to mention a large table to spread things out on and an attentive peer-programmer to bounce questions off of and help debug/understand the new concepts. I like both projects; the tie breaker for me is that Netty has better support for direct (zero copy) buffers; other than that, both are well written, robust and high performance frameworks.

One simple NBD test sequence I used was

  • create a 2G volume file
    • dd if=/dev/zero of=VOL1 bs=1M count=2000
  • start the server (R)un in eclipse...)
  • attach the volume to a linux client
    • nbd-client localhost 12345 /dev/nbd0
  • make a filesystem on it
    • mkfs.ext4 /dev/nbd0
  • mount it and play
    • mount /dev/nbd0 /mnt
    • echo "hello world" > /mnt/file"
    • umount /mnt
    • mount /dev/nbd0 /mnt
    • cat /mnt/file
    • umount /mnt
    • nbd-client -d /dev/nbd0