Sunday, October 21, 2012

Embedded MySQL in Java With Connector/MXJ and 64-bit Linux

http://blog.palominolabs.com/2011/10/03/embedded-mysql-on-java-with-connectormxj-and-64-bit-linux/


This really helped me as I had to write some integration test which uses MySQL import feature and using other in-memory DB was not a good option.

Another tip: if you need to wait in an integration/unit test, you can use CountDownLatch.

Thursday, October 4, 2012

ICONIX Process for OOAD

The ICONIX Process is an open, free-to-use object modeling process. It’s minimal, use case driven, and agile. The process focuses on the area that lies in between use cases and code. Its emphasis is on what needs to happen at that point in the life cycle where you’re starting out: you have a start on some use cases, and now you need to do good analysis and design.

See more at http://iconixprocess.com/iconix-process/


Wednesday, April 11, 2012

Guava Event Bus Example


package guava;

import java.io.File;

import org.apache.log4j.Logger;

import com.google.common.eventbus.EventBus;
import com.google.common.eventbus.Subscribe;

public class FileSizeApp {
private static final Logger LOG = Logger.getLogger(FileSizeApp.class);

private final EventBus eventBus = new EventBus("FileSizeEventBus");

private long filesPending;
private long totalSize;

private long start = System.nanoTime();

private void process(File file) {
eventBus.register(this);

eventBus.post(new ProcessFileEvent(file));
}

@Subscribe
public void processFile(ProcessFileEvent e) {
filesPending++;
eventBus.post(new CalculateSizeEvent(e.getFile()));

if (LOG.isDebugEnabled()) {
LOG.debug(filesPending + ": " + e.getFile().getAbsolutePath());
}
}

@Subscribe
public void calculateSize(CalculateSizeEvent e) {
long size = 0;
File file = e.getFile();

if (file.isFile()) {
size = file.length();
} else {
File[] children = file.listFiles();

if (children != null) {
for (File child : children)
if (child.isFile()) {
size += child.length();
} else {
eventBus.post(new ProcessFileEvent(child));
}
}
}

eventBus.post(new FileSizeEvent(size));
}

@Subscribe
public void fileSize(FileSizeEvent e) {
totalSize += e.getSize();
filesPending--;

LOG.info(filesPending + ": " + e.getSize() + ", " + totalSize);

if (filesPending == 0) {
System.out.println("Total size: " + totalSize);
System.out.println("Time taken (s): " + (System.nanoTime() - start) / 1.0e9);

System.exit(0);
}
}

private static class ProcessFileEvent {
private final File file;

public ProcessFileEvent(File file) {
this.file = file;
}

public File getFile() {
return file;
}
}

private static class CalculateSizeEvent {
private final File file;

public CalculateSizeEvent(File file) {
this.file = file;
}

public File getFile() {
return file;
}
}

private static class FileSizeEvent {
private final long size;

public FileSizeEvent(long size) {
this.size = size;
}

public long getSize() {
return size;
}
}

public static void main(String[] args) {
final String fileName = args[0];
final FileSizeApp app = new FileSizeApp();

System.out.println("Calculating file size for: " + fileName);
app.process(new File(fileName));
}
}

Sunday, April 8, 2012

Akka 2.0 Event Bus Example


package practice

import akka.actor._
import akka.routing._
import akka.event._

case class FileToProcess(file: java.io.File)
case class FileSize(size: Long)
case class CalculateSize(file: java.io.File)

class DirWalker(system: ActorSystem) extends Actor {
  def receive = {
    case CalculateSize(file) =>
      if (file.isFile()) {
        system.eventStream.publish(FileSize(file.length()))
      } else {
        var size = 0L
        val children = file.listFiles()

        if (children != null) {
          for (child <- children) {
            if (child.isFile()) {
              size += child.length()
            } else {
              system.eventStream.publish(FileToProcess(child))
            }
          }
        }

        system.eventStream.publish(FileSize(size))
      }
  }
}

class SizeAggregator(system: ActorSystem) extends Actor {
  val start = System.nanoTime()

  var totalSize = 0L
  var filesPendingForProcess = 0L

  def receive = {
    case FileToProcess(file) =>
      filesPendingForProcess += 1

      system.eventStream.publish(CalculateSize(file))

    case FileSize(size) =>
      totalSize += size
      filesPendingForProcess -= 1
   
      if (filesPendingForProcess == 0) {
        println("Total Size: " + totalSize)
        println("Time taken (s): " + (System.nanoTime() - start) / 1.0e9)

        system.shutdown()
      }
  }
}

object FileSizeApp {
  def main(args: Array[String]): Unit = {

    val system = ActorSystem("FileSizeApp")

    val sizeAggregator = system.actorOf(Props(new SizeAggregator(system)))
    val dirWalkerRouter = system.actorOf(Props(new DirWalker(system)).withRouter(RoundRobinRouter(15)))

    system.eventStream.subscribe(sizeAggregator, classOf[FileSize])
    system.eventStream.subscribe(sizeAggregator, classOf[FileToProcess])

    system.eventStream.subscribe(dirWalkerRouter, classOf[CalculateSize])

    sizeAggregator ! FileToProcess(new java.io.File(args(0)))
  }
}

Tuesday, March 13, 2012

Bookmarks related to Cloudera Training for Apache Pig

Reading and Writing data with Pig

PigLatin in-depth


Debugging Pig scripts


Best Practices for Pig

Sunday, March 11, 2012

OOD Principles & Patterns

Fig 1. OO Design Principles

Fig 2. Design Patterns

Sunday, February 26, 2012

Aperture + Lucene

Though Lucene gives most of the features for creating index and building a search application, it does not provide out of the box crawler for pulling contents from the repository. I found Aperture very good for crawling and indexing. Incremental indexing works great with persistent RDF repository is used.

Once you get an overview of Aperture general structure, you can understand that all you need is a Lucene handler and the proper configuration for the Aperture web crawler.

Listing: LuceneHandler.java


Listing: IntranetCrawler.java

Cassandra - why?

Cassandra Client Requests

Cassandra Snitches

Cassandra Replication