GORM support for Hadoop HBase

  • Tags : nosql, hadoop, hbase
  • Latest : 0.2.4
  • Last Updated: 26 June 2010
  • Grails version : 1.3.2
  • Authors : null
4 votes
Dependency :
compile ":gorm-hbase:0.2.4"

Documentation

Summary

Installation

Pre-requisites

Install

  • Remove the default Hibernate plug-in
grails uninstall-plugin hibernate
  • Install the GORM-HBase plugin
grails install-plugin gorm-hbase

Required Configuration

  • HBase CLASSPATH element in hbase-env.sh to include the transactional jar file, e.g.
export HBASE_CLASSPATH=$HBASE_INSTALL/contrib/transactional/hbase-0.20.4-transactional.jar
  • If Grails is not running on the same machine as your ZooKeeper Master configure <grails-appl-base-dir>/conf/hbase-site.xml to point to the name or address of the machine running your ZooKeeper Master,
<property>
    <name>hbase.zookeeper.quorum</name>
    <value>localhost</value>
    <description>Comma separated list of servers in the ZooKeeper Quorum.
    For example, "host1.mydomain.com,host2.mydomain.com,host3.mydomain.com".
    By default this is set to localhost for local and pseudo-distributed modes
    of operation. For a fully-distributed setup, this should be set to a full
    list of ZooKeeper quorum servers. If HBASE_MANAGES_ZK is set in hbase-env.sh
    this is the list of servers which we will start/stop ZooKeeper on.
    </description>
</property>

Optional Configuration

  • Edit <grails-appl-base-dir>/conf/DataSource.groovy to control whether HBase tables are,
    • either dropped (if they already exist) then created, or, left untouched, and whether
    • sorted queries are acticated by enabling indexing on the HBase datasource when the application or tests start.
Only the dbCreate element is honored. By default both tables and data are not dropped, created, or updated in any way unless this one option is coded,

dbCreate = "create-drop"

If sorting is required on dynamic finder queries then indexing must be activated as shown in this example,

dataSource {
        dbCreate = "update"
        indexed = true
}
  • Edit <grails-appl-base-dir>/conf/Config.groovy to control logging. Until you are comfortable with the behavior of this plug-in it is recommended you at least configure info level debugging,
info   'org.grails.hbase'

Description

Overview

This plug-in persists domain class data to a Hadoop HBase 'nosql' database and implements the GORM api to facilitate retrieval of persisted data.

Release 0.2.4

Upgraded to run on the following,
  • Grails 1.3.2

Release 0.2.3

Upgraded to run on the following,
  • Grails 1.3.1
  • Hadoop 0.20.2
  • HBase 0.20.4

Release 0.2.2

Initial SQL-like capabilities provided in extended,
  • findAll() support
  • find() support

Release 0.2.1

  • Dynamic Finders
    • Ability to define finder filters as native HBase filters
    • find() support
    • Multiple And/Or operators in method names now understood by dynamic finders
    • 'Like' and 'NotLike' are now valid operators in dynamic finder names and use regular expressions
    • 'Operator.LIKE' and 'Operator.NOT_LIKE' are supported by FinderFilters

Release 0.2

  • Dynamic Finders,
    • findAll()
    • findAllBy()
    • findBy()
  • Improved consistency with the default Hibernate based GORM implementation,
    • Return null when get() fails to find an object
    • Return correct values from save()
    • Return null from delete()
    • Support getters on one-to-many association
  • Support of standard HBase 0.20.3 instead of custom version of 0.20.0
  • All Grails supported basic types are persisted in an upgradeable manner

Release 0.1

  • DataSource.groovy - the dbCreate value can be set to "create-drop" to drop existing HBase tables and create new tables. Any other setting should leave the HBase database untouched.
  • A subset of GORM related Domain class methods and properties,
    • 'count()' with no arguments
    • 'delete()' with no arguments
    • 'get()' with the id argument
    • 'hasErrors()' with no arguments
    • 'list()' with support for max , offset and sort arguments
    • 'save()' with no arguments
    • the errors property
  • Optimistic locking
  • Lazy loading
  • Associations between domain classes,
    • One to One
    • One to Many
  • All Grails supported basic types can be persisted
  • Secondary index support using HBase Indexed Tables
  • Table name spaces - a similar idea to schema names in RDBMS - whereby each table created has a name prefixed with an identifier constructed from the Grails application name

findAll()

Instead of the Hibernate Query Language (HQL) being supported this plug-in provides two alternate options,
  • Small number of api classes
  • HBase filters
  • SQL-like capabilities
The api classes, aka built-in filters, are as follows,
DynamicFinderFilter, FinderFilter, FinderFilterList and Operator

For example,

// The domain class

class Book { String title String author Date releaseDate }

// everything def results = Book.findAll()

// a basic example using built-in filter support import org.grails.hbase.api.finders.FinderFilter import org.grails.hbase.api.finders.FinderFilterList import org.grails.hbase.api.finders.Operator

// defaults to 'Operator.EQUAL', DynamicFinderFilter filter = new FinderFilter('author', 'Dan Brown') results = Book.findAll(filter)

// the same finder with a native HBase filter, import org.apache.hadoop.hbase.filter.Filter import org.apache.hadoop.hbase.filter.SingleColumnValueFilter import org.apache.hadoop.hbase.filter.CompareFilter.CompareOp import org.apache.hadoop.hbase.util.Bytes

import org.grails.hbase.store.Constants import org.grails.hbase.util.HBaseNameUtils

def columnName = Bytes.toBytes(HBaseNameUtils.getDomainColumnName('name')) def value = Bytes.toBytes('Dan Brown') Filter filter = new SingleColumnValueFilter(Constants.DEFAULT_DATA_FAMILY, columnName, CompareOp.EQUAL, value)

results = Book.findAll(filter)

// examples with max/offset usage

DynamicFinderFilter filter = new FinderFilter('author', Operator.NOT_EQUAL, 'Dan Brown')

// first 10 books without Dan Brown as the author results = Book.findAll(filter, [max:10])

// 10 books without Dan Brown as the author starting from the 5th book results = Book.findAll(filter, [max:10, offset:5])

// 10 books from Dan Brown staring from 5th book ordered by release date DynamicFinderFilter filter = new FinderFilter('author', 'Dan Brown') results = Book.findAll(filter, [max:10, offset:5, sort:'releaseDate'])

// example with multiple filters

// all books written by Dan Brown or J K Rowling DynamicFinderFilter filterList = new FinderFilterList(Operator.OR)

DynamicFinderFilter filter1 = new Filter('author', 'Dan Brown') filterList.addFilter(filter1)

DynamicFinderFilter filter2 = new Filter('author', 'J K Rowling') filterList.addFilter(filter12)

results = Book.findAll(filterList)

// examples using SQl-like syntax // All books written by Terry Pratchett def results = Book.findAll("author = 'Terry Pratchett’", [sort:'title'])

// All books written by Terry Pratchett or Dan Brown results = Book.findAll("author = 'Terry Pratchett’ or author = ‘Dan Brown’“)

// All books written by Terry Pratchett from May 31st, 2005 results = Book.findAll("author = 'Terry Pratchett’ and releaseDate >= ‘2005-05-31’“)

// All books written by Dan Brown or books written in 2000 by Terry Pratchett results = Book.findAll(“”” author = ‘Dan Brown’ or ( author = 'Terry Pratchett’ and releaseDate >= ‘2000-01-01’ and releaseDate <= ‘2000-12-31’ ) “””)

Operators supported by FinderFilter,

EQUAL, NOT_EQUAL, GREATER, GREATER_OR_EQUAL, LESS, LESS_OR_EQUAL, LIKE and NOT_LIKE
Operators supported by FinderFilterList,
AND and OR

Operators supported by SQL-like queries,

=, <, <=, >, >=, !=

find()

The same syntax as findAll() above, the only difference is that this method returns just one object not a list of objects,
// Find Dan Brown's third book with a single filter, 
   DynamicFinderFilter filter = new FinderFilter('author', 'Dan Brown') 
   results = Book.find(filter, [offset:3])

findAllBy()

This finder is very similar to the default Hibernate based implementation. For example,
// The domain class
class Book {
   String title
   String author
   Date releaseDate
}

// basic examples def results = Book.findAllByAuthor('Dan Brown', [max:10, sort:'title', offset:2] ) results = Book.findAllByTitleNotEqualAndAuthor('Digital Fortress', 'Dan Brown') results = Book.findAllByReleaseDateGreaterThanEquals(firstDate)

// example with Like using a regular expression // All books written by an author called Brown Book.findAllByAuthorLike(".+Brown")

// All books not written by authors called Brown or Rowling Book.findAllByAuthorNotLike(".+(Brown|Rowling)")

// multiple and/or conditions // 'and' is resolved before 'or' (c.f. Java) so this finds all books released in 2001 by J K Rowling books or // in 2002 by Dan Brown Book.findAllByAuthorAndReleaseDateOrAuthorAndReleaseDate("J K Rowling", date2001, "Dan Brown", date 2002)

findBy()

This finder is very similar to the default Hibernate based implementation. Effectively it is exactly the same as the findAllBy() finder above,

// The domain class
class Book {
   String title
   String author
   Date releaseDate
}

Book result = Book.findByAuthor('Dan Brown', [max:10, sort:'title', offset:2] ) result = Book.findByTitleNotEqualAndAuthor('Digital Fortress', 'Dan Brown') result = Book.findByReleaseDateGreaterThanEquals(firstDate)

Contact

Email via redcoat.systems@gmail.com