Big Data and BIRT in San Francisco and New York

Actuate is sponsoring a customer day in San Francisco on November 8 and in New York on November 14.  This one day event is free to all and will focus on Big Data and Data Visualization.  Shaku Atre will be doing a keynote on Big Data entitled “Big Data in Motion and Humongous Data at Rest”.   Stephen Few will be doing a keynote on Data Visualization entitled “Telling Compelling Stories with Numbers” in San Francisco and Geoff McGhee will be presenting “Telling Stories with Data” in New York.  There will also be breakout sessions focused on Dashboards and Scorecards, Tips and Tricks, and Big Data Visualizations.   I will be doing a talk in the Tip and Tricks breakout entitled “BIRT Essentials: Tips and Tricks Every Developer Should Know”.  In this talk I will cover some of the most effective but least understood features of BIRT including how to improve performance, scripting, accessing Big Data sources, debugging reports, and making your reports more dynamic.

If you are interested in attending, check out the registrationpage for more details.

BIRT 4.2.1 Released

The BIRT team is pleased to announce the release of BIRT 4.2.1.  This point release fixes several issues including a data set parameter issue when using Oracle or MS SQL Server.  Additionally a bugzilla entry that involved Safari and Chrome viewing issues when the viewer is deployed to Weblogic has been fixed.  An RCP Designer startup issue has also been resolved and a bug with Joint Data Sets has been corrected.

When using a BIRT dataset within a report, BIRT automatically sets up a binding mapping between the dataset and the report item that will use the dataset.  The bindings are available in the Binding tab of the properties editor.  There are many reasons for using a binding mapping, but one main reason is to allow report item aggregations and computed columns.  These binding columns can not only source data from the data set but can also get data from external sources such as global JavaScript variables or external Java classes.  While this feature offers many benefits, one drawback has always been that if the data set was modified, bindings may become invalid.  In a previous release the team added a refresh button that would add missing columns to the binding map, but this approach did not clear invalid columns.  With this release a clear button has been added that allows a user to clear all the bindings that exist for a data bound report item.  So by clicking the clear button and then the refresh button all bindings will be recalculated.

To see a list of the bugzilla entries fixed in this release, take a look at this bugzilla query.

BIRT, Cassandra and Hector

While BIRT offers many ways to connect to Cassandra, including using the Cassandra JDBC driver, this post focuses on using a Scripted data source to call the Hector Client Java client.  A BIRT scripted data source allows external Java classed to be called to retrieve data for a BIRT report and can be written in Java or JavaScript.  The examples below will use JavaScript.    For this post we used the DataStax community edition which is available here, and created a keyspace with the name users and a column family named User.  The User column family contains three string columns for first name, last name and age.  The script used to load the sample data is available in the example download.

Set Designer Classpath

The first thing that you will need to do is set the classpath for the designer to access the following set of jars. 

• hector-core-version.jar
• hector-object-mapper-version.jar
• slf4j-api-version.jar
• libthrift-version.jar
• apache-cassandra-thrift-version.jar
• guava-rversion.jar
• commons-lang-version.jar

All of these jars, with the exception of the two Hector jars are available in the /install-directory/DataStax Community/apache-cassandra/lib directory.  To get the Hector jars you can download and build the hector source or just download them from a maven repository.

The Hector-object-mapper jar file can be downloaded from here.

The Hector core jar file can be downloaded from here.

One way to setup the classpath is to create a libs directory in your Report Project and then copy all of the jars above to this folder.

Next Select Window->Preferences.  Select the Report Design->Classpath preference and click on the Configure project specific settings link.

 

Select the BIRT Project that you will be using Hector with and click on ok.

Select the enable project specific settings checkbox and add the jars in the lib folder you created earlier.

Creating a Scripted Data Source using Hector

 

You can now create a report that calls the Hector APIs directly.  To do this first create a new report.  Select the data explorer view and right click on the data sources node and click on New Data Source.  Select the Scripted Data Source option and click on finish.

 

Next right click on the Data Sets node and choose the New Data Set option.  Make sure to select the Scripted Data Source that you just created as the data source for this data set. 

 

Click on the Next button and enter each column name and data type for the data set.

 

Click on the Finish button.  You can now enter script for the data set.  To do this first make sure the data set is selected in data explorer view and click on the script tab at the bottom of the report canvas.

In the script editor you will have many events that could be scripted, but in this example all we need is an open script and a fetch script.  First select open from the script drop down list and enter a script similar to the following.

importPackage(Packages.java.util);

importPackage(Packages.me.prettyprint.cassandra.serializers);

importPackage(Packages.me.prettyprint.cassandra.service);

importPackage(Packages.me.prettyprint.hector.api);

importPackage(Packages.me.prettyprint.hector.api.beans);

importPackage(Packages.me.prettyprint.hector.api.factory);

importPackage(Packages.me.prettyprint.hector.api.query);

 

var cluster = HFactory.getOrCreateCluster("Test Cluster",new CassandraHostConfigurator("localhost:9160"));

var keyspace = HFactory.createKeyspace("users", cluster);

var rangeSlicesQuery = HFactory.createRangeSlicesQuery(keyspace, StringSerializer.get(), StringSerializer.get(), StringSerializer.get())

.setColumnFamily("User").setRange(null, null, false, 10).setRowCount(100);            

var result = rangeSlicesQuery.execute();

myrows = result.get();          

rowsIterator = myrows.iterator();

Hector also supports using CQL so you could also use the following open script

importPackage(Packages.java.util);

importPackage(Packages.me.prettyprint.cassandra.serializers);

importPackage(Packages.me.prettyprint.cassandra.service);

importPackage(Packages.me.prettyprint.hector.api);

importPackage(Packages.me.prettyprint.hector.api.beans);

importPackage(Packages.me.prettyprint.hector.api.factory);

importPackage(Packages.me.prettyprint.hector.api.query);

importPackage(Packages.me.prettyprint.cassandra.model);

 

var cluster = HFactory.getOrCreateCluster("Test Cluster",new CassandraHostConfigurator("localhost:9160"));

var keyspace = HFactory.createKeyspace("users", cluster);

            

var cqlQuery = new CqlQuery(keyspace, StringSerializer.get(), StringSerializer.get(), StringSerializer.get());

cqlQuery.setQuery("select * from User");

var resultCQL = cqlQuery.execute();    

rowsIterator = resultCQL.get().iterator();

Next add a fetch script like the following.

if (rowsIterator.hasNext()) {

     var myrow = rowsIterator.next();

     var cols = myrow.getColumnSlice().getColumns();

     for( ii=0; ii < cols.size(); ii++ ){

       row[cols.get(ii).getName()] = cols.get(ii).getValue();

     }

        return true;

}else{

       return false;

}

In the above fetch the script assumes you have named your scripted data set columns the same as the columns in Cassandra.  You should now be able to preview the data set.  Double click on the data set in the data explorer view and select preview.

 

You can now use the data set within your report. 

Deploying a Report that Uses the Hector API

 

If you are using the BIRT Viewer and deploy a report that calls the Hector API, verify that all the jars discussed in the beginning of this Post (Set Designer Classpath) are placed in WEB-INF/lib directory of the Viewer.  If you are running BIRT reports using the BIRT APIs verify that the above jars are also in the classpath.

More information on CQL and Hector is available here.  The example in this post is available on Birt-Exchange.

Using a Global JS Function across BIRT Reports

BIRT provides a scripting model that allows report customizations by implementing event handlers.  These events can be written in Java or JavaScript.  This model is described on the Eclipse Birt Site.

We have written many posts on using scripting to modify BIRT report components.  Below are just a few:

• BIRT Crosstab Scripting
• BIRT Charting– Scripting Overview
• Using script to modify a chart

In this post we will describe one way that you can share a server side JavaScript function across reports.  Suppose that you have a JavaScript function to reverse a string like:

//external js function
function reverseMyString( MyString )
{
 var rString = "";
 for (i = 0; i < MyString.length; i++)
 {
  rString = MyString.substring(i, i+1) + rString;
 }

 return rString;
}

This function can be put in a .js file and then placed in the BIRT resource folder.  If you do not have a resource folder configured for your BIRT project, it can be set from the designer window preferences dialog.

The js file can be added to the report by selecting the general properties for the report and clicking on the add file button under Javascript Files.

The global function can now be called in the expression builder or in script.

The evaluate function within the reportContext object can also be used to evaluate your script at runtime.

var testString = "ZYXWVU";

this.text = reportContext.evaluate("reverseMyString('"+testString +"')");

Using this same method, a handle to the function can also be retrieved.

var testString = "ZYXWVU";

//Evaluate Function Name

var myfunc = reportContext.evaluate("reverseMyString");

this.text = myfunc( testString );

When writing Chart script it is important to understand that the Chart Engine’s Script Context is not the same as the reports.  The Chart Engine also does the bulk of its generation and rendering during the report engine’s render phase.  Chart script events can get access to the reportContext object by using the following script.

//get reportContext

var rC = icsc.getExternalContext().getScriptable();

Once the reportContext object is obtained you can make all of the standard calls available to it.  These include getting a report parameters values, getting or setting a global variable, retrieving a localized message or calling the evaluate function.  For example, to reverse the chart title the following script could be used.

function beforeGeneration( chart, icsc )

{

var currChartTitle = chart.getTitle().getLabel().getCaption().getValue()+"";

//get reportContext

var rC = icsc.getExternalContext().getScriptable();

var myFunc = rC.evaluate( "reverseMyString" );

chart.getTitle().getLabel().getCaption().setValue(myFunc( currChartTitle ));

}

Or you could create a global js function in your js file that takes the chart as a parameter and reverses its title like:

//external js function

function reverseMyTitle( chart )

{

    var MyString = chart.getTitle().getLabel().getCaption().getValue() + "";

    var rString = "";

    for (i = 0; i < MyString.length; i++)

    {

        rString = MyString.substring(i, i+1) + rString;

    }

    chart.getTitle().getLabel().getCaption().setValue( rString);

} 

You could then call this function in chart script as shown below.

function beforeGeneration( chart, icsc )

{

//get reportContext

var rC = icsc.getExternalContext().getScriptable();

var myFunc = rC.evaluate( "reverseMyTitle" );

myFunc( chart );

}

BIRT 4.2 Released

BIRT 4.2 is now available and with this release many improvements and new features are available. BIRT 4.2 now provides a new Excel data source that supports multi-sheet data sets, derived measures are now available on cubes, better filter support with aggregates that allows cumulative data to include or exclude filtered rows, and support for an OSGi or POJO runtime.  In addition BIRT now supports a Donut chart type and the build process has been modified to add Maven support for the BIRT engines.

To read more about these and other new features for BIRT, see the BIRT 4.2 New and Notable.

BIRT Area Chart Modifications

BIRT supplies a very robust and extensible chart engine, that can be used standalone or in conjunction with the report engine.  Currently the chart engine supports fourteen different main chart types and many sub-types.  Charts can be emitted in PNG, JPG, BMP, SVG within reports and can be also emitted to SWT, PDF and Swing outside of the report engine.    Virtually every area of the chart engine is also extensible, from adding new chart types to new output formats.  These are done with Eclipse extension points.  In addition the chart engine supports client side interactivity and server side event scripting.  Both of which have been discussed on the site before.

BIRT Chart Scripting Overview

One of the most often used chart types is the Area chart. 

Simple Area Chart

While this type of chart is an effective visualization, we often get questions on how to extend the Area chart to the extents of the plot.  In this post we will put together an example that illustrates one way of extending the covered area.

As discussed extensively in the post referenced above, the chart can be modified using script event handlers.  These handlers can be written in Java or JavaScript.  With certain chart types the render engine renders to the center of a data point intersection.  The Area and Line Charts are examples of the types of charts that exhibit this behavior.  To extend the area chart, a beforeDrawSeries event can be implemented to change the x location of the first and last data point to cover more area.  This approach will work for both 2D and 2D with depth types of charts.  The beforeDrawSeries event is fired for each runtime series and once for the category series.  So in the script you must first check that the locations you are going to modify are for the right series.  If you are not using optional grouping this check is as simple as just getting the series identifier.  This identifier is set as the series title in the third tab of the chart wizard.

Series Identifier

So in the beforeDrawSeries event handler you can check the series identifier by calling the following code.

if( series.getSeriesIdentifier() == "Series 1" ){}

The chart renderer stores all the data point information in a data point hints array.  This array can be retrieved in the beforeDrawSeries event handler by calling:

var dpharray = seriesRenderer.getSeriesRenderingHints().getDataPoints();

Each data point element in the array stores information like the category value, orthogonal value, and x/y location values that the renderer will use to draw the chart.  To get the x location of the area chart we first call the getLocation method and then the getX method.  This needs to be done for the first and last data points in the array.  You can set the x value using the setX method on the location object.  You can also get the width of a data point by calling the getSize method.  This method returns the width show in red in the following diagram.

The getSize method

Using the above methods we can subtract half the width from the first data point x value and add half the width to the x value of the last data point value.  The complete script is shown below.

function beforeDrawSeries( series, seriesRenderer, context )

{      

      if( series.getSeriesIdentifier() == "Series 1" ){

      var dpharray = seriesRenderer.getSeriesRenderingHints().getDataPoints();
                  var xval = dpharray[0].getLocation().getX();
                  var wid = dpharray[0].getSize();
                  dpharray[0].getLocation().setX(xval-(wid/2));
                  var xval = dpharray[dpharray.length-1].getLocation().getX();
                  dpharray[dpharray.length-1].getLocation().setX(xval+(wid/2));  

      }

}

A before and after example 2D with depth chart is shown below.

Before Example

After Example

This example is available on Birt-Exchange.

Add Values to a BIRT Chart

When building reports that contain Charts, the BIRT data engine is responsible for creating the chart series data points. These data points are generally tied to BIRT data sets or cubes. While these mechanisms handle a lot of the grouping an aggregation of the data to be charted it may be desirable to add some manual data points to the chart. Fortunately this can be done with a fairly simple chart script event handler. If you have not done any chart scripting before, you may want to read over this post on scripting.

Chart event handlers are fired on the server and can be written in Java or JavaScript. The event order is listed in the post described earlier. The before and after DatasetFilled events are fired first. These events are fired for every runtime series that will be plotted. For example if you have one bar series, these events will be fired once for the category series values and once for the bar series values. If you use optional grouping, these events will be fired for every optional group the data engine encounters. The afterDataSetFilled event handler is passed a reference to the current series and the data set that will be used by the chart engine. This is an ideal location to change values, check for nulls or add values to the chart.
Lets assume we have the following Chart:

Generic Chart
This chart contains one bar series with four data points. In this example the afterDataSetFilled event will be fired twice, once for the category values and once for the bar series values. We can then use the following script to add a value to the beginning and the end of the series.

function afterDataSetFilled(series, dataSet, icsc)
{
	importPackage( Packages.java.util );
	importPackage(Packages.java.lang);	
	importPackage( Packages.org.eclipse.birt.chart.model.type.impl );
	importPackage( Packages.org.eclipse.birt.chart.model.data.impl);
      var list = dataSet.getValues();
	var narray1 = new ArrayList( );

//Check Series Type
//SeriesImpl used for category series
//AreaSeriesImpl
//BarSeriesImpl
//BubbleSeriesImpl
//DialSeriesImpl
//DifferenceSeriesImpl
//GanntSeriesImpl
//LineSeriesImpl
//PieSeriesImpl
//ScatterSeriesImpl
//StockSeriesImpl

	
	if( series.getClass() == BarSeriesImpl ){
		narray1.add(new Double(40.6));
	}else{
		narray1.add("AddBefore");
	}	
	
	var llen =list.length;
    for ( i=0; i < llen; i++)
    {
		narray1.add(list[i]);
    }
//Chart Data Set Types
//BubbleDataSetImpl
//DateTimeDataSetImpl
//DifferenceDataSetImpl
//GanttDataSetImpl
//NumberDataSetImpl
//StockDataSetImpl
//TextDataSetImpl
    
	if( series.getClass() == BarSeriesImpl ){
		narray1.add(new Double(25.6));
		series.setDataSet(NumberDataSetImpl.create( narray1 ));
	}else{
		narray1.add("AddAfter");
		series.setDataSet(TextDataSetImpl.create( narray1 ));
	}	
   
}

The first thing this script does is to get the current values for the given series and creates a new ArrayList. Next it checks to see which series triggered this event. It does this by checking the series class. You could also check the series identifier. Once the series type is determined we add an initial value to the ArrayList, followed by adding all existing values to the ArrayList. Finally a last value is added to the ArrayList and a new chart data set is created. The type of chart data set that is created will depend on how you configured the chart and what type of chart you are using. The comments show additional options. In this example we are using a Text data set for the categories and a number data set for the bar series values. The output of the chart should now look like:

Chart After Script
This example is available for download at Birt-Exchange. For an example on adding a whole new series to a chart, see this post.