Friday, April 18, 2008

Return To Eureka

Two weeks ago I had a Eureka moment. In that post I described how I was able to use reflection to get the DesignElementHandle from an ICell object. It was a fairly simple solution since there was only one super class to the ICell.

Yesterday, I started to re-factor my other example classes to use this helper utility and I figured out that the helper utility need a little more work. It turns out that the ITable class is a bit more complicated then the ICell class. For the ITable object there are five levels super classes to get back to the DesignElement class.

So the ITable object actually resolves to:

  • class org.eclipse.birt.report.engine.script.internal.element.Table
The hierarchy for this class is:
  • class org.eclipse.birt.report.engine.script.internal.element.Listing
  • class org.eclipse.birt.report.engine.script.internal.element.ReportItem
  • class org.eclipse.birt.report.engine.script.internal.element.ReportElement
  • class org.eclipse.birt.report.engine.script.internal.element.DesignElement
When we use reflection to find the protected field we have to navigate all the way back to DesignElement which can be n levels up the hierarchy. So we need to use a loop to search for the parent.

Once we have the DesignElement we can use reflection to get access to the protected field for the simple api DesignElement. The simple api has the same hiearchy issue, with one additional super-class.
  • class org.eclipse.birt.report.model.simpleapi.Table
  • class org.eclipse.birt.report.model.simpleapi.Listing
  • class org.eclipse.birt.report.model.simpleapi.MultiRowItem
  • class org.eclipse.birt.report.model.simpleapi.ReportItem
  • class org.eclipse.birt.report.model.simpleapi.ReportElement
  • class org.eclipse.birt.report.model.simpleapi.DesignElement
Fortunately, the solution is not that difficult. The addition of one relatively simple loop allows us to find the parent classes we need.


private static final String DESIGN_ELEMENT_NAME = ".DesignElement";

private static Class findDesignElementParent(Class childClass) throws NoSuchFieldException {
System.out.println(childClass.toString());
do {
childClass = childClass.getSuperclass();
if (childClass == null)
throw new NoSuchFieldException("Did not find DesignElement");
System.out.println(childClass.toString());
} while (!childClass.toString().endsWith(DESIGN_ELEMENT_NAME));
return childClass;
}


You may ask scratch your head and ask why am I searching for the class using the name of the class as opposed to a reference to the actual class name. This goes back to the issue that the classes that we are referencing are not actually exposed through the runtime interface.

In any case, once I have made these changes my utility method is still relatively simple.

public static DesignElementHandle getDesignElementFromScript(IDesignElement scriptObj)
throws IllegalAccessException, SecurityException, NoSuchFieldException {

Class scriptParentClass = findDesignElementParent(scriptObj.getClass());
Field fieldFromScript = scriptParentClass.getDeclaredField("designElementImpl");
if (fieldFromScript == null) {
return null;
}

// Access the DesignElement field from the Script object
fieldFromScript.setAccessible(true);
Object designElement = fieldFromScript.get(scriptObj);

// The DesignElement object has a field named "handle"
// which is the DesignElementHandle for that DesignElement
// Use reflection again to get this field
Class simpleApiParentClass = findDesignElementParent(designElement.getClass());
Field fieldFromSimpleApi = simpleApiParentClass.getDeclaredField("handle");
if (fieldFromSimpleApi == null) {
return null;
}

// Now access the DesignElementHandle from the DesignElement
fieldFromSimpleApi.setAccessible(true);
return (DesignElementHandle) fieldFromSimpleApi.get(designElement);
}


The complete source is available in the script.lib project on the http://longlake.minnovent.com/repos/birt_example subversion project. I have moved the code into the ScriptUtil class.

There is an underlying question here. You may ask why do I have to go through those gyrations to get a handle to the object? Shouldn't the API just support this method. That's a good question. The issue is that working with the DesignElementHandle object directly is a relatively advanced feature.

The BIRT script objects are meant to be a report developer interface and have been designed to protect the report developer from obscure and hard to debug conditions. When you start working with the ElementHandles there is very little that is protecting the developer from doing something they don't want to do.

For example, in my Eureka example I demonstrated changing the column span for a cell. What I didn't mention is that for the example in question, the change I made was always going to be appropriate. In reality when you change the column span on one cell in the row, you have to account for the total number of cells. This means that you have to:
  • Modify the column span of another cell in the row
  • Drop cells from the row
  • Add cells to the row
These operations are not difficult, but they do require a level of sophistication and thought that many beginning report developers are not ready to take on. So is there a way to expose methods, but somehow warn that these methods are for power users only? Is it better to just not expose the power methods and use a utility class such as the one I have demonstrated?

I have opened a bugzilla item to expose the DesignElementHandle through the API. I would like to hear what other people think of this idea. How do you expose the power interface while still protecting users that don't want or need access to objects that powerful?

Thursday, April 17, 2008

Testing and Debug of Reports

At EclipseCon this year I did a two hours tutorial on BIRT integration. I think it went fairly well. I went a little long, but covered all the best bits. I would like to do a series of posts on this site that go over some of the same material.

The stuff I really want to talk about is a technique that I have developed for debugging / stepping through your Java Event Handler classes. My method requires a fair amount of setup, but in the end it makes it far easier to step through your event handlers. In version 2.3 of the product, this process is simplified. That said many people are on version 2.2 and will continue to be on version 2.2 for a while.

So if you intend to do any Java event handler development in BIRT 2.2. this is the post for you.

Background

The first thing that we need to do is to understand the report test framework that I use on BIRT projects. This is a really simple format that allows me to batch run a large number of pre-defined BIRT reports. I am not going to dive into the details of how all of this works, but you will have all of the source code, and it is relatively simple to figure out.

All of the code for these examples is stored on my subversion server at:
http://longlake.minnovent.com/repos/birt_example

You will find a lot of projects there, they are all available under EPL. The ones that you will need to follow these examples are:
birt_api_example
birt_runtime_lib_222
script.lib

First a couple of words about the projects. If you take a look at these projects you will notice that they are plug-in projects (look under builders). I use the plug-in mechanism to associate the projects together. You do not need to do this, but I have found that it allows people to download them into their workspace and they will just run.

The birt_api_example project contains the majority of the code that you will work with. The REAPI package contains code showing how to run reports. The DEAPI package contains code showing how to create report designs through code. The Reports folder contains report designs that are used by both the REAPI and DEAPI. In addition, Reports contains reports that use the DEAPI.

The birt_runtime_lib_222 project contains the BIRT runtime libraries. It also acts as BIRT_HOME for both the REAPI and DEAPI examples. The runtime lib takes all of the libs and plug-ins from the and exposes them as a plug-in project. I will return to the what exactly the runtime_lib does in a later post. First, I want to get to the fun stuff.

The final project is the script.lib project which contains java event handler code that you want to run or debug. If all goes well, I will tie all of this back to that project at the end of this post.

So lets start with the birt_api_example project. In large BIRT report development projects, you often have multiple reports that need to be tested under a variety of conditions. Typically, we build a relatively small number of rptdesigns that can create a large number of rptdocuments based on the parameters used when the report runs.

This technique is great since it dramatically reduces the amount of code that needs to be maintained. The problem is that one rptdesign may have a large number of test conditions that need to be run. Manually running each condition by entering the required parameters is going to be difficult at best.

What we wanted was a relatively simple solution where we could store report run configurations and then run them in batch. This is the role of the tester package. There is a single java class TestAllReports.java which can be run as a java application.

When you run the TestAllReports project the program will look in the tester_run folder for any .properties files. For each properties file the tester will try to run according to the configuration information in the file. The format of the properties file is very simple.

file_name=customers.rptdesign (relative path from ./Reports directory)
param1_name=param1_Value
param2_name=param1_Value

Currently I have added support for Strings and java.sql.Date parameter data types. It is trivial to modify to support Integers or Booleans if you require these data types.

Once the tester has opened your properties file it will run the report design using Run then Render tasks to build PDF files. You can modify the code in the testReport method to enable output to HTML, Excel, or Word formats (the code is there but commented out). Reports are run into a new folder that will have the format:

tester_output_20080417_133034

Using the tester you can batch test large number of reports. The tester does not have a way to automate the comparison of rendered files, but there are a number of tools that can be used if your processes require this level of automation.

Debugging Java Event Handlers

Now that we have the tester running, it is time to go ahead and look at stepping through event handler code. The 'official' technique for debugging BIRT event handlers is to create a runtime workspace that runs your reports as defined here. This process is relatively straight-forward, but there is one problem, MemoryWoes. I have a two gig laptop and running MaxPermSize I still run into memory issues when running the Runtime workbench.

What I wanted was a method where I could run my reports through the debugger, without having to load the runtime workbench. Well it turns out that I can use my tester program to do just that.

The starting place for this is the script.lib project. This project is where I am going to place all of my event handlers. I then have an ant build file that I can use to compile my event handlers into a jar file. Back in the tester project, when it loads up the EngineConfig, I add that jar file:

    EngineConfig config = new EngineConfig();
config.setProperty(EngineConstants.WEBAPP_CLASSPATH_KEY, getScriptLibFileNames());

...

/*
* The engine needs to see a list of each jar file concatenated as a string
* using the standard file system separator to divide the files
*/
private static String getScriptLibFileNames() {
File scriptHome = new File(SCRIPT_LIB);

if (!scriptHome.exists()) {
scriptHome.mkdir();
}

File[] dirFile = scriptHome.listFiles(new JarFilter());
StringBuffer scriptlibClassPath = new StringBuffer(); //$NON-NLS-1$
for (int i = 0; i < dirFile.length; i++) {
if (scriptlibClassPath.length() > 0) {
scriptlibClassPath.append(File.pathSeparatorChar);
}
scriptlibClassPath.append(dirFile[i].getAbsolutePath());
}

return scriptlibClassPath.toString();

}


Now, I know what you are thinking. Why did you have to add the file as a jar file, why not just add the folder to the classpath. The short answer is that I don't know how to add folder structures to the classpath, and I didn't bother to learn. This is not due to laziness, my thought is that when I deploy to my application, I am going to deploy the jar file. By using the jar when I run the tester, I have a better test environment.

Testing It Out

Now that you have an explanation of what we are doing, lets test it out. If you go back to the birt_api_example project and look in the tester_save folder you will find a dyn_table.properties file. Copy this file to the tester_run folder. Examining the file you will see that I am running the report found in:

Reports/EventHandlerReports/a_dyn_table_columns.rptdesign

Opening this design directly and run it and you will see that a three column table generates a report with eight columns. How does that happen? If you look on the properties for the table, you will see that I am using an event handler named TableMagic, which is in the script.lib project. (I have pictures coming, but blogger is having issues with images right now)

Now put a breakpoint in the TableMagic.java class. (try using from the Java perspective). If you have copied the dyn_table.properties file to the tester_run folder, and you execute the tester in debug mode, it will take you directly to the break point.

How cool is that? You are now debugging your java classes without launching a runtime workspace. The best part is using this type of debugging, I can walk you through the DEAPI calls that I have made to dynamically modify my report based on data from the database.

Now I feel like Howie Mandel on Deal or No Deal. Just when I get to the good stuff, I am going to have to take a break. Next week, I will go through the TableMagic event handler to discuss how you can use the DEAPI to create data driven report designs.


Monday, April 14, 2008

BIRT 2.3 Milestone 6 New and Notable Features.

BIRT 2.3 Milestone 6 has been released and contains improvements to scripting, the addition of the DTP graphical Query Builder prototype, cross tab and chart improvements, and a couple new extension points. Be sure to try out the Query Builder. The DTP team is looking for feedback. Cross tabs can now be modified using script and marker script events are available for charts.

To read more about these features and others available in the 2.3 M6 release, go here.

Friday, April 04, 2008

Eureka....

I wanted to share a little DEAPI related discovery. If I wait to type it up the 'right' way I may not ever get it done, so here it is. I have been doing a lot of work with the DEAPI inside of a report.

If you have not worked with the DEAPI inside of a report, I should remind you that using the DEAPI is only valid within the OnPrepare method of the reportItems or from the BeforeFactory method of the ReportDesign.

When you work with the DEAPI you need to be aware of three basic types of objects in the report engine. Event handlers are passed objects that are part of the script api package. Script objects have limited functionality. For instance, an ICell script object can tell how many cells the column is spanning cell.getColumnsSpan(), but does not allow you to set the column span.

The type of object you want to work with is the DesignElementHandle, which for a cell is CellHandle. DesignElementHandles are defined in the Model API and give you access to all of the functionality in the model. Once you have the DesignElementHandle, you can change the report design in almost any manner.

In between the DesignElementHandles and the script objects are the DesignElements. The DesignElements objects represent the actual instance of the object in the report. DesignElement objects are not exposed by the report engine, and are not particularly useful except for one key role, they provide a link between the script objects and the DesignElementHandles.

If a report were a house: the DesignElementHandles are the blue prints, the DesignElements are the house itself and the script objects are pictures of the house. If you modify the blue print (DesignElementHandle) early in the build process, you can completely change the way the report is built. To stretch the metaphor you are looking at a picture of someone else house (script object) and using that to change the blue prints (DesignElementHandle) for the house that you are building.

Clear as mud? Good...

I have always navigated to the DesignElementHandle by using the name value of the script object.

   DesignElementHandle designHandle = rptContext.getReportRunnable().getDesignHandle();
ModuleHandle moduleHandle = designHandle.getModuleHandle();
TableHandle tableHandle = (TableHandle) moduleHandle.findElement(inTable.getName());
//modify the table handle as I want



Some script objects (cells, rows, columns) do not support the name value, in that case I can use the getParent() method to navigate to a container which supports the name value. Typically I end up back at the table.

From the tableHandle you can then navigate to the appropriate, row, column, group within the report. This works okay, but it can be tricky. In some cases, if you are looking to modify one particular cell it is down right hard. What I wanted to be able to do is get the CellHandle directly from the scriptable object that that is passed to the event handler. You end up having to write a bunch of code to get to navigate back to the object where you started.

It turns out that the ScriptObject has a protected field named designElementImpl which is a handle to the DesignElement object. The DesignElement object has a field "handle" which is the DesignElementHandle. The problem is that neither of the fields are exposed through the API.

Don't you hate it when you know something is there but you just can't figure out how to access it?

I finally got tired of writing all kinds of code to navigate back to an object. It turns out you can use reflection to get access to the fields that are there, but not exposed through the API.

I have created a simple generic method that will get the associated DesignElementHandle from the script object. This is fairly fresh, and has not been fully wrung out for all issues, but it is a decent place to start.

/**
* Modify the columnSpan of a cell item
*/
public void onPrepare(ICell cell, IReportContext reportContext) {

Integer unitSpan = reportContext.getGlobalVariable("cell_span");

try {
// call generic method to get DesignElementHandle
DesignElementHandle cellHdl = getDesignElementFromScript(cell);
if (cellHdl instanceof CellHandle) {
((CellHandle) cellHdl).setColumnSpan(unitSpan);
} catch (Exception e) {
// trap errors
}

/**
* Generic method to access DesignElementHandle from a scriptable object
*
* @param scriptObject
* @return
* @throws NoSuchFieldException
* @throws IllegalAccessException
*/
public DesignElementHandle getDesignElementFromScript(IDesignElement scriptObj)
throws NoSuchFieldException, IllegalAccessException {

// The designElementImpl field is a protected field
// that is declared on the super class
Field fieldFromScript = scriptObj.getClass().getSuperclass()
.getDeclaredField("designElementImpl");
if (fieldFromScript == null){
return null;
}

// Access the DesignElement field from the Script object
fieldFromScript.setAccessible(true);
Object designElement = fieldFromScript.get(scriptObj);

// The DesignElement object has a field named "handle"
// which is the DesignElementHandle for that DesignElement
// Use reflection again to get this field
Field fieldFromSimpleApi = designElement.getClass().getSuperclass()
.getDeclaredField("handle");
if (fieldFromSimpleApi == null){
return null;
}

// Now access the DesignElementHandle from the DesignElement
fieldFromSimpleApi.setAccessible(true);
return (DesignElementHandle)fieldFromSimpleApi.get(designElement);
}

Wednesday, April 02, 2008

BIRT and Ecore!

As many of you are aware BIRT uses the Data Tools Platform (DTP) project’s Open Data Access (ODA) standard to access data. Currently BIRT supports the following data sources:

JDBC
XML
Flat File
Web Services
Scripted Data Set

In addition, BIRT provides two project wizards for creating new ODA drivers.

The ability to query Ecore models, for use within BIRT reports, has frequently been requested. Jeff Ramsdale and Tim Myer have recently contributed an ODA driver to the DTP project that queries the Ecore model using the Object Constraint Language (OCL). To read more about this driver take a look at the proposal.
Be sure to drop by the DTP newsgroup to comment on or discuss this new ODA.

Wednesday, March 19, 2008

EclipseCon BIRT Sessions

EclipseCon continues to go really well with a lot of great talks. I have found the OSGi Services talks to be very interesting. Perhaps there are opportunities to add greater flexibility to the BIRT runtime through the adoption of more services for some functions? Going to have to think about that for a while.

On Tuesday Paul Clenahan presented a general intro to BIRT talk to a standing room only crowd. The nice lady at the door had to pull out her baseball bat to turn away the overflow crowd.

In the afternoon Justin Miranda from the OpenMRS team talked about how they are using BIRT to help provide better health care in some of the most difficult places in the world. If you are interested in making a difference, OpenMRS is looking for volunteers. If you are interested, send me an email at (scottr at innoventsolutions dot com) Please leave a comment and I can put you in touch with Justin at OpenMRS.

Jason Weathersby provided an in-depth review of the Chart API's. If you didn't know Jason has spent the last couple of months diving into the chart API's. He has walked through most of the code that makes up the charting plug-ins and he has created his own Chart extension. As a result Jason has really improved the charting chapters in the BIRT Integration book. In addition he provided a really great tutorial on chart scripting. He wrapped it all up with a tour of the chart api's. We will be posting the slides to the web site shortly. If you are interested in charting, make sure you get the second edition of the BIRT Integration book.

Today, I got a preview of the new DTP Query Builder. It won't be available until 2.3 M6, but I think that you will all like the new query builder. Two very cool features:
- re-factoring of the sql query e.g. changing alias names
- ability to dock the query builder as an editor, so that you don't have to continually re-open the dialog.

We also had a series of five really cool BIRT short talks. Virgil Dodson, Pierre Tessier, John Ward, Phillipe Coucaud and Santosh Kumar teamed up to show what can be done to really make BIRT dance. My personal favorite was the way that Phillipe wrapped the BIRT charts with a simple interface and then used it to provide visual feedback about the project.

In five minutes, we will be starting the BIRT BOF, I look forward to having an informal discussion with the BIRT community, I'll let you know how it goes tomorrow.

Tuesday, March 18, 2008

Fun At EclipseCon

Pierre Tessier made it to EclipseCon this year and is very excited about the shwag that can be picked up from the exhibition hall as well as his BIRT Rocks t-shirt. This is the first time that I have ever seen Pierre blush like a little girl (with five o'clock shadow). Some things need to be shared.