Alpha Release of Drools Flex Editor - 0.5

Orange Mile is proud to announce the long awaited alpha release of a Drools Rule Editor in Flex.

http://code.google.com/p/drools-flex-editor/

The current release includes all the Flex pieces without the rather basic server side code for rule compilation, and code completion. This will be available in the final 1.0 release.

This is the beginning of having rich enough components available within the system that can allow the user/admin to directly manipulate the business rules without having the long development cycle.

Drools + Spring Security + Annotations + AOP= ?

I am starting a new open source project:

http://code.google.com/p/dynamic-rule-security/

No code has been released yet, but I am hoping to have an alpha version out soon. The project integrates Drools Rule Engine with Spring Security to provide dynamic, rule based, field level ACL security to a system.

Once complete, the system administrator will be able to create business rules to restrict fields, objects, pages, content, whatever based on dynamic rules. But, that's not all. The current crop of security requires the security logic to be embedded with the code and is quite brittle and complex when security rules become very granular. For example, imagine having to implement a requirement that says when a trade belongs to account "abc" hide the trade from anyone not in group "abc-allowed". No problem, you say. You create the security group "abc-allowed". Now you have some choices regarding implementation, you can integrate the rule at the data retrieval layer, at the presentation tier, or in the middle. Either way, somewhere in your system, you'll have a chunk of code like this: if ( trade.account == "abc" && !isUserInRole("abc-allowed") ) then hide.

That was easy. Probably only took 10 minutes to write, 10 minutes to test, and a few days to get it deployed to production. No problem.

A few days go by, and the user comes back and says, I need to expand that security. It seems that group efg can actually see abc account trades but only when the trading amount is less than $50m. Ok, you say. A bit messy, but do-able. So, you create security group "efg-allowed", and change your prior rule to say:
if ( trade.account == "abc" && (!isUserInRole("abc-allowed") && ( trade.amount > 50 && !(isUserInRole("efg-allowed") ) then hide.

Probably only took 10 minutes to code, and another 10 minutes to test, but damn there is QA, UAT, production release. A few days later, you finally release the new feature.
Aren't you glad that's over. A few more days go by, and the user says, wait, he forgot that the efg group can't change the trader name on the trade, and can't see the counterparty, but should be able to see and change everything else. Oh, one more thing, they can change the trader name if the trader is "Jack", because trader Jack's accounts are actually managed by the efg group even if the account belongs to the "abc" group.

Crap you say, that's going to be a bit of work. You may need to change the presentation tier, to hide the fields in some cases, but not others. And boy, how much does it suck to hard code the trader's name somewhere.

Anyways, you get the point. Security Rules may get very complex and very specific to the data they interact with and the context of the request. This means that the rule needs to be aware of the data, and who is requesting it. The rule is then capable of setting the security ACL. The presentation tier then only needs to worry about following the ACL rather than actually dealing with the security rules themselves. Not only that, but security rules will be in a single place rather than being sprinkled throughout the system. You can also change them on the fly allowing you to react very quickly to additional security requests.

How to retrieve the fields used in a Drools Rule (DRL)

Sometimes it maybe useful to know what fields the rule-set relies on. For example, let's imagine you have a freaky dynamic system that's able to populate beans with only the data needed. The problem then becomes how do you know what data is needed by your vast set of dynamic rules.

One way to do this is to assume that you're dealing with standard pojo's. This means that each variable is private and has an associated getVar and setVar method. Drools currently supports their own language, DRL, java (backed by Janino compiler), and MVEL. I will present how to retrieve the fields form DRL and Java. I am sure the same principles can be applied to MVEL.

First, your pojo:

package com.orangemile.ruleengine;

public class Trade {
private String traderName;
private double amount;
private String currency;
public String getTraderName() {
return traderName;
}
public void setTraderName(String traderName) {
this.traderName = traderName;
}
public double getAmount() {
return amount;
}
public void setAmount(double amount) {
this.amount = amount;
}
public String getCurrency() {
return currency;
}
public void setCurrency(String currency) {
this.currency = currency;
}
}


Now the magic:


package com.orangemile.ruleengine;

import java.io.StringReader;
import java.util.ArrayList;
import java.util.Collection;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;

import org.codehaus.janino.Java;
import org.codehaus.janino.Parser;
import org.codehaus.janino.Scanner;
import org.codehaus.janino.Java.MethodInvocation;
import org.codehaus.janino.util.Traverser;
import org.drools.compiler.DrlParser;
import org.drools.lang.DrlDumper;
import org.drools.lang.descr.EvalDescr;
import org.drools.lang.descr.FieldConstraintDescr;
import org.drools.lang.descr.ImportDescr;
import org.drools.lang.descr.PackageDescr;
import org.drools.lang.descr.PatternDescr;
import org.drools.lang.descr.RuleDescr;

/**
* @author OrangeMile, Inc
*/
public class DRLFieldExtractor extends DrlDumper {

private PackageDescr packageDescr;
private Map variableNameToEntryMap = new HashMap();
private List entries = new ArrayList();
private Entry currentEntry;

public Collection getEntries() {
return entries;
}

/**
* Main Entry point - to retrieve the fields call getEntries()
*/
public String dump( String str ) {
try {
DrlParser parser = new DrlParser();
PackageDescr packageDescr = parser.parse(new StringReader(str));
String ruleText = dump( packageDescr );
return ruleText;
} catch ( Exception e ){
throw new RuntimeException(e);
}
}

/**
* Main Entry point - to retrieve the fields call getEntries()
*/
@Override
public synchronized String dump(PackageDescr packageDescr) {
this.packageDescr = packageDescr;
String ruleText = super.dump(packageDescr);
List rules = (List) packageDescr.getRules();
for ( RuleDescr rule : rules ) {
evalJava( (String) rule.getConsequence() );
}
return ruleText;
}

/**
* Parses the eval statement
*/
@Override
public void visitEvalDescr(EvalDescr descr) {
evalJava( (String) descr.getContent() );
super.visitEvalDescr(descr);
}

/**
* Retrieves the variable bindings from DRL
*/
@Override
public void visitPatternDescr(PatternDescr descr) {
currentEntry = new Entry();
currentEntry.classType = descr.getObjectType();
currentEntry.variableName = descr.getIdentifier();
variableNameToEntryMap.put(currentEntry.variableName, currentEntry);
entries.add( currentEntry );
super.visitPatternDescr(descr);
}

/**
* Retrieves the field names used in the DRL
*/
@Override
public void visitFieldConstraintDescr(FieldConstraintDescr descr) {
currentEntry.fields.add( descr.getFieldName() );
super.visitFieldConstraintDescr(descr);
}

/**
* Parses out the fields from a chunk of java code
* @param code
*/
@SuppressWarnings("unchecked")
private void evalJava(String code) {
try {
StringBuilder java = new StringBuilder();
List imports = (List) packageDescr.getImports();
for ( ImportDescr i : imports ) {
java.append(" import ").append( i.getTarget() ).append("; ");
}
java.append("public class Test { ");
java.append(" static {");
for ( Entry e : variableNameToEntryMap.values() ) {
java.append( e.classType ).append(" ").append( e.variableName ).append(" = null; ");
}
java.append(code).append("; } ");
java.append("}");
Traverser traverser = new Traverser() {
@Override
public void traverseMethodInvocation(MethodInvocation mi) {
if ((mi.arguments != null && mi.arguments.length > 0)
|| !mi.methodName.startsWith("get") || mi.optionalTarget == null) {
super.traverseMethodInvocation(mi);
}
Entry entry = variableNameToEntryMap.get(mi.optionalTarget.toString());
if ( entry != null ) {
String fieldName = mi.methodName.substring("get".length());
fieldName = Character.toLowerCase(fieldName.charAt(0)) + fieldName.substring(1);
entry.fields.add( fieldName );
}
super.traverseMethodInvocation(mi);
}
};
System.out.println( java );
StringReader reader = new StringReader(java.toString());
Parser parser = new Parser(new Scanner(null, reader));
Java.CompilationUnit cu = parser.parseCompilationUnit();
traverser.traverseCompilationUnit(cu);
} catch (Exception e) {
throw new RuntimeException(e);
}
}


/**
* Utility storage class
*/
public static class Entry {
public String variableName;
public String classType;
public HashSet fields = new HashSet();

public String toString() {
return "[variableName: " + variableName + ", classType: " + classType + ", fields: " + fields + "]";
}
}
}



And now, how to run it:


public static void main( String args [] ) {
String rule = "package com.orangemile.ruleengine;" +
" import com.orangemile.ruleengine.*; " +
" rule \"test rule\" " +
" when " +
" trade : Trade( amount > 5 ) " +
" then " +
" System.out.println( trade.getTraderName() ); " +
" end ";

DRLFieldExtractor e = new DRLFieldExtractor();
e.dump(rule);
System.out.println( e.getEntries() );
}


The basic principle is that the code relies on the AST tree that's produced by DRL and Janino. In the case of Janino walk, the code only looks for method calls that have a target, start with a "get", and take no variables. In the cast of DRL, the API is helpful enough in providing callbacks when a variable declaration and field is hit, making the code trivial.

That's it. Hope this helps someone.

Drools - Fact Template Example

Jboss Rule Engine ( Drools ) primarily works based on an object model. In order to define a rule and have it compile, the data referenced in the rule needs to exist somewhere in the classpath. This is easy enough to accomplish by using any of the dynamic libraries such as asm, cglib, or antlr. Once your class is defined, you can either inject your own implementation of a Classloader or change the permission on the system classloader and call defineClass manually.

But, there is another way, which is a bit over simplistic, but maybe useful for some of you out there. Drools has introduced support for fact templates, which is a concept introduced by Clips. A fact template is a basically a definition of a flat class:

template "Trade"
String tradeId
Double amount
String cusip
String traderName
end

This template can then be naturally used in the when part of a rule:


rule "test rule"
when
$trade : Trade(tradeId == 5 )
then
System.out.println( trade.getFieldValue("traderName") );
end


But, there is a cleaner way to do all of this using the MVEL dialect introduced in Drools 4.0.
You can code your own Fact implementation that's backed by a Map.

package app.java.com.orangemile.ruleengine;

import java.util.HashMap;
import java.util.concurrent.atomic.AtomicLong;

import org.drools.facttemplates.Fact;
import org.drools.facttemplates.FactTemplate;
import org.drools.facttemplates.FieldTemplate;

/**
* @author OrangeMile, Inc
*/
public class HashMapFactImpl extends HashMap implements Fact {

private static AtomicLong staticFactId = new AtomicLong();

private FactTemplate factTemplate;
private long factId;

public HashMapFactImpl( FactTemplate factTemplate ) {
factId = staticFactId.addAndGet(1);
this.factTemplate = factTemplate;
}

@Override
public long getFactId() {
return factId;
}

@Override
public FactTemplate getFactTemplate() {
return factTemplate;
}

@Override
public Object getFieldValue(int index) {
FieldTemplate field = factTemplate.getFieldTemplate(index);
return get(field.getName());
}

@Override
public Object getFieldValue(String key) {
return get(key);
}

@Override
public void setFieldValue(int index, Object value) {
FieldTemplate field = factTemplate.getFieldTemplate(index);
put( field.getName(), value );
}

@Override
public void setFieldValue(String key, Object value) {
put(key, value);
}
}


To use this class, you would then do this:


String rule = "package com.orangemile.ruleengine.test;" +
" template \"Trade\" " +
" String traderName " +
" int id " +
" end " +
" rule \"test rule\" " +
" dialect \"mvel\" " +
" when " +
" $trade : Trade( id == 5 ) " +
" then " +
" System.out.println( $trade.traderName ); " +
" end ";

MVELDialectConfiguration dialect = new MVELDialectConfiguration();
PackageBuilderConfiguration conf = dialect.getPackageBuilderConfiguration();
PackageBuilder builder = new PackageBuilder(conf);
builder.addPackageFromDrl(new StringReader(rule));
org.drools.rule.Package pkg = builder.getPackage();
RuleBase ruleBase = RuleBaseFactory.newRuleBase();
ruleBase.addPackage(pkg);

HashMapFactImpl trade = new HashMapFactImpl(pkg.getFactTemplate("Trade"));
trade.put("traderName", "Bob Dole");
trade.put("id", 5);

StatefulSession session = ruleBase.newStatefulSession();
session.insert(trade);
session.fireAllRules();
session.dispose();


Notice, that in the then clause, to output the traderName, the syntax is:

$trade.traderName

rather then the cumbersome:

$trade.getFieldValue("traderName")

What makes this possible is that the Fact is backed by a Map, and the dialect is MVEL, which supports this type of operation, when the map keys are strings.

The interesting thing about using the fact template, is that it makes it easy to perform lazy variable resolution. You may extend the above HashMapFactImpl to add Field Resolvers that may contain specific logic to retrieve field values. To do this with an object tree, especially dynamic objects, would require either intercepting the call to retrieve the field via AOP and injecting the appropriate lazy value, or setting the value to a dynamic proxy, which then performs the lazy variable retrieval once triggered. In either case, this simple Fact Template solution maybe all that you need.

Corticon Rule Engine Review

I've recently had an opportunity to give Corticon a test run. Corticon consists of desktop rule studio and a server component that runs the rules. The server component can be embedded inside a Java application, or run as a standalone server. The studio is used to create a vocabulary and the rule sets.

The approach Corticon uses is quite novel in comparison to the other rule engine vendors. They don't use the Rete Algorithm, instead they rely on compile time rule linking. This means that the rules are deployed to the server compiled, and the rule firing order is already calculated. The concept is undoubtedly more palatable to some organizations that find traditional rule engines a little unyielding.

The other novel idea is that the rules are compiled against a vocabulary (an ontology, if you will) rather than an object model. This means that you can submit dynamic datasets to the rule engine rather than relying on strict structures as is the case with Jboss Rules. It also seems that Corticon has licensed the Oracle XML SDK, which allows them to query the database, retrieve the result in XML, pipe it to the rule engine, and produce a result, all without any custom code. The actual rule writing and vocabulary management occurs in the desktop rule studio. The studio gives a user a decision tree type of interface but with some enhancements. The rule writing is done in a 4th generation language, meaning that the rule author uses the vocabulary, boolean algebra, truth tables, discrete math type of programming, "table driven algorithm programming". It's quite nifty but comes off a little limiting to a hard core developer.

And now for the negative:
In order to extend the language, you need to write java code, and then add it to the classpath of the desktop rule studio. This means that if you distributed the desktop rule studio to your business analyst, now you have to redistribute your jar file. You will need to redistribute a new jar file every time you extend the library with a new function. This is almost impossible to do in large organizations with dumb terminals, software packaging, etc...

The actual desktop rule studio is cluncky and is missing some basic keyboard shortcuts, like the "delete" key. There is also no security of any type. The rule author is able to change the vocabulary and any rules. The actual 4gl language at times seems more confusing than it's 3GL counterpart.

For the developer, Corticon is a disaster. The API library consists of a handful of classes. One to create a server, and run the rules. Another to deploy the rules to the server, and yet another to do some admin RMI calls to the desktop rule studio. Mind you, RMI calls for an enterprise level application is a little laughable. The functionality of the RMI calls are also a little silly. You can start the studio, shut it down, and open a vocabulary. In comparison, ILOG has an immense API library. I would wager that you can pretty much control any aspect of ILOG. And of course, JBoss Rules is open source, which makes them quite friendly to developers.

And yet more negativity:
There is no way to control the rule firings. There is also no way to disable certain rules from firing, or manage the rules in a central location. The mere fact that the rules are compiled in the desktop rule studio by the rule author and written as a file in binary format makes any form of enterprise rule management a joke. A nice web 2.0 gui for rule management like with Drools or ILog would also be nice. Why is it that I need a desktop app to manage my rules?

In summary, I think Corticon is quite a nifty concept, but is not a mature enterprise framework. It maybe useful in some limited fashion as glue logic, but it doesn't belong as an enterprise class rule engine. At least, not yet.

An addendum: Tibco iProcess has licensed the Corticon Rule engine for it's decision server. This does not change my opinion of Corticon, but reduces my opinion of Tibco. As a side topic, I think Tibco Business Work is an impressive tool, but the entire iProcess stack is quite bad and convoluted. I think Tibco just wanted to put something out quickly, so, they pieced together some varying half finished technologies.

Rule Engines

Recently, there has been a proliferation of rule engines. A rule engine is by product of AI research. The basic premise is that a user is able to create a bunch of atomic units of knowledge. When the rule engine is presented with a state of the world, the rules all fire. After all the firings have settled down, the new state of the world is an outcome. A lot of problems are easier to implement using rule engines than the more conventional programming. For example, system that relies on heavy usage of knowledge with deep trees - imagine many layers deep of if/elif.

There are a couple of major contenders. For the corporate world, there is ILOG and FairIsaac. For the open source, there is Jboss Rules and Jess. Jess being the original java rule engine, and the closest to the original NASA Clips system. Clips being the system that created the rule engines. Personally, I am most familiar with Jboss Rules, ILOG and to a much lesser degree with Jess. This should not be taken as a diss on FairIsaac or any other rule engine.

Each rule engine, at its core, is based on the RETE algorithm. There are a lot of variations and enhancements, but each rule engine implements the core algorithm. The algorithm is used to find rules that need to be executed for a given world state. Imagine thousands of rules, and a good search algorithm becomes critical to a useful rule engine. The RETE algorithm acts as the control flow in a regular language.

The major blocking point to a wide adoption of rule engines is their dynamic nature and unpredictability. If you define a thousand rules, it becomes difficult to know how the rules will interact in every situation. This means testing and scenario generation is critical. This also means a much more mature infrastructure and process than most organizations have. The advantages are huge. You can explain to your user exactly how a given outcome was reached. Display the rules, modify the rules, add rules, all dynamically. You can even simplify the rule model such that your users can create their own rules.

The next blocking point is the rule language itself. The language has many requirements. For example, some people want the language to have a natural language feel. Others, want a clean interact with the existing java system, while others seek some middle ground with a scripting language. ILOG does this very well, with a natural language translation tool. Jboss rules has a more rudimentary natural language translation (DRL - DSL) but supports a wider language group.

I find Jboss Rules to be easier to get started with, but a large and mature organization should probably take a look at a vendor product for the scenario generation, and rule management infrastructure, something Jboss doesn't quite have yet. The vendors also have much more mature rule editing GUI's.

Rules and Monads

Word of the day, Monad: I believe Leibnitz first applied it to science or perhaps philosophy. In any case, the word is defined as "...an indivisible, impenetrable unit of substance viewed as the basic constituent element of physical reality..."

I am currently contemplating a system that will allow the creation and testing of validity of atomical units, monads. Small units build up to larger units, to even larger, to ever so more larger. At the end, by testing the indivisible, impenetrable unit of substance, I can find out whether the sum of sums of sums of sums equaling to an even larger and more complicated entity is valid.

Well, solution one, write lots of lines of code and hope to God that nothing changes and your infinite amounts of if-then-else statements nested ever so many layers deep across ever so many classes in your lovely object oriented patterned out piece of shit.

Solution two requires a bit of imagination, a workflow, and rules on monads. Let's assume that you've built some wonderful gui that allows you to create and modify rules.

Now, each monad can actually be represented in a 2 dimensional space. Each monad is for a specific type of data: stock price, live stock, live stock weight, insurance policy, etc... Each monad represents a specific type of data. Let's say that's the X-axis. Each monad is also for a given instance of that data. For example, a price is uniquely identified as cusip/date, and each cow is uniquely identified by its id. Each transaction has a transaction id, etc... We can call this the Y-axis. So, given the X and the Y, we can uniquely identify every monad.

At this point, we can right a rule that tests that monad. So, we can test every indivisible unit, and then we can test the summation of these units using other rules that perhaps only group the indivisible monad rules. At the end, all this adds up to a single high level rule that tells you whether your data is valid or not. Along the way, your rules may change data, or take you on a different path, but that's another blog entry.

It's actually extremely elegant. I apologize if my description skewed the beauty or confused the matter. The more I think about this, the more I believe in it. Now you may be wondering: am I not describing an expert system built using a rule engine. Somewhat true, in fact I am describing an over simplified rule engine. The big difference is that instead of allowing each rule to be a self contained entity, this model assigns a rule to a particular type, and the rule outcome to the type and instance of the data object. So, instead of having a bunch of free for all rules, you have a controlled rule structure, and a controlled firing. This is one thing that I found to be very brittle in rule engines. The writer of the rules does not know when the rules will fire and has to be very careful in writing the rule filter. Now, some will argue that that's the whole point. Well, try selling a none-deterministic system to your business manager who is weary of the whole thing to begin with.

Anyways, as you can see I am a big fan of externalizing the ever changing business rules from the underlying system framework. This is an interesting idea, and really should be developed further.