Pons Asinorum

Business Language

2012-01-15T09:28:00.000-08:00

I've been toying with a new type of architecture - or more specifically another approach to solving an age old problem of ad-hoc reporting.

The question usually goes something like this:

We need a flexible, ad-hoc reporting solution which will allow us to do what ever we want whenever we want. It has to be dynamic and easy to use, and should give us the complete power to answer any business question. But we also want the ability to create presentable formatted reports and link together any type of data we so chose.

The answers vary from business vendors like SAP Business Objects or Oracle Business Intelligence tools, to smaller vendors, to various products pieced together like SSAS cubes with SSRS reporting. But, in all cases, the solution is always lacking that flexible and dynamic nature. With all tools, there is always a heavy IT presence. Some of the tools attempt to simplify this by creating an abstraction - such as the Business objects universe concept. This is effectively a meta-model around a physical representation. But this is very limiting as it restricts the use-case to a relational structure. Cubes are a very cool technology but also very limiting and the truth of the matter is that the users care about small data sets such as current days data or even a small subset of current days data - and in a number of cases the cube becomes too burdensome.

So the alternative approach was to actually give a new language to the business user directly. Goldman did something like this with their slang language. Imagine a language which is simple enough and descriptive enough for the user's to use and lacks all the very technical details of an actual technical language like Java or C#. We should accept that user's already do a fair amount of programming with the tools at hand such as VBA, and combination of Excel/Access. At times, it is very impressive some of the things these guys produce and at other times it is quite scary.

So, let's say there is a functional language that allows the user to manipulate the various available artifacts - i.e. back-end systems. Some functions retrieve data based on the supplied parameters and other functions can perform actions like calculations or data changes. Before you know it - you've provided a set of very powerful tools and what emerges is that the users start tinkering with the functions to build more dynamic and abstract features that were not previously imagined. What I came to realize is that revolutions are slow things that require tinkering, and at some point, something emerges which could not at all be previously imagined but once created makes perfect sense and is the most natural. The user's need tools to tinker, and that tinkering actually creates tremendous value and innovation. Innovation drives investment and growth, etc...

So, back to the language. The language should of course work within Excel, but should not be restricted to excel. Ideally, we bridge the language with perhaps a more functional cousin like F# and create a natural set of functions with a pure functional language. At the heart of this language is a data set - to borrow from any database driver - a TDS class - a tabular data set. Let's say the language only understands data sets not objects. Everything is a table of data or perhaps, a table of tables. Some functions create tables and some functions manipulate tables. It is also interesting to see how the roles of systems and enterprise architectures change. Systems become components which are forced to expose "functions" which are manipulated from this functional language. This in turn forces systems to become more dynamic loosely coupled modules as it's not always known how an external function may want to manipulate the components of a given system. Looked broader at a higher level, the systems start to make up an overall platform - with the functional language acting as a glue that binds it all.

I think this approach gives the most flexibility - coupled with some of the more traditional options like cubes and canned reporting - perhaps even a vendor product. The approach of course is more technical and requires some learning but it strengthens the bond between user and system and makes a user an investor as they are now in affect developing the system - or platform. It is a wonderful thing to create something that you're proud of that adds value - that feeling can be given to the user to strengthen the bond to the platform and the overall enterprise architecture.

0% unemployment

2012-01-15T08:57:00.000-08:00

I'd like to once again deviate from my regular technological themes and instead discuss philosophy or rather some shade of philosophic economics. I was watching CSPAN the other night and a show came on discussing poverty in America. There was a panel of speakers including heavy weights in the topic along with some celebrity personalities sprinkled in. Some of the speakers espoused what is almost close to pure socialism that it was the responsibility, nay a requirement for corporations to hire people that they don't need and, which in turn, will create jobs for the poor people, and will help them get out of poverty. Now what the corporations are doing instead is hording the cash and rewarding their management ranks for driving efficiency - i.e. doing more with less people.

I am in technology and in laymen terms that means that my job is to build systems, which create automation, which in turn, create efficiencies or in other wards reduces headcount. I am even horrified when management hires too many people for one project as it destroys efficiency, makes the process tremendously burdensome and bureaucratic, and this eventually leads to either a failed system or a very poor system. These projects always cost more and deliver much less.

So - how do you achieve 100% employment - or more specifically, some higher number like 95% as in every society there will be some group of people which either cannot or won't work. In America, the official unemployment rate is just under 10%, but there is probably another 5% or more of unemployed which do not collect unemployment benefits. This means America has 85% employment - looked from this angle, that's not bad at all. But, it would be important to understand the distribution of the income across the 85%. Is the income completely skewed - i.e is it a nice bell curve or a steep hill with a cliff drop off.

If the bell curve is nice and even that would mean that we have a strong middle class and no fear of a revolution. On the other hand, if the curve is more of a cliff, then there will be tremors produced from the mountain, which could erupt and bring down everything around it. The problem is that individuals are rational, but people are not. Our political process is also looking more and more weak with the least worthy striving for notice. Politics and education should be the epitome of the most intellectual of us all - rather finance is where we spend our energies.

I am confusing various topics - but they are all related. Our wealth and the most brilliant talent seems to be focusing either on financial engineering or entertainment be it social type websites or some form of media.

It is very easy to get into a negative feedback loop and it is not easy to get out of it as the decisions and choices are counter intuitive and not at all obvious.

One should always better oneself - and perhaps the forces of good and right will prevail over the forces of ignorant.

This is not the most fluid of posts as I don't yet fully understand how all the pieces fit together.

Serializers

2011-10-21T09:19:00.001-07:00

https://github.com/eishay/jvm-serializers/wiki/

http://code.google.com/p/thrift-protobuf-compare/wiki/Benchmarking

Zion Business Intelligence

2010-10-31T19:32:00.000-07:00

How about a hypothetical scenario -

You are an Enterprise Architect in a Global Investment Bank and tasked with finding a solution to the data fragmentation problem.

And this is how it happened -
This particular bank has many data files flowing whether which way between various groups in the organization. Product controllers take a feed from front office systems, and market risk another feed, and credit analysts take a third feed, except the 3 feeds are generated differently, and have slightly different points of view, and in some cases, even different trade populations. But then, someone comes along and says they need to see 3 numbers - 1 from market risk, 1 from credit analysts, and 1 from product controllers side by side. And you say - that's impossible because it will take months just to analyze whether the data populations are the same, and even if there were, the numbers are produced at different levels, and even if they weren't, there is no way to show them on the same report. And even if there was, this is a 6 month project, and who is going to pay for it. And then they say, "aren't you an architect", and you hesitantly say "yes", and then they say "so go fix it."

Rabbit Hole
So, how to fix it. Core data primarily generates from front office systems and then flows through the rest of the organization. You can think of it as streams of water branching off into many smaller sub-streams, and then further branching off, until eventually the stream is to weak to branch of and just sips into the ground.

Well, easy breezy you say. Pervasive BI, data warehousing, Online Analytical Processing, bottom up data-warehousing, top down, bus architecture, centralized architecture, federation, hub and spoke, relational, dimensional, operational stores, data marts, Inmon, Kimball, conformed dimensions, Boyce-Codd normal form, 3NF, .....

We just need to take this mix - shake it up, and we'll get ourselves a fancy enterprise data architecture or perhaps something strong enough, maybe with a cherry - so we can forget the whole thing ever happened.

Blue pill or the red pill
What to do.

Well, Kimball likes the bus architecture, so, let's give that a go. The bus architecture consists of a bottom up or was it top down approach where you basically start off with a bunch of data marts, which then flow into a data warehouse. The data marts are primarily operational store type structures, while the data-warehouse is a pure data-warehouse, star-schema and all.
The problem, of course, is that you effectively already have a ton of little data marts all over the place, which don't conform. Ah, that's the problem, we need to have conformed dimensions. Right, and we do that how exactly? The other problem, of course, is that this is the same data being treated in a slightly different way, maybe with a slightly different trade population or attribute set or granularity or perhaps with a different temporal point of view. Seems awfully wrong to have a bunch of data marts storing the same data, which you then have to reconcile all together.

Right you say, let's go the other way, Inmon likes top down, so, let's create a big data warehouse, which then populates the data marts. So, who is going to build this monstrosity exactly? Well, can't be the individual business groups, because they are not stupid enough to take on a project like this, so, it would have to be some central group away from any particular business line, and close to senior management, 'cause this is going to cost a lot of money. So, the group is created, except they don't know what they are doing - very technical guys, but don't understand the business at all. And if by some miracle they do, they can't keep up with it. If this project actually succeeds, which is highly unlikely, and actually have the right amount of data, which is frankly impossible, it will still fail, because the business moves just too damn fast, and at the end of the day, they will never ever be able to fully understand and own what they are storing. So, after some time, and a whole lot of money, this will be dramatically killed off.

What about federation you say? We just need a magical vendor, and all our problems will disappear. You see this vendor will create an abstraction on top of our asylum, and this way, we will present a clear simple view shielding the end user from the underlying complexity - easy, breezy. I suppose that could work if it was actually possible to build enough complexity into this thing to actually bridge something which is fundamentally diverging and is actually able to perform at the required speed, and is actually supportable. So, let's just kill that idea for now, before we embarrass ourselves much further. Perhaps, if works for you readers out there - if you happen to like this federation thing - check out Composite, Inc - it's all the rage these days.

Red pill
Reality it is then. Well, seems to me we need a new philosophy. Let's call it Zion. There are a few core principles of this theory:

1. Each piece of data has 1 and only 1 owner.
2. The owner is the only one that can change this piece of data.
3. Each piece of data has a natural key and a surrogate key

The distinction between this theory and all the above is that this one says that first you need to understand what data is before you start deciding on how to deal with it. And the first question to answer is who is responsible for it? Responsible is not the same as which IT group owns the data-warehouse, but responsible, as in which business group is responsible for this data. If you need to change it, who best knows how to change it? How to evolve it? What it means?

If you answer this question, now you know who is going to build the data store for this type of data - and this will become the golden source for this data.

To be continued....

Social Networking

2010-05-21T14:46:00.000-07:00

There is something interesting occurring in the world. People are exposing the most mundane bits of their life via twitter, friendster, youtube, etc...

What is interesting is where all of this is going. I think the next step will be that teens will record their life continuously and post bits of it for the rest of the world to see.

There are some examples of this already occurring:
http://research.microsoft.com/en-us/projects/mylifebits/

http://qik.com/

It needs to be effortless, continously recording, maybe an addon device behind the ear, connected to an iphone in your pocket.

After something interesting happens, you can then open the iphone, scan back, cut out the relevant section and share it via some social networking site.

Connecting to Sybase IQ 12 from Analysis Services 2008

2009-09-17T10:07:00.000-07:00

I've been struggling with this task for awhile now. Since MSSAS 2000, you can only use the .NET provider or the OLE DB provider.

If you have the Sybase 12 client installed, you will have the "Sybase ASE OLE DB Provider". If you also install the Sybase IQ client, you'll additionally have the "Sybase Adaptive Server Anywhere OLE DB Provider 9.0"

On the ODBC side, you should have the "Sybase IQ", "Adaptive Server Anywhere 9.0", and "Sybase ASE ODBC Driver".

1. Create an ODBC connection using "Sybase IQ" data source.
a. Give it a name in the Data source name field in the ODBC tab
b. Provide your username and password in the Login tab.
c. Enter the server field in the Database tab. You'll need to get this information from your DBA. For me, it was <hostname>_<instance name>
d. For the network tab, select only TCP/IP, and enter: host=<host>;port=<port>

2. Open Analysis Services, create a new data source using "Native OLE DB\Sybase Adaptive Server Anywhere OLE DB Provider 9.0".
a. In the server or file name entry put the name of your ODBC data source that you created in step 1.
b. Press Test Connection, everything should work.

3. In the data source views section, create a new data source view given your data source that you just created. It seems importing the tables directly fails with some arcane error.
a. Don't select any tables, just click next, until an empty view is created.
b. You can now create named queries like "select * form <table name>"

Healthcare

2009-08-18T10:58:00.000-07:00

Let's ignore the red versus blue and blue versus green and all the other colors of the rainbow. Let's also ignore all the stupid nonsense coming out of news talk shows.

What's at stake seems to be a lot more fundamental.

Is it or is it not the role of Government to protect the life of its citizens. Must the Government protect the "inalienable rights" of man - life, liberty and the pursuit of happiness? Or is such responsibility more fickle and has dependencies on array of variables such as politics, and budgets and costs, and personal gains?

Let's see if we can look at this problem at an even more basic way. Life is fundamentally unpredictable - complexity theory. Tomorrow we can either be hit by the bus, get cancer, both, or neither. In the case of neither, we, humans are conditioned to ignore the perils until either one or both of the above occurs. Now, is it not the role of a stable, evolved society to protect its weak and young? What does it say about our society when we concisely ignore the weak and the young? Are we as a society egotistical, thinking like every other great society before us, that we can do no wrong, strong survive, weak perish, one must accept the law of nature?

Some statistics on the makeup of America, total uninsured by various categories:
http://www.kff.org/uninsured/upload/7451_04_Data_Tables.pdf

http://facts.kff.org/chartbooks/State Variation and Health Reform.pdf

XML/A via Flex

2009-02-17T10:18:00.000-08:00

XML/A stands for XML for analysis and is an industry standard for using XML to retrieve data from an OLAP cube.
The spec is available at http://www.xmlforanalysis.com/xmla1.1.doc


<SOAP-ENV:Body>
 <cxmla:ExecuteResponse xmlns:cxmla="urn:schemas-microsoft-com:xml-analysis">
     <cxmla:return>
           <root xmlns="urn:schemas-microsoft-com:xml-analysis:rowset" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:EX="urn:schemas-microsoft-com:xml-analysis:exception"> 
               <xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema" targetNamespace="urn:schemas-microsoft-com:xml-analysis:rowset" xmlns="urn:schemas-microsoft-com:xml-analysis:rowset" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:sql="urn:schemas-microsoft-com:xml-sql" elementFormDefault="qualified">
                   <xsd:complexType name="row">
                       <xsd:sequence>
                           <xsd:element minOccurs="0" name="_x005b_Counterpart_x005d_._x005b_All_x0020_Counterparts_x005d_._x005b_ABN_x0020_AMRO_x0020_Bank_x0020_N.V._x005d_" sql:field="[Counterpart].[All Counterparts].[ABN AMRO Bank N.V.]"/>
                           ...
                       </xsd:sequence>
                  </xsd:complexType>
               </xsd:schema>
               <row>
                   <_x005b_Counterpart_x005d_._x005b_All_x0020_Counterparts_x005d_._x005b_ABN_x0020_AMRO_x0020_Bank_x0020_N.V._x005d_ xsi:type="xsd:double">10</_x005b_Counterpart_x005d_._x005b_All_x0020_Counterparts_x005d_._x005b_ABN_x0020_AMRO_x0020_Bank_x0020_N.V._x005d_>
                   ...
               </row>
           </root>
      </cxmla:return>
 </cxmla:ExecuteResponse>
</SOAP-ENV:Body>

This is a sample response. In this case, I've used Pentaho Mondrian as the cube provider.
This XML can be rather easily walked with Adobe Flex.


var message:XML = ...
var soapEnv:Namespace = message.namespace("SOAP-ENV");
var cxmla:Namespace = new Namespace("cxmla", "urn:schemas-microsoft-com:xml-analysis");
message.addNamespace(cxmla);

var xsd:Namespace = new Namespace("xsd", "http://www.w3.org/2001/XMLSchema");
message.addNamespace(xsd);
        
var sql:Namespace = new Namespace("sql", "urn:schemas-microsoft-com:xml-sql");
message.addNamespace(sql);
    
var body:XMLList = message.soapEnv::Body;
var executeResponse:XMLList = body.cxmla::ExecuteResponse;
var schema:XMLList = executeResponse..xsd::schema;
var complexType:XMLList = schema.xsd::complexType.(@name="row")
var elements:XMLList = complexType.xsd::sequence.xsd::element;

The above will return a XMLList of elements within the complex type row.

A slight change, will produce a list of actual data rows.


var rows:XMLList = executeResponse..rowset::row;

Problem's for your CIO

2009-01-10T11:36:00.000-08:00

1. Business does not know itself.
Every organization struggles with the problem of data. Either it's too much data, or not enough data.

In the case of too much data, the analysts and the IT teams struggle to make sense from the barrage of data. A lot of times, IT is forced to archive data, and by so remove it from analysts access. In other cases, large amounts of data is simply stored, but no viable reporting is available, or only a small segment is reported on.

In the case of not enough data, local analysts are unable to get complete pictures because data is stored in multiple systems, and a lot of times, is localized to regions. This means that organizations struggle to get complete pictures. Imagine what that means for financial risk: What is my exposure to Bear Sterns? I don't know, it will take a few days to compile the report from various places. Don't believe me, this was the case at Lehman, and is the case at a Global Asiatic Bank. I would also bet, is the case, in almost very other institution.

2. Business doesn't trust IT to deliver.
The business simply does not not trust IT to deliver the solutions it needs. When solutions are delivered, they become large monolithic entities locking the organization in, and becoming an expensive constant expense. Any change to the systems, becomes an expense at times larger than the cost to build the system in the first place. In other times, IT simply looks like a dog chasing it's own tail. They move very fast, and have short time-lines, but the end products are not even remotely close to what the business wanted. IT seems to always have an excuse. I was once in a 3 hour meeting. The business analyst wanted account numbers. He wanted a system where he can debit one account and credit another account. For that, he needed account numbers, and amounts within those accounts. The developer for the system was trying to explain that the analyst didn't need accounts, but only had to enter the transactions and attributes of the money movement. The system can then aggregate those transactions in what ever way the analyst wanted. You see, the system was designed generically, and adding account numbers would intrude on that design. And, so they went, back and forth. Neither understanding the other. The end result, nothing. No account numbers. It was deemed that this feature wasn't go live critical, and can be addressed in the next phase.

The fault is actually neither IT nor the business. The problem is more basic. The business is very agile. It's able to move quickly and easily. If you've ever read a legal document or saw financial models, you'll understand. They can be very complex and very nuanced. Most technical architectures are not build to be nuanced. They are built to solve concrete Boolean problems. IT wants the problem defined in concrete terms, but the business is only able to articulate it's current understanding of the world. Unfortunately, tomorrow it will be a new understanding. And so, IT builds a monolith, because, that's what it knows how to build, the business changes, and the organization suffers.

This really saddens me. Technology is capable of solving so many problems, but instead it's relegated to addressing the most basic of operational problems, and even there, we struggle. Consider all the technical possibilities, neural networks, expert systems, machine learning, data mining ....

3. Business data is not clean.
So much of data is simply bad. Imagine how complex some of the systems are, and then imagine the possibility of an error. An error that doesn't generate an exception, perhaps, rounding, perhaps, a logic error, perhaps, an unforseen condition. The output data becomes bad, and so it flows through the system. A lot of times, enterprise architectures don't even do basic reconcilliation. Reconcilliaton requires active design, time and thought. Most architectures are organically produced, and are feature driven rather than any thoughtful design. The end result is frankenstein architectures and garbage data.

4. There are a lot of manual processes.
How many times have you created a new system, which actually creates more manual processes than it removes. New systems sometimes require users to enter data in multiple places, verify multiple places, etc... Some firms have massive operational groups, hundreds of people. Their sole job is to do what systems fail to do. Enter data in multiple places, reconcile, data entry, data messaging, normalization, etc... This is a horrible model. It's extremely error prone, not too mention expensive.

5. Heavy reliance on people.
I believe that machines and people have their roles. Unfortunately, a lot of jobs done by people to day, should be done by machines. Machines and people need to find a harmony. Some things we do very well, other things, machines. Sifting through large amounts of data should be the role of machines. Alerting us to unforseen circumstences, should be machines. Allowing us access to data, should be the role of machines.

6. Presentation
So few firms give through to the presentation tier. In most cases, it's an after thought. Some UI's are desktop, other's web, other's, something else. The end user is required to memorize what features are available where, what report has what, etc... God forbid, we standardize and unify the disparate systems. The user should have a single place to go, a single way to do something. The learning curve to learn how to use everything should either not exist, or be extremely small.

Lehman Brothers

2008-12-22T18:38:00.000-08:00

The last 4 months have been a crazy whirlwind. I was an employee of Lehman Brothers during it's demise. For those who know, that weekend went fast, without a single doubt, each of us expected a normal resolution; maybe a industry consortium bailout, maybe the Fed will extend a loan, maybe a last minute buyer. None of us expected a bankruptcy filing. That Monday felt like being shell shocked. Everything you worked for, all your hard work, gone, abandoned. All those meetings about tiny little minutia's of detail. All that time, coming up with the perfect design, the clean integration, the cleanest build file, the good interface - puff.

Some of the guys have worked their whole lives at the Firm. Think about it, you spend the last 20 years of your life building something. Sometimes you work late, sometimes even on weekends. You meet tough deadlines, you build out the system, you have projects, and Jira items, and future ideas for improvements, and then one Sunday evening, it all disappears in a single 20 second news-clip.

What can I say about that Monday, a lot of guys were afraid. Single earners, mortgage, kids, car payments, private school, dance lessons, ... It was never a problem before, the Firm was known for it's bonuses. The irony of having more money, is that your expenses rise proportionately. Some of the guys were hit very hard. A lot of the bonus is paid in Lehman Stock, part of the 401K is in Lehman, the firm also promoted personal investing in the stock. All in all, if you were with the firm for 20 years, you have a major percent of your worth tied with the company, and then it's gone.

The next few weeks gyrated between moments of team comradery and private lonely introspection. Every person came to work on time and stayed for the full day. Most of the guys, myself included, kept to the Lehman dress code of full business attire. I think it was a out of respect for the Firm, or perhaps, just routine.

Barclays purchase of the firm provided a rare glimmer of hope, a possibility of normality. It wasn't so much as having your job saved, which is of course very important, as a possibility of saving what you've built. Someone using what you've worked so hard to create. As time dragged on, it became clear that Barclays had no intention of taking our system, so, a glimmer of hope slowly changed into despair with a dash of anger. Barclays, of course, wound up laying off most of us. The system I've worked on was trashed.

Today, I am at another company, assigned the task of comparing Lehman system's to my new employer systems. This is how Barclay's employees must have felt when they found out their company was buying Lehman. It's us or them. And, so I do what is requested of me. I criticize and destroy the very systems I promoted, just a few months ago.

I have the deepest respect for the Firm. I have never worked for a company like that. They really tried to create something more then the sum of themselves. I shall miss it.

Intentional Software

2008-10-24T07:06:00.000-07:00

After working on the Drools Flex Editor, I started thinking a lot more of what it would take for it to be usable by non-developers, i.e. domain experts. This narrative of thought brought me to heavyweights such as Charles Simonyi, founder of Intentional Software and
Gregor Kiczales, founder of AOP, latest paper. It also led me to a lot of other concepts such as the law of leaky abstractions, history of software, Lagom Process, along with the more obtuse concepts such as the omega number, and people like Gregory Chaitin.

And of course, what would our industry be without it's acronyms: MDA, DSL, BPM, UML, DDD, SOA. There are plenty of others, but I think I've made my point.

That's a lot of information to digest especially if you looked up Chaitin, which would have led you to our Founding Fathers: Leibniz , Turing, and Godel. Now ask your self: what kind of a thought narrative can take a person from Adobe Flex to Godel. It would almost be funny.

Now, let's step back for a moment. A lot of people argue that software development should be reduced to visual tools. A counter argument can be that mathematicians do not use visual tools to draw up their equations. They use blackboards and chalk, the most primitive of tools. Excel has been praised as the most successful intent based system. But, if you look at it, it's not visual at all. At best, it's a basic grid, with cell co-ordinates, and a blank text input box allowing manipulation of cells. Another argument is that business people are somehow not smart enough to program. It takes a special kind of mind to generate code. The fallacy with this statement is that business people already code, just not in the typical "tech" way, but rather in their own domain. Their interaction with excel, the domain expertise, the manipulation of that domain expertise - can all be considered coding. They manipulate their symbols to achieve their goals. The only thing separating technology developers and domain experts is which domain they are experts in.

There was a quote from Charles Simonyi that went something like this: If we don't expect business people to learn how to code, why do we expect coders to learn the business. Each path is extremely inefficient and is rife problems. So, instead, let's allow each group to focus on what they do best. Developers should stick to technology, business people should stick to business.

So, we have established that a business person is capable of performing some form of "development" to encode their domain expertise into a set of steps, "their intent". It is also probably safe to assume that a business person understands concepts like if-then-else, for-each and standard algebra. It is also safe to assume that they know nothing of JSP, Servlets, JMS, EJB, transactions, XA, JDBC, SQL, Java, Class, public/private, encapsulation, polymorphism, design pattern, singleton, facade, heap, stack, binary search tree, NP-complete, and on, and on, and on, .... So, where does this leave us? I think it means that software development stops being the pure domain of developers, and instead is split between developers and business people.

If we look at a typical business system, we can see that it has inputs(JMS, GUI, etc...), a concrete data representation model in the form of a database schema, complex output in the form of varied reports, processes that criss-cross the system that have some triggers such as external events (JMS, Schedule, User, etc...) . There is also business logic in the form of calculations, business steps, if-blocks, etc... sprinkled through the system. Some of it lives embedded in the report logic, others in the processes, and some, perhaps, even implicit in the data storage or data format.

I think we can start to take steps to separate the domains. Process flows attempt to separate the process logic from the system logic. Web Services attempts to expose the individual services and by so reduce the hard linking between services. Business Intelligence is attempting to expose the data to the users and allow ad-hoc manipulation. Proliferation of domain specific languages, online compilers, rule engines is a sign of the desire to separate the system from the business rules. Hibernate, JDO, etc... are attempting to isolate the system from the underlying data stores, and map out the data definitions. Ontology's are attempting to bridge the interaction of a human defined relationships and a system. Mashups - i.e. http://www.programmableweb.com/, Yahoo Pipes, are yet more examples of technical concepts being exposed to non-technical people. All these things, in my opinion, are converging on the same topic of intentional programming.

Alpha Release of Drools Flex Editor - 0.5

2008-10-21T12:27:00.000-07:00

Orange Mile is proud to announce the long awaited alpha release of a Drools Rule Editor in Flex.

http://code.google.com/p/drools-flex-editor/

The current release includes all the Flex pieces without the rather basic server side code for rule compilation, and code completion. This will be available in the final 1.0 release.

This is the beginning of having rich enough components available within the system that can allow the user/admin to directly manipulate the business rules without having the long development cycle.

Orange Mile is Expanding

2008-10-21T08:23:00.000-07:00

We are proud to welcome Venu to the Orange Mile Team. Venu will also act as a contributing author to the blog.

Although, Orange Mile started as a small math with a single avout, it has since expanded, grown and matured.

We have become a devout following. At times, looked down upon by the saecular world, but always, pursuing our dreams.

Entry written in the style of Anathem.

Orange Mile Security Release - 1.1

2008-10-16T06:45:00.000-07:00

Orange Mile is proud to release version 1.1 of Dynamic Rule Based Security.

The new features include:
1. A complete example based on Spring Security - see orangemile-security-test.war
2. isGranted JSTL Tag

http://code.google.com/p/dynamic-rule-security

Doom and Gloom and the Economy

2008-10-10T06:25:00.001-07:00

Although, this is a tech blog, I strongly believe that a good developer must be an Erudite. Given this, I feel justified in writing about the economy in this blog.

We are living through historic times. Mid last year, I started getting very scared and started blogging about randomness and the economy.
http://orangemile.blogspot.com/2007/05/black-swan.html
http://orangemile.blogspot.com/2007/07/supply-of-money.html

You will notice that in the Supply of Money entry, I actually wrote about the likely hood of the collapse of world economy due to the unsustainable supply of fiat money.

What's interesting about those entries and that time period is why did my mind shift to the arcane topics of money supplies, carry trades, fiat money, when the blog entries before and after clearly deal with the arcane topics of technology. Perhaps, my subconscious started to pick up on the feelings of uneasiness in the global market; hiccups if you will. I can't possibly attribute those entries to knowledge, because I am simply not qualified to speak of money supplies, fiat currency, and carry trades.

So, what is happening today is a global loss of confidence. What is interesting is that companies, specifically, banks are hoarding cash rather then people. I would argue that if people started hoarding cash than we're all doomed. The global economy would come to a scretching halt. Chinese economy will collapse, probably throwing that country into either Marshall law or revolution. America and Europe will fall into severe and prolong depression taking the rest of the civilized world with it. Africa will fall into an even lower level of sustainability with probably wide ranging civil wars due to lack of food and an acute demand for natural resources such as diamonds, gold, and oil. India's economy will also take a severe beating, but I think they will remain a loose democracy. If they position themselves well, they may end up being the next superpower.

Right now, the US government is printing money at an ever faster clip, giving it away, almost for free, and nationalizing large areas of the financial industry. Money is flooding the global economy. What's interesting is that we are actually in the period where money is actually disappearing. As the perceived value of assets fall, money disappears. The US Government then tries to fill in the gap of lost money, by providing more money to the institutions whose wealth disappeared; hoping against all hope that the newly provided money will be used to create more money by the institutions. Let's recap how money gets created. A person decided to do something on credit, let's say by a house. They go to the bank and say give me 300k to buy a house. The banks gives you the money, and you go buy a house. The thing is that the 300k is actually some other depositor's money. Money the bank doesn't actually have. What's happening now is that the bank thought it had 300k loan asset. But instead, the 300k is really only a 200k asset. This means that if you default, the bank looses 100k of someone Else's money. If enough loans do this, the bank won't be able to cover the loses, confidence in the bank erodes, people start to retrieve their deposits, and of course, after some interval, the bank simply runs out of money to give out to depositors. This is why the FDIC was created. This is a standard Ponzi scheme. In other words, if the asset side of the bank balance sheet starts to reduce, they will reduce the amount of new loans they can give out, and by so, reduce lending, which will probably drive the interest rates up because there are less institutions lending. This actually means the opposite of what I said earlier, money isn't destroyed, the rate of creation just reduces. The bank industry is structured as a very calibrated entity, with a minor hiccup in cash flows or perceived cash flows destabilizing the entire industry. The Fed is trying to erase the perceived losses from the Bank's balance sheet, and by so, start up the loan process. Another interesting thing is how many industries rely on having a continuous supply of new loans. Imagine a ponzi scheme applied to the car industry. If Ford borrows money to pay it workers, hoping that in the future it will sell enough cars to pay back the loan, except it doesn't, so it borrows more to keep going. At some point it actually needs to borrow from Person A to pay Person B, and on and on. This should end at some point when there is no-one else willing to lend to the said company. But, unfortunately, for most companies there is always someone willing to lend. This is partly due to the obscurity/opaqueness of the financial industry. Now, our current scenario, where the said company can't get a loan not because of the financial condition of the company, but because of the financial condition of the lender; all lenders.

What the Fed is trying to do now is fight the deflationary path. Money is becoming a scares resource, not because there is not enough of it, but because banks are hoarding it. A lot of people are also talking about hyper-inflationary model. I don't see this happening, even if I believed it for awhile. In a hyper-inflationary model, you have too much money. This is unlikely because the Fed can always mop up the money supply; and because the world is dollar denominated, at least for the foreseeable future. I think what is more likely is nationalization of a number of areas, and banks, a significant increase of money available to the banks, guarantee of bank assets, forced reduction of inter-bank rate, and a forced reduction of the interest rate payed by home owners.

Interesting reads:
http://en.wikipedia.org/wiki/Fractional_reserve_banking
http://en.wikipedia.org/wiki/Credit_default_swap

There is another animal that hasn't received much news: credit default swaps. This instrument is a form of insurance against default. The problem is that this is not based on anything, and the current amount of outstanding CDS is a few times larger than all money combined ever produced. This means that the government needs to do its darnest to make sure those CDS contracts never come due, because if they do, all financial institutions will file for bankruptcy, governments will default, end of the world, etc....

So, what is the government to do. Money must be made cheaper to a point where it is basically free. This will allow the banks to start to give out loans, this would spur the market for re-financing, which should save some borrowers. At the same time, the government will probably start with the regulation. We will see a period of a slow down - recession, in which the disaster of the day, will become a distant memory, and we will start up with the next bubble, maybe energy, maybe the housing sector again to a lesser form, although, this will probably be regulated to the gills. My guess energy or commodities. But the bubble won't start until the people regain confidence, which will take a few years.

I repeat, the Fed must make money cheap. After a recovery, they will attempt to make money more expensive again to starve off another bubble, but they will tread very lightly to avoid any more panic. This means that rates will stay cheap, or only very gradually over a long interval will start to go up.

Of course, there is another animal in this picture: US treasury bonds backed by our taxes. I don't fully understand this animal and it's relationship to the money supply, but hopefully, in the next few blog entries...

Release of Microsoft Analysis Services 2005 Automation SDK

2008-10-07T10:37:00.000-07:00

I am proud to announce the long awaited release of the Microsoft Analysis Services 2005 Automation SDK.

http://code.google.com/p/mssas-automation/

The library allows a java developer to automate the creation and modification of a MSSAS 2005 cube. The design consists of codifying most of the XMLA 1.1 specification into java pojo's via Jibx binding framework. On top of this core library, it then becomes trivial to codify specific design patterns or utilities to automate or speed up the creation/modification of a cube.

How not to be a turkey - a dead turkey!

2008-10-02T06:45:00.000-07:00

The idea is to build a little web app that will scan the common news sources nightly, and compile a score for different words on how negative or positive the topic is described. For example, regarding the economy, the system should pick up speeches from the Fed, congress discussions, etc... The idea behind all this is from the Black Swan Book. The theory goes that the night before Thanksgiving, the turkey should have the highest confidence in the goodness of humans.

To achieve this, I will need an NLP mood analyzer, or in other words, Sentiment Analysis. Some open source tools to accomplish this are:

NPL Libraries:

http://www.opencalais.com/ (Reuters Web Service)
RapidMiner
JavaNPL
LingPipe
Jane16
http://garraf.epsevg.upc.es/freeling

Knowledge Understanding

News Sources

http://cookbook.daylife.com/
http://thomas.loc.gov/home/c110query.html (Government Enrolled Bills)

Ekman's research on universal facial expressions
[happy, sad, anger, fear, disgust, surprise]

Frustration – Repetition of low-magnitude anger
Relief – Fear followed by happy
Horror – Sudden high-magnitude fear
Contentment – Persistent low-level happy

Informatica PowerCenter Automation SDK 1.0.0 Released!

2008-09-23T12:33:00.000-07:00

I am proud to announce the release of Informatica PowerCenter Automation SDK; brought to you by the good people at Orange Mile, Inc.

http://code.google.com/p/informatica-powercenter-automation

The library allows the automation of repetitive patterns when creating or changing Informatica PowerCenter Mappings.

With the proliferation of development abstraction platforms like Informatica Power Center, Tibco Business Works, BPM, rule engines, etc... it becomes more and more possible to build automation software to automate the development abstraction software. In other words, these tools provide a meta definition to build certain services. In the case of Informatica, those meta definitions are geared towards ETL tasks. The automation software is then able to manipulate the meta pieces to automatically generate those services. In other words, you have a system that knows how to manage the meta service pieces.

Dynamic Rule Security is Released!!!

2008-09-22T18:36:00.000-07:00

I would like to announce the release of the 1.0 version of Dynamic Rule Security brought to you by the good people at Orange Mile, Inc.

http://code.google.com/p/dynamic-rule-security/

I think this release may very well revolutionize the way application level security is handled. Although, the first release is somewhat simplistic in how it manages the rules, I believe it will serve a large majority of the systems out there.

The next major release will focus on expanding the rule management, adding tag libraries, and adding support for direct instantiation.

Let me count the ways... I HATE Spring Security ACL

2008-07-24T19:24:00.000-07:00

Elizabeth Barrett Browning wrote a Love Poem "How do I love thee? Let me count the ways..." It's a lovely poem, but I am forced to use the same core phrase in a very negative connotation, and applying to technology, which "some" believe has no soul. More on that in a different post.

Now, Let me count the ways in which I hate the Spring Security Acl implementation. In any other setting, I would have written this off as some poor wanking by some poor wanker, but unfortunately, in my prior post, I've vowed to add property based security via a rule engine as an add-on for Spring Security. What I failed to realize at that writing is that Spring Security seems to be split into 2 sections. The core security, which has things like app server plugins, role, and principle management, etc... This section seems rather decent enough. Perhaps, a bit configuration heavy, but hey, that's Spring for ya. Now, this other section, the Acl section is a complete and outer fuckup. The irony is that this is a re-write of an even worse implementation.

Now, listen you Spring theists:
Why create an ObjectIdentity interface that wraps a serializable identifier, and then implement a ObjectIdentityImpl, only to cast the serializable identifier to a Long in both the BasicLookupStrategy, and the JdbcMutableAclService. As a side note, keep with the fucking naming convention. If you're going to call all the db accessors with Jdbc, then why name the jdbc lookup class BascLookupStrategy? And oh yeah, what's the point of the LookupStrategy pattern considering that you already have a lookup strategy pattern called MutableAclService, which has a Jdbc Accessor called JdbcMutableAclService?

So, even if I extend the ObjectIdentity and add support for property management, the implementation will go to hell, if someone decides to use any of the persistence classes. Oh, almost forgot, for all the bloody abstraction and interfaces, the JdbcLookupStrategy accepts an ObjectIdentity, yet, performs a direct instantiation for ObjectIdentityImpl, with a Long as a serializable id. So, there goes the ability to extend the class, or define anything but a long as an identifier. So, what's the point of creating the ObjectIdentity interface? And, what's the point of making the identifier serializable?

Ah, there is support for an Acl tree via parent/child Acl. I could create a parent Acl to represent the object, and then subsequent children for each of the properties, ah, but the damn ObjectIdentity cast to a long kills that as well.

What would be quite nice is to add property level support directly to the Access Control Entry. Of course, there is an interface, and an implementation, and supporting classes that require the implementation, making another useless interface. What's needed here is a factory pattern.

I am sorry I am angry. I've been reading Buddhist books lately, and they teach you to channel your anger, understand it's source, manage your emotions, so as to balance the negative and positive of Karma. The problem is that all this is going to force me to break from the Acl implementation in Spring, which would mean yet another Acl implementation with a subset feature set. Spring, for all it's problems, seems to provide a large feature set, and if at all possible, I prefer to enhance rather than replace.

Ok, back to Spring Security Acl bashing. The Acl interface and the AclImpl class are capable of encompassing the entire Sid structure. So, if I have 10k users, than, my poor little Acl class will start to look like an ACL cache rather than a simple pojo it was meant to be. What the ACL object should be is a representation of an object, which has properties, and is an instance of security for a single Sid. I highly disagree that a Single Acl needs to start supporting multiple Sids. Granted your approach is more flexible, but flexible to a point that there will be a single ACL class in the system, with a large array of all permissions. Acl is not a cache, it's a simple wrapper around what a single user/principle/granted authority has access to for the given object. The ACL Entry is actually supposed to be a wrapper around a property and a permission mask. That's the whole point of having a permission mask. A mask is an int, which means that you have a single integer (all those bits) that represent all the possible access control rights for a single property of a single object. The beauty of adding property support is that you're no longer limited to a 31 possible permissions, but rather unlimited, with a limit of 31 per property of an object. This means that you can conceivably have different rights per object attribute. And we all know that some objects have a lot more than 32 attributes. So, if you just wrapped the Permission mask in an ACL Entry class, then, what was the point of an ACL Entry class. You could simple collapse the whole structure into the ACL class and be done with it.

Deep breaths, I was reading another blog, which was talking about another blog that mentioned that "Every time you use Acegi... A fairy dies." My daughter love's fairy's.

Drools + Spring Security + Annotations + AOP= ?

2008-07-19T09:53:00.000-07:00

I am starting a new open source project:

http://code.google.com/p/dynamic-rule-security/

No code has been released yet, but I am hoping to have an alpha version out soon. The project integrates Drools Rule Engine with Spring Security to provide dynamic, rule based, field level ACL security to a system.

Once complete, the system administrator will be able to create business rules to restrict fields, objects, pages, content, whatever based on dynamic rules. But, that's not all. The current crop of security requires the security logic to be embedded with the code and is quite brittle and complex when security rules become very granular. For example, imagine having to implement a requirement that says when a trade belongs to account "abc" hide the trade from anyone not in group "abc-allowed". No problem, you say. You create the security group "abc-allowed". Now you have some choices regarding implementation, you can integrate the rule at the data retrieval layer, at the presentation tier, or in the middle. Either way, somewhere in your system, you'll have a chunk of code like this: if ( trade.account == "abc" && !isUserInRole("abc-allowed") ) then hide.

That was easy. Probably only took 10 minutes to write, 10 minutes to test, and a few days to get it deployed to production. No problem.

A few days go by, and the user comes back and says, I need to expand that security. It seems that group efg can actually see abc account trades but only when the trading amount is less than $50m. Ok, you say. A bit messy, but do-able. So, you create security group "efg-allowed", and change your prior rule to say:
if ( trade.account == "abc" && (!isUserInRole("abc-allowed") && ( trade.amount > 50 && !(isUserInRole("efg-allowed") ) then hide.

Probably only took 10 minutes to code, and another 10 minutes to test, but damn there is QA, UAT, production release. A few days later, you finally release the new feature.
Aren't you glad that's over. A few more days go by, and the user says, wait, he forgot that the efg group can't change the trader name on the trade, and can't see the counterparty, but should be able to see and change everything else. Oh, one more thing, they can change the trader name if the trader is "Jack", because trader Jack's accounts are actually managed by the efg group even if the account belongs to the "abc" group.

Crap you say, that's going to be a bit of work. You may need to change the presentation tier, to hide the fields in some cases, but not others. And boy, how much does it suck to hard code the trader's name somewhere.

Anyways, you get the point. Security Rules may get very complex and very specific to the data they interact with and the context of the request. This means that the rule needs to be aware of the data, and who is requesting it. The rule is then capable of setting the security ACL. The presentation tier then only needs to worry about following the ACL rather than actually dealing with the security rules themselves. Not only that, but security rules will be in a single place rather than being sprinkled throughout the system. You can also change them on the fly allowing you to react very quickly to additional security requests.

How to retrieve the fields used in a Drools Rule (DRL)

2008-07-19T09:36:00.001-07:00

Sometimes it maybe useful to know what fields the rule-set relies on. For example, let's imagine you have a freaky dynamic system that's able to populate beans with only the data needed. The problem then becomes how do you know what data is needed by your vast set of dynamic rules.

One way to do this is to assume that you're dealing with standard pojo's. This means that each variable is private and has an associated getVar and setVar method. Drools currently supports their own language, DRL, java (backed by Janino compiler), and MVEL. I will present how to retrieve the fields form DRL and Java. I am sure the same principles can be applied to MVEL.

First, your pojo:


package com.orangemile.ruleengine;

public class Trade {      
       private String traderName;
       private double amount;
       private String currency;
       public String getTraderName() {
           return traderName;
       }
       public void setTraderName(String traderName) {
           this.traderName = traderName;
       }
       public double getAmount() {
           return amount;
       }
       public void setAmount(double amount) {
           this.amount = amount;
       }
       public String getCurrency() {
           return currency;
       }
       public void setCurrency(String currency) {
           this.currency = currency;
       }
   }

Now the magic:


package com.orangemile.ruleengine;

import java.io.StringReader;
import java.util.ArrayList;
import java.util.Collection;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;

import org.codehaus.janino.Java;
import org.codehaus.janino.Parser;
import org.codehaus.janino.Scanner;
import org.codehaus.janino.Java.MethodInvocation;
import org.codehaus.janino.util.Traverser;
import org.drools.compiler.DrlParser;
import org.drools.lang.DrlDumper;
import org.drools.lang.descr.EvalDescr;
import org.drools.lang.descr.FieldConstraintDescr;
import org.drools.lang.descr.ImportDescr;
import org.drools.lang.descr.PackageDescr;
import org.drools.lang.descr.PatternDescr;
import org.drools.lang.descr.RuleDescr;

/**
* @author OrangeMile, Inc
*/
public class DRLFieldExtractor extends DrlDumper {

   private PackageDescr packageDescr;
   private Map variableNameToEntryMap = new HashMap();
   private List entries = new ArrayList();
   private Entry currentEntry;
         
   public Collection getEntries() {
       return entries;
   }
 
   /**
    * Main Entry point - to retrieve the fields call getEntries()
    */
   public String dump( String str ) {
       try {
        DrlParser parser = new DrlParser();
        PackageDescr packageDescr = parser.parse(new StringReader(str));
        String ruleText = dump( packageDescr );      
        return ruleText;
       } catch ( Exception e ){
           throw new RuntimeException(e);
       }
   }
 
   /**
    * Main Entry point - to retrieve the fields call getEntries()
    */
   @Override
   public synchronized String dump(PackageDescr packageDescr) {
       this.packageDescr = packageDescr;
       String ruleText = super.dump(packageDescr);
       List rules = (List) packageDescr.getRules();
       for ( RuleDescr rule : rules ) {
           evalJava( (String) rule.getConsequence() );
       }
       return ruleText;
   }

   /**
    * Parses the eval statement
    */
   @Override
   public void visitEvalDescr(EvalDescr descr) {
       evalJava( (String) descr.getContent() );
       super.visitEvalDescr(descr);
   }

   /**
    * Retrieves the variable bindings from DRL
    */
   @Override
   public void visitPatternDescr(PatternDescr descr) {
       currentEntry = new Entry();  
       currentEntry.classType = descr.getObjectType();
       currentEntry.variableName = descr.getIdentifier();
       variableNameToEntryMap.put(currentEntry.variableName, currentEntry);
       entries.add( currentEntry );
       super.visitPatternDescr(descr);
   }

   /**
    * Retrieves the field names used in the DRL
    */
   @Override
   public void visitFieldConstraintDescr(FieldConstraintDescr descr) {
       currentEntry.fields.add( descr.getFieldName() );
       super.visitFieldConstraintDescr(descr);
   }
 
   /**
    * Parses out the fields from a chunk of java code
    * @param code
    */
   @SuppressWarnings("unchecked")
   private void evalJava(String code) {
       try {
           StringBuilder java = new StringBuilder();
           List imports = (List) packageDescr.getImports();
           for ( ImportDescr i : imports ) {
               java.append(" import ").append( i.getTarget() ).append("; ");
           }
           java.append("public class Test { ");
           java.append(" static {");
           for ( Entry e : variableNameToEntryMap.values() ) {
               java.append( e.classType ).append(" ").append( e.variableName ).append(" = null; ");
           }
           java.append(code).append("; } ");
           java.append("}");
           Traverser traverser = new Traverser() {
               @Override
               public void traverseMethodInvocation(MethodInvocation mi) {
                   if ((mi.arguments != null && mi.arguments.length > 0)
                           || !mi.methodName.startsWith("get") || mi.optionalTarget == null) {
                       super.traverseMethodInvocation(mi);
                   }
                   Entry entry = variableNameToEntryMap.get(mi.optionalTarget.toString());
                   if ( entry != null ) {
                       String fieldName = mi.methodName.substring("get".length());
                       fieldName = Character.toLowerCase(fieldName.charAt(0)) + fieldName.substring(1);
                       entry.fields.add( fieldName );
                   }
                   super.traverseMethodInvocation(mi);  
               }
           };
           System.out.println( java );
           StringReader reader = new StringReader(java.toString());
           Parser parser = new Parser(new Scanner(null, reader));
           Java.CompilationUnit cu = parser.parseCompilationUnit();
           traverser.traverseCompilationUnit(cu);
       } catch (Exception e) {
           throw new RuntimeException(e);
       }
   }

 
   /**
    * Utility storage class
    */
   public static class Entry {
       public String variableName;
       public String classType;
       public HashSet fields = new HashSet();
     
       public String toString() {
           return "[variableName: " + variableName + ", classType: " + classType + ", fields: " + fields + "]";
       }
   }
}

And now, how to run it:



   public static void main( String args [] ) {
       String rule = "package com.orangemile.ruleengine;" +
       " import com.orangemile.ruleengine.*; " +
       " rule \"test rule\"     " +
       " when                     " +
       "    trade : Trade( amount > 5 )    " +
       " then                             " +
       "     System.out.println( trade.getTraderName() );         " +
       " end ";

       DRLFieldExtractor e = new DRLFieldExtractor();
       e.dump(rule);
       System.out.println( e.getEntries() );
   }

The basic principle is that the code relies on the AST tree that's produced by DRL and Janino. In the case of Janino walk, the code only looks for method calls that have a target, start with a "get", and take no variables. In the cast of DRL, the API is helpful enough in providing callbacks when a variable declaration and field is hit, making the code trivial.

That's it. Hope this helps someone.

Drools - Fact Template Example

2008-07-16T19:30:00.000-07:00

Jboss Rule Engine ( Drools ) primarily works based on an object model. In order to define a rule and have it compile, the data referenced in the rule needs to exist somewhere in the classpath. This is easy enough to accomplish by using any of the dynamic libraries such as asm, cglib, or antlr. Once your class is defined, you can either inject your own implementation of a Classloader or change the permission on the system classloader and call defineClass manually.

But, there is another way, which is a bit over simplistic, but maybe useful for some of you out there. Drools has introduced support for fact templates, which is a concept introduced by Clips. A fact template is a basically a definition of a flat class:


template "Trade"
    String tradeId
  Double amount
  String cusip
    String traderName
end

This template can then be naturally used in the when part of a rule:

rule "test rule" when $trade : Trade(tradeId == 5 ) then System.out.println( trade.getFieldValue("traderName") ); end

But, there is a cleaner way to do all of this using the MVEL dialect introduced in Drools 4.0.
You can code your own Fact implementation that's backed by a Map.


package app.java.com.orangemile.ruleengine;

import java.util.HashMap;
import java.util.concurrent.atomic.AtomicLong;

import org.drools.facttemplates.Fact;
import org.drools.facttemplates.FactTemplate;
import org.drools.facttemplates.FieldTemplate;

/**
* @author OrangeMile, Inc
*/
public class HashMapFactImpl extends HashMap implements Fact {

private static AtomicLong staticFactId = new AtomicLong();

private FactTemplate factTemplate;
private long factId;

public HashMapFactImpl( FactTemplate factTemplate ) {
      factId = staticFactId.addAndGet(1);
      this.factTemplate = factTemplate;
}

@Override
public long getFactId() {
       return factId;
}

@Override
public FactTemplate getFactTemplate() {
           return factTemplate;
}

@Override
public Object getFieldValue(int index) {
           FieldTemplate field = factTemplate.getFieldTemplate(index);
           return get(field.getName());
}

@Override
public Object getFieldValue(String key) {
           return get(key);
}

@Override
public void setFieldValue(int index, Object value) {
           FieldTemplate field = factTemplate.getFieldTemplate(index);
           put( field.getName(), value );
}

@Override
public void setFieldValue(String key, Object value) {
           put(key, value);
}
}

To use this class, you would then do this:

String rule = "package com.orangemile.ruleengine.test;" + " template \"Trade\" " + " String traderName " + " int id " + " end " + " rule \"test rule\" " + " dialect \"mvel\" " + " when " + " $trade : Trade( id == 5 ) " + " then " + " System.out.println( $trade.traderName ); " + " end "; MVELDialectConfiguration dialect = new MVELDialectConfiguration(); PackageBuilderConfiguration conf = dialect.getPackageBuilderConfiguration(); PackageBuilder builder = new PackageBuilder(conf); builder.addPackageFromDrl(new StringReader(rule)); org.drools.rule.Package pkg = builder.getPackage(); RuleBase ruleBase = RuleBaseFactory.newRuleBase(); ruleBase.addPackage(pkg); HashMapFactImpl trade = new HashMapFactImpl(pkg.getFactTemplate("Trade")); trade.put("traderName", "Bob Dole"); trade.put("id", 5); StatefulSession session = ruleBase.newStatefulSession(); session.insert(trade); session.fireAllRules(); session.dispose();

Notice, that in the then clause, to output the traderName, the syntax is:

$trade.traderName

rather then the cumbersome:

$trade.getFieldValue("traderName")

What makes this possible is that the Fact is backed by a Map, and the dialect is MVEL, which supports this type of operation, when the map keys are strings.

The interesting thing about using the fact template, is that it makes it easy to perform lazy variable resolution. You may extend the above HashMapFactImpl to add Field Resolvers that may contain specific logic to retrieve field values. To do this with an object tree, especially dynamic objects, would require either intercepting the call to retrieve the field via AOP and injecting the appropriate lazy value, or setting the value to a dynamic proxy, which then performs the lazy variable retrieval once triggered. In either case, this simple Fact Template solution maybe all that you need.

Corticon Rule Engine Review

2008-06-12T18:42:00.000-07:00

I've recently had an opportunity to give Corticon a test run. Corticon consists of desktop rule studio and a server component that runs the rules. The server component can be embedded inside a Java application, or run as a standalone server. The studio is used to create a vocabulary and the rule sets.

The approach Corticon uses is quite novel in comparison to the other rule engine vendors. They don't use the Rete Algorithm, instead they rely on compile time rule linking. This means that the rules are deployed to the server compiled, and the rule firing order is already calculated. The concept is undoubtedly more palatable to some organizations that find traditional rule engines a little unyielding.

The other novel idea is that the rules are compiled against a vocabulary (an ontology, if you will) rather than an object model. This means that you can submit dynamic datasets to the rule engine rather than relying on strict structures as is the case with Jboss Rules. It also seems that Corticon has licensed the Oracle XML SDK, which allows them to query the database, retrieve the result in XML, pipe it to the rule engine, and produce a result, all without any custom code. The actual rule writing and vocabulary management occurs in the desktop rule studio. The studio gives a user a decision tree type of interface but with some enhancements. The rule writing is done in a 4th generation language, meaning that the rule author uses the vocabulary, boolean algebra, truth tables, discrete math type of programming, "table driven algorithm programming". It's quite nifty but comes off a little limiting to a hard core developer.

And now for the negative:
In order to extend the language, you need to write java code, and then add it to the classpath of the desktop rule studio. This means that if you distributed the desktop rule studio to your business analyst, now you have to redistribute your jar file. You will need to redistribute a new jar file every time you extend the library with a new function. This is almost impossible to do in large organizations with dumb terminals, software packaging, etc...

The actual desktop rule studio is cluncky and is missing some basic keyboard shortcuts, like the "delete" key. There is also no security of any type. The rule author is able to change the vocabulary and any rules. The actual 4gl language at times seems more confusing than it's 3GL counterpart.

For the developer, Corticon is a disaster. The API library consists of a handful of classes. One to create a server, and run the rules. Another to deploy the rules to the server, and yet another to do some admin RMI calls to the desktop rule studio. Mind you, RMI calls for an enterprise level application is a little laughable. The functionality of the RMI calls are also a little silly. You can start the studio, shut it down, and open a vocabulary. In comparison, ILOG has an immense API library. I would wager that you can pretty much control any aspect of ILOG. And of course, JBoss Rules is open source, which makes them quite friendly to developers.

And yet more negativity:
There is no way to control the rule firings. There is also no way to disable certain rules from firing, or manage the rules in a central location. The mere fact that the rules are compiled in the desktop rule studio by the rule author and written as a file in binary format makes any form of enterprise rule management a joke. A nice web 2.0 gui for rule management like with Drools or ILog would also be nice. Why is it that I need a desktop app to manage my rules?

In summary, I think Corticon is quite a nifty concept, but is not a mature enterprise framework. It maybe useful in some limited fashion as glue logic, but it doesn't belong as an enterprise class rule engine. At least, not yet.

An addendum: Tibco iProcess has licensed the Corticon Rule engine for it's decision server. This does not change my opinion of Corticon, but reduces my opinion of Tibco. As a side topic, I think Tibco Business Work is an impressive tool, but the entire iProcess stack is quite bad and convoluted. I think Tibco just wanted to put something out quickly, so, they pieced together some varying half finished technologies.

J2ee, Jee, EJB 2 and 3, Spring, Weblogic,

2008-05-11T17:38:00.000-07:00

Bla, bla, bla, bla. First, I'd like to say that I hate EJB 2, and starting to seriously dislike EJB 3. J2ee, and the stupid rebranding to JEE. There are many things that are wrong with EJB 2. One of them is the security model, another is the QL language, another is their retarded persistence approach. The only gain, and even that's a stretch would be the transaction management. But, even there, the EJB authors completely f'd up and created a convoluted model.

EJB's, originally, were supposed to save the day. Usher in a day where corporate code monkeys can get a bit dumber and focus on the "business logic" and forget all that complicated plumbing like transaction management, and threading. Threading, ha, said the EJB spec writes. No threads needed. The container will take care of that for you. Don't worry your pretty little head, little developer, just go play in your little sandbox with your pretty little business logic. Well, many developers did just that. And in time, they've realized that not only did EJB's complicate everything but you needed EJB experts just to support the frankenstein monstrosity that get produced.

Somewhere around there Spring has dawned, and flowers bloomed. Now Rod has the right idea, but the implementation lacked. Spring has proved to be another monster. The issue really is in the complexity. In order to get any value from spring, or even understand how spring works requires deep internal knowledge, something that Spring and EJB fail to mention. Spring has an insane dependency on configuration. In fact to such a degree as to make the code unreadable. A developer now has to funnel through massive amounts of XML config, plus a massive amounts of useless interfaces only to find some impl classes that's AOP'd into the damn dao. Now, don't get me wrong all you spring wankers, I am more than aware of all the benefits of IOC and AOP and unit testing. I know that's what's running through your head. Heck, he doesn't understand proper test driven development, bla, bla, bla. How dare he criticize that which is holy and has saved the day. Well, I don't like it. It replaces something really bad. I agree with that. I believe in the simplicity of design and the readability of the code. I agree in the principle value of AOP and IOC, but I wonder if there is a better and cleaner way to achieve some of the same things Spring set out to do.

EJB 3.0 persistence part is yet another over-arching spec-iness of the writes. It's basically useless for anything slightly larger than a pet-store website. The goals are definately admirable, but they must have known how far of the target they would actually hit. They are attempting to map pojo's into a relational table structure. What they don't tell you is that it's impossible to accomplish without seriously hampering the design of your relational model. Now, if you had an object database, perhaps, I wouldn't be saying that. Perhaps, a more valid attempt for this will come from the space of semantic networks, and companies like Business Objects with the concepts of a universe design, which provides a layer above the raw data.

To return to EJB and Spring bashing, I disagree with the basic notion of the goals. Each attempts to reduce how much a developer needs to know. Of course the reason for that, is to dummy down the developer, replace him/her with a body from a cheap foreign country, and keep-on churning code. A different model would be to replace the army of code monkeys with a few diligent developers, but move the responsibility of business development to the BUSINESS. At the end of the day, no one knows the end goal except the business. And no developer will ever be better than the business user at solving the business problem. Now, a lot of developers try, and they are usually bad developers. So, they are bad at business, they are bad at programming. And yes, they need yet another framework to keep them from hurting themselves. I think we should focus our attention at reducing our field to a few competent experts that deserve the title of the developer who focus on the technical development that enables the business users to do the thing that most of the developers do today: business coding.

A better compression algorithm

2008-05-11T12:57:00.000-07:00

I am presented with an interesting problem. Usually, my employer burns money indiscriminately, but lately, with the market in tailspin, all costs are being evaluated. To avoid being one of those costs, I need to find a way to save money for the company. One of those ways is file storage space for document management systems. Unlike your basic $50, 100gb drives that you buy at Circuit City for you dell, corporate disk storage is highly expensive with EMC-SRDF storage running for 1TB at 1m.

Audit and regulatory rules requires that basically all files are kept. A large number of those files are data feeds from external systems. The files are structured and are in a readable format such as fixed length, delimited, or XML. My idea, which is not that unique, is to apply a heuristic compression algorithm to the data files. I am going to leverage the work done by the FIXML Protocol committee on the FAST specification, which defines a number of optimal heuristic encoding schemes. FAST defines a compression algorithm for market-data, but the same principles apply to file storage.

http://www.fixprotocol.org/fast

The concept is quite interesting. The compression algorithm basically attempts to find data patterns in the file, and encode them away. Let's say you have column that's an incrementing number: 1, 2, 3, ... n, n+1. The encoder will identify that this is an incrementing column, and encode it as algo: { previous + 1, starting with 0 }. We've just encoded away an entire column and took no space to do it. Let's try another example: abcdefg, abcdefe, abcdefn, abcdef5, etc... In this case, the first "abcdef" is the same in all the columns, and only the last character changes. We can encode this as a constant, and only send the last character: g, e, n, 5, etc...
There are a lot more sophisticated algorithms defined in the FAST protocol, but you get the idea.

The data in the file starts to mean something. The encoder actually attempts to represent the patterns present in the file. The patterns have a potential to save a lot more space then a traditional compression algorithm based on Huffman Encoding. How much space: how about average case of > 80%, compared with best case of 40% for ZIP. And don't forget, the result can still be zipped.

The program will read a file, scan through all the data points, figure out the optimal encoding algorithm, and then actually do the compression. The encoding algorithm will be needed to decompress the file. The first field in the file will carry the bytes needed for the encoding algorithm, followed by the encoding algorithm, and finally the data. This allows us to store the encoding scheme with the file.

One enhancement to FAST would be to allow the pre-processor to re-arrange the file. Data optimization is mostly based on previous records, so the more similar subsequent entries are, the higher the compression rate. Another enhancement maybe to bit map away typed fields. If a million entry file has 100 unique types, it might be more optimal to encode the bitmap separately, and then encode away the type id. Another extension maybe to see whether a corollary between fields rather then between subsequent records exists.

Another extension to this architecture is to write the files in a way as to improve lookup cost: index the files, and an intuitive UI, for the user to jump to the needed entry.

I have high hopes for this algorithm. If it can really encode away 90% of the file, then the space savings just might save my job. Well, at least until the next round of cost cutting.

Business Intelligence

2008-02-13T13:09:00.000-08:00

Imagine a world where the people that own the data, actually have access to it. Sounds obvious, but think about it. Unless the business user is also the developer, this is never the case.

Your system users own the data that's in your system. Unfortunately for them, your system is also what's keeping them from their data. Every little thing they need has to go through you. The irony is that all you care about is serving the business user.

So, what are the parts of the problem. We have data that has some logical structure and some semantic definition to that structure. The business user implicitly understands the semantic definition, and wants to exploit it to its full potential. A semantic definition will define a relationship such as a cusip is an attribute of a stock, quantity is an attribute of a trade. Stocks are traded on exchanges. etc... Now, the user wants to aggregate quantity by cusip across all trades within an exchange. Fine you say. You go off, come back a few days later with a beautiful brand new report. Great the user says, now I want to see the average quantity traded. Well, of you go again, etc....

So, the data has some semantic definition. The same semantic definition exists in the users head. The user exploits the structure. This is analytics. The user should be able to manipulate the data with the only constraint being the semantic definition. At the moment, this space is filled by cube technology on the data warehouse side, and Business Objects on the relational side. The only real difference between BO and Cube technology is the size of the dataset. Cubes are pre-aggregated while BO is real-time SQL. It should be interesting to link cube technology to BO for drillthrough. So, once you have the data you point pump it into a rich visualization component. But, be careful not to link the visualization with the data. Each piece of technology is independent, but has a well defined interface in how to leverage it. The visualization component can receive data. So, now we have our analytics and visualization. The next part is to take both pieces and generate a static report that can be presented to senior management. This report can be saved, or automatically updated with new data; archived daily, quarterly, etc...

So, not too bad. But, I also want to understand how good my data is. I want to understand the integrity of the data at the lowest level. I need to know the story of every data point. This is where rule engines come in to play. The user will define the rules that will validate integrity. The trick is to have the rule engine tell you how good or bad your data is at any aggregated level. The data isn't discarded but just measured.

So far, the user has the data, can analyze it, visualize it, knows its quality and can report it. The next step is to manipulate it. A lot of times, analytics takes a flavor of what-if analysis. The user should be able to locally modify any data point, analyze the impact, visualize, report, etc...

Well, are we done. Have we satisfied everything the user wants. No. No. No. Now that you have some analysis you need to act on it. The data that you derived has some attributes which via rules can be applied to certain actions. One action can be to feed it into another system for further enrichment.

Are we done now? Damn it no. Once you have all this, you can take the data to the next step. You can mine the data for patterns. The patterns can then feed back to calibrate the data integrity rules.

As the user analyzes the data, the system watches the user. The more analysis done, the more the system can understand the user's intent. At this point, the system can start to infer what the user is trying to do. Now, we are starting to take the flavor of having the system solve equations and then acting on the outcome.

Think about it, but think about it in a context of a massively large data repository, and a Wall Street type firm.

In the interest of buying a solution, I present a vendor list:
Microsoft Analysis Services
Business Objects Web Intelligence
Panorama
Microsoft Sharepoint
Microsoft Excel 2007 (has a lot of cube technology)
Business Objects Dashboard + Reporting + etc...
ILog Jrules

Philosophy of Architecture

2007-10-02T19:06:00.000-07:00

I have recently come to face with two distinct philosophies of architecture.

The first philosophy holds the business in the highest esteem to the determent of the system. All projects are done as strategic. This means that the management pushes on the development team to deliver as soon as possible and with a sub-optimal solution. This is what's commonly termed as "getting it done". With this philosophy, requirements tend to be spotty or none existent. In most cases, the requirement document is created after development has already completed. Development is incremental and the system follows an incremental evolution. The business receives the minimum of what was asked, but with the impression of quick delivery. Unfortunately, an incremental evolution causes the development time to continuously increase. This increase is caused, because code is only added and rarely removed. Removing code requires analysis and re-factoring: time which is not factored into the project schedule. Adding code in this way will balloon the system and make adding any future enhancements/changes incrementally more difficult.

The second philosophy is more methodical in its approach. In this case, development goes through some established cycles such as understanding what needs to be built, designing, reviewing, and finally building. This approach has a longer upfront cost before actual development begins but causes the system to move in revolutionary jumps rather then in continuous, increasing steps. With revolution jumps, the system tends to get more compact as code gets re-factored and multiple functionalities folded into a single framework.

Most shops follow the first philosophy. This philosophy is more natural to organic growth. When the user tells you to do something, and they are paying your salary, you do it. With the second philosophy, you would need to have the guts to tell the user, no, wait, let me first understand what you're trying to accomplish, and then we'll review, and then I'll build. This is very difficult. For example, most, if not all, Wall street firms follow the "getting is done" model. The "beauty" of the system is secondary, delivering the project is primary above all rest.

My argument is that beyond creating a simple report, no project should follow the "getting it done" philosophy. Every project needs to have a more methodical approach. Building the first thing that comes to your mind is dangerous and stupid when working with an enterprise system. All projects need proper analysis: what already exists, what should change, what the user wants, what else might they want. Then, draw up the architecture, review it, and only then, build it.

Data Warehousing

2007-09-14T18:45:00.000-07:00

I have recently been immersed in the world of BI, OLAP, XMLA, MDX, DW, UDM, Cube, ROLAP, MOLAP, HOLAP, star schema, snowflake, dimensions and facts.

A data warehouse is a special form of repository that sacrifices storage space for ease of retrieval. The data is stored in a special normalized form that literally looks like a star schema. If you change one attribute, an entire row is duplicated. The data is normalized in a way to ease retrieval and reduce table joins. The data warehouse is nothing but a giant relational database whose schema design makes using plain SQL downright ugly. On top of this repository, lies one or more cubes that represent an aggregated view of the massive amounts of data. There are multiple forms of the cube: multi-dimensional online analytical processing(MOLAP), relational online analytical processing (ROLAP), and hybrid online analytical processing (HOLAP). A ROLAP cube is nothing but a special engine that converts the user requests into SQL and passes it to the relational database, a MOLAP is a pre-aggregated cube that allows the user fast retrieval without consistently requiring the underling data store, and a HOLAP is a hybrid of those two approaches. The reason for the cube technology is that it allows the user to slice and dice massive amounts of data online without any developers involvement. On top of the cube technology, there are a set of user front ends either web based or desktop. One such company is Panorama. Each GUI tool communicated with the cube in a standard language called MDX. A multi-dimensional expression language. An XML version of this language is the XMLA protocol, which was originally invented by the cube GUI company Panorama. Microsoft bought out their original tool, and further developed it into what is today called Microsoft Analysis Services 2005, which is a leading cube framework.

So to summarize:
UDB(Relational Database)
Microsoft Analysis Services 2005 (Cube)
Panorama (GUI)

Now the price, well, for a full blow BI (Business Intelligence) solution, you're easily looking into millions just on storage alone not to mention the license costs of the products. There are free solutions at least on the GUI side: one good one is jpivot.

A Data warehouse is a very powerful concept. It allows you to literally analyze your data in real-time. The business users use a friendly GUI to slice and dice their data, aggregate the numbers in different ways, generate reports, etc... The concept allows you to see ALL your data in any way the user imagines or at least in the number of dimensions defined on your cube. A dimension, by the way, is an attribute that it makes sense to slice by. For example, dates or type columns are good dimensions. A fact on the other hand is the business item that you're aggregating. For example, a trade would be considered a fact.

Once you have a data warehouse the next logical extension is KPI (Key Performance Indicators). Imagine looking at a dashboard, and have pretty colors with green, yellow, and red telling you how much money you're making/losing at that point. KPI are special rules that are applied to the data at the lowest level. When you aggregate up, the colors change depending on how you're slicing the data. This allows you to start at the very top of which region isn't doing so well, and then drill down to the very desk that's loosing money.

A further extension of data warehousing is data mining. This is a off-shot of AI and covers areas such as cluster detection, association rules, etc... There will be further blogs covering this in more detail.

So, if you have a huge budget, I recommend you give this a try. Your company will thank you for it(later). And if you don't have a huge budget, understand whether your problem fits in the BI world, and ask for a huge budget. I've seen too many companies take a cheap route and end up with half baked solutions that have no future.

Rule Engines

2007-08-26T20:00:00.000-07:00

Recently, there has been a proliferation of rule engines. A rule engine is by product of AI research. The basic premise is that a user is able to create a bunch of atomic units of knowledge. When the rule engine is presented with a state of the world, the rules all fire. After all the firings have settled down, the new state of the world is an outcome. A lot of problems are easier to implement using rule engines than the more conventional programming. For example, system that relies on heavy usage of knowledge with deep trees - imagine many layers deep of if/elif.

There are a couple of major contenders. For the corporate world, there is ILOG and FairIsaac. For the open source, there is Jboss Rules and Jess. Jess being the original java rule engine, and the closest to the original NASA Clips system. Clips being the system that created the rule engines. Personally, I am most familiar with Jboss Rules, ILOG and to a much lesser degree with Jess. This should not be taken as a diss on FairIsaac or any other rule engine.

Each rule engine, at its core, is based on the RETE algorithm. There are a lot of variations and enhancements, but each rule engine implements the core algorithm. The algorithm is used to find rules that need to be executed for a given world state. Imagine thousands of rules, and a good search algorithm becomes critical to a useful rule engine. The RETE algorithm acts as the control flow in a regular language.

The major blocking point to a wide adoption of rule engines is their dynamic nature and unpredictability. If you define a thousand rules, it becomes difficult to know how the rules will interact in every situation. This means testing and scenario generation is critical. This also means a much more mature infrastructure and process than most organizations have. The advantages are huge. You can explain to your user exactly how a given outcome was reached. Display the rules, modify the rules, add rules, all dynamically. You can even simplify the rule model such that your users can create their own rules.

The next blocking point is the rule language itself. The language has many requirements. For example, some people want the language to have a natural language feel. Others, want a clean interact with the existing java system, while others seek some middle ground with a scripting language. ILOG does this very well, with a natural language translation tool. Jboss rules has a more rudimentary natural language translation (DRL - DSL) but supports a wider language group.

I find Jboss Rules to be easier to get started with, but a large and mature organization should probably take a look at a vendor product for the scenario generation, and rule management infrastructure, something Jboss doesn't quite have yet. The vendors also have much more mature rule editing GUI's.

Supply of Money

2007-07-07T08:12:00.001-07:00

I know this should be a technology oriented blog, but I am starting to be afraid, because I don't understand what is happening.

Money is intrinsically worthless:

"Paper money eventually returns to its intrinsic value - zero." ~ Voltaire - 1729

Our economy is one of exponentially increasing debt. All money (dollar) is loaned at interest from the Fed. The Fed creates money by printing it as basically zero cost. This means that to pay interest you need to borrow more money(get a loan), by so creating more money. Notice the exponential function in all of this. The US economy basically no longer produces anything, and imports everything necessary for basic survival. To import requires purchasing, to purchase requires money, money that needs to be borrowed. Borrowing requires paying interest. How does the government borrow, it borrows from the Fed, which prints more money.

The interesting thing is the bond market which acts as a money sponge. A US treasury bond pays a certain yield. Japan has historically bought billions and billions of US treasuries to the tune of 16% of all US treasury bonds. This is interesting, Japan buys a bond of $100 paying 4% yield. This means that Japan hands over 100 dollars to the US government in exchange for 4% yield. In an essence, $100 dollars disappears from circulation and was replaced by a continuous stream of $4 dollars. Now, $4 dollars has to come from somewhere, it's borrowed from the Fed. This is an ever increasing cycle, growing exponentially fast. What ever money exists in circulation was borrowed at interest. I think all this means is that money can never be destroyed. It can only ever exponentially increase.

What happens on the way back. What happens if the money was to be re-payed to the Fed. The dollar will need to traverse the entire route back. I don't understand how that's possible, but if it was to happen, money would return to its intrinsic value of 0.

A little confusing. Right now, Tokyo's interest rate is extremely low. Tokyo is also trading at about 125 yet to a dollar. Tokyo's rate is around 1 percent, while US and the rest of the western world is at 4 to 5 percent. This means you can get cheap money from Tokyo, convert it into dollars, buy US bonds, and earn a hefty 4.5 percent without doing anything. But you can also leverage your position, by taking on more risk. In this case, you don't buy more yen, but plan to buy later, but also simultaneously use what you don't own. In an essence, you've just created even more supply of money. One day, you will need to reverse you position buy actually buying the yen you promised to buy. This will cause the supply of yen to drop, the demand to sky rocket, and the price to act accordingly. The US dollar is going to continue to drop or in other words go up. The currency must continue to weaken, as it will take more dollars to service the exponentially increasing debt.

China and India will undoubtedly delay the inevitable, but the world economy must and will collapse. An exponential function cannot last indefinitely. This is the conclusion I am drawing, but I must admit I don't understand all the factors. All I know is that I am becoming increasingly uneasy.

Black Swan

2007-05-27T17:46:00.000-07:00

I am becoming obsessed with randomness and probability. What follows is based very heavily on Nassim Nicholas Taleb research. Imagine a turkey on a farm. Everyday the turkey has ever known, the farmer comes every morning and feeds it. From the turkey's point of view, the farmer is a friend, a trusted being. Then, one morning, the farmer kills the turkey. A black swan has occurred from the point of view of the turkey. A completely unexpected event.

Take our stock market, heck take the entire global market, companies, and global economies have created multiple levels to guard against risk. Options trading, derivatives, options on derivatives, credit default swaps, and so on, and on, and on. Each product is designed to allow some risk, some profit, and some safety. Some products have two components such as derivatives, allowing a company to sell its risk to others. Risk, actually, is an interesting side of the coin. Companies have large staffs of risk professionals, calculating, and guarding the said corporations from risk. Recently, companies started to realize that risk comes in many forms, and a new area was born "operational risk". This is the risk where an employee goes crazy and shoots everyone. So, you would argue that all this guards the said companies from risk. Now, Nassim Taleb, and myself, actually believe that this enhances risk. All this calculating is simply creating an impression of safety. Like the turkey, we go day in and day out believing we are safe, until one day, the farmer kills the turkey.

The basic problem is that we can't understand the future. In fact, we can't understand that we can't understand the future. We keep believing in things, looking for correlations, patterns in randomness. We find them, in fact we tend to create patterns in randomness. Are the markets random? I would argue no. In fact, I would argue that the markets are becoming very much un-random. The markets are starting to be governed by machines following very concrete rules. There are also very few players in the market that have the weight to move markets, and a lot of those players are using machines. All of this is very scary.

Another interesting example is China. An unprecedented amount of common people are investing heavily in the market. And, the market is going up and up and up. But, like everything else in life, it will come down, and boy will it come down hard. And there will be ripples throw the global markets, and global economies. But, this isn't the black swan I am afraid of. I am afraid of something more. I am afraid of something we don't know is going to happen.

Global Development

2007-05-27T17:29:00.000-07:00

It is all the rage these days to do global development. One "system", global implementation. The idea being is economy of scale. Any region can perform development allowing other regions to reap the rewards. There are different ways for a single system to achieve global development.

1. The system is being developed by 1 region. All global requirements are funneled to this region. The actual system maybe run centrally or locally within the regions.

2. Each region has a separate system which, based on an agreed protocol, feed a shared central system.

Ah, but there is another way. You maybe able to have a single system, and yet global, parallel development. You can split the system into areas of concern, and assign different parts to different systems. Unfortunately, at one point or another, the areas will overlap. This brings up an interesting scenario. Single system, many teams split across different timezones answering to different management, have different requirements, different users, schedules, etc... Quite a mess. Now, each region is actually working on a common goal. The system is a specific system, serving a specific goal, but different masters. The trick is to split the system into a common framework and a regional implementation. If the regions are using the same system, and there is a core of the system which is indeed universal, but there is also an aspect of the system which is very much unique to a given region. Understand the problem the system is solving. Then understand the fundamental aspect of the system, the raw materials, if you will. This is the common framework. Each region may modify the framework, but what they are doing is enhancing the breadth of the system. Imagine a graph, links and nodes going every which way. Imagine dark areas of the graph, unexplored. These dark areas represent parts of the system developed by other regions, but not yet used locally. When a given area matures to that functionality, it will be there for it. The unexplored areas of the graph become used, and therefore visible. This seems a very interesting way to create a global enterprise architecture. Model the system as a graph, allow each region to build out the graph, but in such a way as to allow other regions to use only what they need. Then allow the graph to be customized to the regions needs. If done correctly, the system will become a set of loose shared modules, with concrete implementation by each region. The regions decide how the modules are used and how they link. Of course, some linkages is defined. Regions may enhance existing modules, build new ones, or create region specific enhancements to existing.

Equilibrium

2007-04-22T19:30:00.000-07:00

I had a chat with a Rabbi the other day. He told me a story from his life. When he was a young man, he had trouble sleeping. He would sleep at most 4 hours a night. He was worried that he had a sleeping disorder, so he found a top doctor on sleeping disorders. The doctor had him keep track of the number of ours he slept every night for a month. At the end, the doctor identified that the Rabbi slept an average of 4 hours every night. Sometimes, 4:15, other times, 3:50, but on the average 4 hours. What the doctor told the Rabbi was that he was one of the lucky ones. Most people are in the middle, and require 7 hours of sleep. The Rabbi was an extreme exception on the far side of the curve, requiring only 4 hours. The Rabbi was lucky because he has 3 hours more a day than everyone else. This story is interesting in that in this day and age, in this country, the rabbi would be put on sleeping medication. I am pretty sure that a number of hours people sleep fits a bell curve. Most people are in the middle sleeping somewhere between 6 and 8 hours. But the tails of the curve expend in both directions; some, requiring more like 9 or 10, while others requiring less like 4 or 5. Now, the established medical principle of the day and age is to fit everyone into the middle with no tails. I see this in everything. For example, medical community preaches that cholesterol should be below 200. Now, what makes 200 a magic number that applies to the entire population regardless of background. I would imagine that cholesterol, like everything else, follows a bell curve. Most people's normal average is 200, but the tails of the curve, go out in both directions. Some have a high average cholesterol number, and that is considered normal for their bodies, while others, have a low average. It is very troubling that most things are being applied indiscriminately. We, as a society, are loosing the equilibrium in favor of the standard.

2007-02-05T20:59:00.000-08:00

I usually stay away from such posts, but I can't resist. Check out these two sites:

http://www.zoho.com/
http://services.alphaworks.ibm.com/ManyEyes/

A priori

2007-02-03T20:03:00.000-08:00

A priori is a term that describes sequence of events, time. More specifically, development is a sequence of steps, events, that produce a desired result. The question that bothers me is why does it freaking take so long.

A colleague of mine was recently complaining that his users were upset that it takes his team a long time to develop seemingly simple functionality. Why does it take weeks to read some data, apply some business rules, send some messages, and produce a report.

The world of business tools can be thought of as a giant, ever increasing graveyard. The business tools are being continuously and artificially given life to. Like little Frankensteins, they roam the earth, used and abused by both users and developers, growing up, until being killed off and replaced with a younger Frankensteins that are doomed to the same fate.

Excel is the only tool that comes to mind that has escaped this fate. It allows the business user to solve his own problems. Unthinkable to a crack smoking code monkey. The user can load data, build his models, produce reports, export them out. The power is in the users hands. On the other side, the developer attempts to give the user exactly what the user asked for and nothing, and I mean nothing else. In fact, the majority of the time, the developer neither understands the business user nor the business nor the problem being solved.

I think the industry is starting to realize this and is attempting to shift the power back to the business user. For example, specs like BPEL and the hype surrounding web-services are all meant to give more power to the business user and reduce the turn-around time of development. I believe software will become less like software and more like legos. Individual pieces will still need to be built, but the business user is the one that will put the legos together to produce a result. Things like forms, business rules, reports, data loading, data extraction will go away. Instead, time will be spent on producing richer widgets to do more sophisticated things. Honestly, how many developers does it take to build a relatively large system that does a whole lot of variations of the 5 things mentioned above? 1, 2, 5, 7, 10, 40? How big is your team?

2007-01-26T19:47:00.000-08:00

I've been away for a long time. For that, I am sorry. But, now I am back.

What's been on my mind lately is whether its possible to encode a business intention in an intermediary language and then build an interpreter to read this language. One system would encode an intention, the second system would evaluate it. Interesting, no? Perhaps, all this means is that system A sends a message to system B. System B reads the message and based on hard-coded business rules performs the work. But, let's say there are no hard-coded business rules. Let's say the message is the rules and the data. Would that be possible? What would this language look like. It would need to contain meta-data that could be evaluated and mapped to business rules. Let's step back a little. What's the point of this. System B is a specific system that does a specific thing. It should know what to do with the message without needing System A to tell it. A new trade message arrives, your system receives the trade. It knows its a new trade, because it says so on the message. What is the action, book the trade. So, your system dynamically looks up all the supported actions, and passes the data-set to that rule-set. Now, some of you are thinking, great, all this and he describes a bloody factory pattern. But wait, forget messages. It's an event. Something, some how raises an event that says there is a new action with the given payload. Some controller accepts the event and routes it to the appropriate implementation for that event, or perhaps a set of implementations, or even better triggers the work-flow. Now, we're getting somewhere. The event name maps to a business intention, which is specified as a work-flow. But, the work-flow is a generic concept. It's not real unless there is code behind it. So, we build a bunch of modularized code that does specific functions, we wire it together with dependency injection and have a dynamic work-flow define the execution path.

An Ode to a Slacker

2006-10-17T19:09:00.000-07:00

I need to get around to writing this, and more in general, to finish some of the shit I start.

A week later:
Of course, this entry is a lot more complicated than a simple statement such as "get shit done." The issue is a delicate balance of life and work, and the venn diagram where they cross.

Projects start as fun little side things. You play around with them for a few hours, put some things together, and call it a day. Then, you get an email from the user saying, hey this is pretty cool, but it needs to be a lot more to be useful. Sure, you say, I'll add a few more lines of code. Unfortunately, now its not a small little project but a junior system. And not only that, but its a junior system which is poorly tested. You try to maintain the same schedule, but you realize that you can't add the kind of functionality that's needed, or maintain the level of quality necessary for a production system. You make mistakes, take shortcuts. Before you know it, your users are pretty angry. They are starting to question the whole thing. And frankly, so are you. You want to finish it. You desperately need to finish it. You've came this close, dedicated this much, but you realize that finishing will require even more.

This is an interesting struggle. The few lucky of us actually enjoy building things. So, this side little project may seem like work to some but is actually seen as basically a hobby. Unfortunately, some people are relying on your hobby, and that's when the pressure kicks in, and the problems start. On the other hand, unless you had a user who wanted something, you probably wouldn't have choose to build this particular thing as your hobby.

The other interesting observation is you are starting to see this project as something more. Maybe this project is the way out of the rat race. If it works, it could be your ticket. But its so much work you say.

How do you maintain the delicate balance? Is it even possible to maintain the balance? You're working with fixed items. There is a fixed amount of time. That amount is then reduced by constants such as actual work hours, sleeping, eating, showering, spending time with the family.
A week has 168 hours, 45 hours is spent at work, 49 hours is spent sleeping, 14 hours is spent eating, 4 hours is spent on toiletries. What remains is 54 hours to spend time with the family, work on the side projects, wash the dishes, do laundry, go to the movies, sleep in, watch tv, do the bills, etc... What ends up happening is you can probably take maybe 9 hours for the week - 1 per workday and 2 per weekend. Unfortunately, as everyone knows spending 1 hour programming is like watching ballet dancers do hip hop (it's not right). You can't accomplish anything major in 1 hour or even 2 hours. So you may start, but you tend to aim lower, and make a lot of mistakes in the process.

Wish me luck!

Randomness

2006-08-24T20:42:00.000-07:00

Here is an interesting question; if you know the past, can you guard against a similar event in the future? You know that the great depression took place. A lot of research has been done to understand what led to the great depression, and a lot of research has been done to understand how to get out of the great depression. In fact, the current chairman of the Federal Reserve is a specialist on the great depression. So, after all that, do you think it can happen again. With all this acquired knowledge, would we see it coming and be able to guard against it?

This question has been occupying me lately, and I am leaning towards a no. I don't think we'll see it coming. We may know how to guard against that specific event, but I am starting to believe that history never repeats itself. Events may seem similar, but there are infinite combinations of how they are triggered, how we react to those triggers, consequences, possibilities, and, of course, conclusions. If history never repeats, then studying history may not provide much value other than protecting us from that exact event.

My other opinion is that the world is getting more interconnected and more complicated. By this I mean that connections are forming that we may not realize exist or are even correlated. The world of the past will never happen again, and if the event of the past happens in the current world, the consequences will be quite different than before. Unfortunately, some other event may take place that may bring us the same type of devastation. Basically, my theory is that history can never repeat itself because the world is continuously changing.

There is some kind of random under-tone to the world. Some people call it luck, others misfortune. Let's say you trade stocks. You've read all there is about the company. You think you understand the fed, the government, the currency, etc... You believe very strongly in the financials of this company. You buy the stock. First it goes up, but then it drops like a rock. It seems that this company was dependent on the knowledge of a single engineer who was hit by the train. An unforeseen circumstance knocked you out of the market. What is that circumstance, is it randomness. Can you foresee it? Can you calculate its probability of occurrence? Do you understand its impact? I don't know, but it doesn't seem likely especially with our current understanding of probability. What is more likely is that we may get a safe feeling of security due to our acquired knowledge or perhaps our previous fortune, and this if nothing else will lead us to ruin.

The other item I wanted to cover was noise. This blog is noise. CNN is noise. In fact, a large part of the internet is noise. First off, the question is whether more information is noise or valuable artifacts. And if more information is noise, is it harmful? Does having more information actually increase your probability of making a wrong decision? Can you measure what information is valuable and what is noise. These statements seem very counterintuitive. What I am basically saying is that knowledge may actually be bad for you. Our brains seem to have adapted to this by actually reducing the large amounts of knowledge into manageable chunks. A lot of knowledge is simply forgotten, other knowledge gets reduced into some basic concepts and understandings. Does learning everything there is to know about a company such as all their news statements, their financial statements, statements made by their peers, etc... somehow takes away from the bigger picture?

If anyone out there has an answer, please, do write a comment.

Reflexive Theory

2006-07-07T19:30:00.000-07:00

The word reflexive means to direct back on itself. Don't confuse this with reflection, which means careful consideration or self reflection, which means careful consideration of oneself.

Reflexive Theory was originally created by a Russian mathematician Vladimir Lefebvre who is now part of a US think tank dealing with terrorism. Reflexive Theory was born during the cold war in Russia as a response to game theory which was widely adapted by the West.

What brought this theory to my attention is an article by Jonathan David Farley, San Francisco Chronicle: The torturer's dilemma: the math on fire with fire which was published on the Econophysics blog. I started trying to get a bit more information on the Reflexive Theory - Wikipedia had nothing, Google came up short, in fact, the only reference I could find is a link from a Russian site to a very old publication - go to page 86 for the relevant paper. And even there, the theory is never defined but is just applied in a simplified mathematical model of border protection from terrorism. I am wondering whether the fact that there is almost no mention on the web of reflexive theory has anything to do with the founder of the theory now being employed by the United States Government. Of course this thought pattern is better pursued on a big brother paranoia blog.

Reflexive Theory tries to explain mathematically why individuals take certain actions and what the consequence of those actions are. The theory takes into consideration how individuals perceive themselves whether good or evil, and whether those perceptions are valid or not.

The interesting thing about reflexivity is that its derived from psychology. The term actually implies that "reality and identity are reflexive". One implies the other. What we perceive is how we view ourselves and what we believe is true. This is a very powerful statement. This means that our reality is based on what we know, which is derived from our perceptions, which are based on our reality. This is a bit tough to swallow, but stay with me a bit longer. The whole point is that our reality defines us and influences our actions. In order for us to get a better understanding of our actions, and the consequences, and make additional evolutionary leaps, we need to step outside of our reality and view our knowledge and actions from that medium. I wonder whether traveling across realities is simply an evolutionary step where we can let go of our reality and understand the possibility of another. Alright, this last sentence is something that belongs in a sci-fi book rather than a blog on technology.

O. K. This blog is about technology not philosophy nor psychology or even mathematical models of terrorism. I am still working on how to tie this with technology. It's do able, but a bit theoretic, so I'll leave it for future entries.

Scene 1

2006-06-20T14:55:00.000-07:00

Scene 1:
The following conversation took place between two co-workers over an instant messaging product called Sparc.
The setting is a corporate office with many cubicles. The two co-workers have been asked to design
an enterprise, grid enabled architecture. They received a single 8 by 11 Visio diagram of the architecture.
They were also directed to leverage a FpML (financial markup language) as a messaging protocol between enterprise
systems.

Ben is an existentialist with a bend on fatalism. Mike is generally an optimist, unless Ben gets too fatalistic.

[4:45 PM] Ben: Boss wants to discuss FpMl
[4:52 PM] Mike: As long as we don't have to use it internally ....
[5:00 PM] Ben: I love it when the architecture is dictated from above. It makes designing so much easier.
[5:04 PM] Mike: And simpler, too. The whole system fits into a picture with a few boxes in it
[5:05 PM] Ben: it's pretty cool.
[5:05 PM] Ben: What's wrong with using FpML internally? You suck!
[5:06 PM] Ben: It's a nice intermediary format that allows system to communicate in a well defined language. Honestly Mike, what kind of an architect do you call yourself?
[5:06 PM] Mike: Hey, we've got enough on our plate writing our own database, our own operating system and our own programming language. I just don't need to have to use FpML as well as all that. I think it could jeopardize the entire project!
[5:08 PM] Ben: come on, that's crap. We can introduce this conversion in our custom db level, or even add it natively into our custom language.
[5:08 PM] Ben: think about it seamless integration with FpML, beautiful. Too bad FpML only covers derivatives, haha
[5:09 PM] Ben: I am sure we can work through that. We just need to work with the FpML working group to add a few parts to their spec.
[5:10 PM] Ben: 10-4?
[5:11 PM] Mike: Yeah, OK, it's taking me a while to think of a witty reply. 10-4
[5:11 PM] Ben: sorry to rush you, take your time. I just thought you were ignoring me because you are working.
[5:13 PM] Mike: Does FpML support images? In case we need to attach screenshots when we're reporting errors in market data?
[5:14 PM] Ben: Not yet, but I think we should bring this up when we discuss with them about expanding their specification to support other products.
[5:15 PM] Mike: I think whatever solution we go with, it's vital that we can scavenge unused cycles from people's mobile phones
[5:15 PM] Ben: and PDA's
[5:16 PM] Mike: and pacemakers
[5:16 PM] Ben: and watches
[5:16 PM] Mike: and elevators
[5:16 PM] Ben: maybe we can work something out where we can use the employee’s home machines.
[5:17 PM] Mike: Or people with Bluetooth devices in their briefcases as they wander past the building
[5:17 PM] Ben: good thinking, what about fax machines?
[5:17 PM] Mike: I wouldn't like to see any proposal signed off until we've really considered all these factors
[5:18 PM] Ben: I am glad at least you and me are on the same page.
[5:18 PM] Ben: We need to write up a document, someone will sign off, and then we can proceed with the development.
[5:19 PM] Ben: I think this conversation is sufficient as design.
[5:19 PM] Mike: Especially once we've deleted the Sparc logs
[5:19 PM] Ben: man, if someone has a sniffer, we're doomed.
[5:20 PM] Mike: That would be sad - especially as you introduced this product into CompanyX
[5:21 PM] Ben:
[5:21 PM] Mike: Do I have to 10-4 the smileys ?
[5:21 PM] Ben: no worries, I believe encryption is on.
[5:22 PM] Ben: no, don't worry about the smileys
[5:24 PM] Mike: I don't think we should restrict ourselves to FpML, either. We should have a meta-markup framework where we can just plug-in any standard that comes along - in case we need to support fPml or FPml or fPmL later on
[5:24 PM] Ben: I love it. Consider it added to the spec.
[5:24 PM] Mike: The spec which I hope is written in specML ?
[5:25 PM] Ben: should we look into whether we can leverage specML along side FpML as the messaging protocol?
[5:25 PM] Mike: Absolutely
[5:26 PM] Ben: http://www.mozilla.org/rhino/
[5:26 PM] Ben: I think we should use rhino tool to build out entire framework
[5:26 PM] Ben: think about it, we can release partial code, no reason to compile.
[5:27 PM] Mike: I've used Rhino before (indirectly) - it's built into JWebUnit
[5:27 PM] Ben: so, what do you think of using it as our core language?
[5:30 PM] Mike: Might be a bit low-level. I want something high-level that maps 4 boxes on a diagram into a fully built-out, productionised, resilient, performant, scaleable, internationalised system that runs on everything from a supercomputer to a Beowulf cluster to a digital watch.
[5:32 PM] Ben: do you have a copy of our entire conversation; I think it would make a wonderful blog entry.

Commodity, Speed of Light, EAB

2006-06-10T20:41:00.000-07:00

This post was actually going to be about technology commodity. I actually even wrote a part of it. I had all kinds of things in there - a definition of the word commodity from the wikipedia, a reference to Karl Marx, processing grid, service grid, data grid, etc... It was going to be a pretty good post before I erased it.

What the hell? Well, I had no point, I was just writing the obvious. Let me start over. Any enterprise architecture is going to be distributed, but that's not enough. Systems need to communicate and share data. Some systems may provide a service to other systems. Some systems may be in-charge of routing messages. Other systems may be in charge of doing calculations, other systems provide auxiliary services like calendar, or caching. The point is that a whole lot of systems are going to be communicating. In fact, in some cases, that communication will be very heavy and may become a liability. An Enterprise Architecture Bottleneck(EAB). You gotta love acronyms. They make everything sound so much more impressive.

In order to reduce EAB, your system will need to reduce the amount of data being transferred, figure out a faster transfer method, go faster than the speed of light, or all of the above. For the sake of simplicity, let's assume the last point is currently not feasible. For the second item, you can buy a bigger pipe, but you are still stuck to a certain latency. The cost to transfer a bit from NY to London will always be bound to the speed of light. So, can the system reduce the amount transferred. I think it's possible if the system is aware of the data patterns and can remove any unnecessary or redundant information. For example, let's say me and you are carrying a conversation. Certain things are obvious, other things can be deduced by you without me saying anything. For other things, I may only need to say a few things for you to understand much more, and in other cases, you may already know certain things, because I've already mentioned them. What does this all mean? The sending system will need to analize the data stream and learn to reduce the load. Of course, this assumes that the sender and receiver have agreed on some transfer protocol.

Well, this post is still pretty obvious, but maybe a bit more interesting than another tirade on processing grids.

Business Process Modeling

2006-06-08T19:14:00.000-07:00

I was discussing a system process with a business client, and during the conversation it hit me that the client has no idea what the business process is. At some points during the development of the system, a business specification was written, perhaps even by this same person, but overtime, the system evolved, and the process forgotten. Now, my business client has no idea what the system is actually doing. To him, the system is a black box, one big scary unknown. He has a general idea of what it does - it takes data from some place, does some filtering and calculation, populates a few tables, does some more calculations and filtering and eventually generates some results. The business analyst is then charged with the verification of the results - the irony is that the business analyst isn't completely sure of how the number was generated. He has a certain level of trust, but overtime that trust erodes and he is left feeling uneasy and unsure.

Later that day, I sat in a meeting where a discussion was taking place about a new business system. It was a usual meeting, many people spoke about many things, most of the conversations strayed far from the actual deliverable or even reality, nothing was agreed, a few people tried to cock fight each other, and, overall, the meeting reversed progress - usual meeting. Anyways, during the meeting, specifically during one of the cock fights, one of the analysts spoke up and said something very profound and interesting: this was followed by a cock block, but that's a different story. So this interesting statement was that he believed that quality analysis testing should not end when the system goes to production but should be an ongoing part of the system. He believed that it was important to have a stable validation layer in the system in order to provide basic sanity checks that the system is performing as expected in an endless parade of changing data. My team-members rose up in anger against him, some claimed he was a heretic, others threatened ex-communication. I sat silent listening and wondering.

Each system is basically a workflow. Once you remove some of the techy parts, you end up with a business process. In fact, at some point this system was a very clean visio diagram. Each box was then blown up into a class diagram, and then some crack smoking code monkey (developer) defecated all over it - an enterprise system is born. This workflow is then overlaid with data. The workflow reacts differently to different pieces of data, but its still functionally a flow - actually more of a graph. The graph is a mix of generalized technical aspects and business logic. The problem these days is that the business logic is sprinkled all over the system making it very hard to re-create exactly what happened.

So, I wonder if it would be possible to overlay an actual system with a meta-system. Would it be possible to create a set of, let's say annotations, to add along side code and possible some additional hooks to allow another system to walk the system code to generate the graph, and overlay the graph with the business documentation - sprinkled throughout the code. The end result can be a self-documenting system. No, I am not talking about javadoc, or external specification. I am talking about a tool for the business user to verify what a given system is doing. Because the documentation and the code are living side by side, perhaps are even the same thing, the business user can be confident in what they are seeing.

The second part is that a lot of data centric systems live and die by the data they are receiving. Garbage in, garbage out they say. Well, I am not quite sure this statement needs to be true. After a long deep thought, I agreed with the business analyst and took a stand to support him. I think he is right, QA should not end once the system is in production. Each system should be built to be in a constant state of testing itself. The point isn't to test the code, the point is to test the data. The data is the most important thing. As developers and architects we treat data as a second citizen. What comes in to the system should be checked, what happens in the system should be checked, and what comes out of the system should be checked. It would help if the checks are a hypothesis test. The analyst proposed having a parallel testing dataset. He figured that a constant check against a constant may provide a basic sanity check or at least maybe raise some red flags if the data is too far from the norm. Of course, this type of test is context specific, but I think the basic principle has value. Data isn't just data, it's the most important thing. When the business analyst receives the end result, and the end result is wrong, the analyst spends hours trying to narrow down what went wrong. Sometimes the problem is the inputs, sometimes the problem is the business logic, other times, he just doesn't know.

I wanted to get this post out, but overall I am still thinking through a lot of these concepts. I think there is something conceptually there, but its a bit foggy.

Robo Blogger

2006-05-23T18:25:00.000-07:00

I recently had a discussion with an associate of mine, let's call him Philip. I made a wager with Philip. I bet him that if I wrote a robo-blogger, his blog will be more popular than mine. The robo-blogger is going to subscribe to all the popular blogs. He will summarize the blogs, and then comment on them. The comments will range from outright bashing to a more supportive tone. The bashing will also occur using different regional dialects such as Australian. For example, this blog is just piss-farting around - translation: this blog is just wasting time. Another example, this program is so cactus - translation: this program is just wrong. For more: http://en.wikipedia.org/wiki/Australian_words

I believe the only way I can loose this bet is if I don't build it. Watch out Philip, YOU ARE GOING DOWN!!!!

Ontology

2006-04-23T11:35:00.000-07:00

Ontology is "...a systematic arrangement of all of the important categories of objects or concepts which exist in some field of discourse, showing the relations between them. When complete, an ontology is a categorization of all of the concepts in some field of knowledge, including the objects and all of the properties, relations, and functions needed to define the objects and specify their actions."

So, an ontology is a language to represent all the objects in your field, along with their properties, and relationships. For example, a trade contains an instrument. A trade contains a price. An option instrument is a type of instrument. An option instrument contains an underlying instrument. An index instrument is a type of instrument. An index instrument consists of sub instruments. etc... I am thinking that in theory, you could represent an entire industry or a body of knowledge using an ontology. In theory, once you have a representational space, and given a new instance of an object, you could apply the object to the ontology; find your place, and then be able to walk the ontology to figure out all the possible causes and effects.

There has been some work done. W3C came out with a language specification - owl. Also, see the links below for more references.

Links
http://jena.sourceforge.net/
http://sweetrules.projects.semwebcentral.org
http://protege.stanford.edu/

Interesting Links - need to organize

2006-04-01T11:37:00.000-08:00

More to follow...
http://opennlp.sourceforge.net/projects.html

http://wordnet.princeton.edu/

http://weta-group.net/

http://www.cs.waikato.ac.nz/ml/weka/

Metaverse continued ...

2006-03-31T20:46:00.000-08:00

I've recently came upon this site: http://www.youos.com/

and this site: http://www.ajaxwrite.com/

and I've found these to be absolutely exciting. The YouOs is absolutely great. They redefined the concept of an operating system.

I think what needs to happen is for applications to break out of the browser. YouOs, and AjaxWrite are a beginning of this. What is the point of the browser. It is a very heavy and limiting medium. What if you created a browser wrapper, something that lives neither here nor there. With Ajax taking over, it is possible to move a piece of the website to your desktop and work with that chunk as if it was a real application running on your PC. The whole browser is only confusing the matter. It's really not needed. A browser is nothing but an interpreter in an interpreted language. The only difference is that, right now, the browser is also imposing a look and feel and is constraining the interaction between the user and the service.

Imagine a powerful application like the AjaxWrite. Now remove the browser, and create a link on your current desktop that instead of opening a browser just runs the app. The app decides the look and feel, etc... I know there are security concerns, but for the sake of progress I would rather ignore them for the moment. Now, you have a part of your desktop running a web-based application. The whole thing is running on some other server, and you are simply interacting with it. The difference is that you have a seemless integration with your environment.

At the moment, there is such a clear separation between desktop and web. In my opinion, they really are the same thing. What is the difference between running something locally and running something remotely and only bringing back the display. Both systems react to the user events, the only difference, is that the desktop system is bound to your machine.

Rules and Monads

2006-03-26T14:59:00.000-08:00

Word of the day, Monad: I believe Leibnitz first applied it to science or perhaps philosophy. In any case, the word is defined as "...an indivisible, impenetrable unit of substance viewed as the basic constituent element of physical reality..."

I am currently contemplating a system that will allow the creation and testing of validity of atomical units, monads. Small units build up to larger units, to even larger, to ever so more larger. At the end, by testing the indivisible, impenetrable unit of substance, I can find out whether the sum of sums of sums of sums equaling to an even larger and more complicated entity is valid.

Well, solution one, write lots of lines of code and hope to God that nothing changes and your infinite amounts of if-then-else statements nested ever so many layers deep across ever so many classes in your lovely object oriented patterned out piece of shit.

Solution two requires a bit of imagination, a workflow, and rules on monads. Let's assume that you've built some wonderful gui that allows you to create and modify rules.

Now, each monad can actually be represented in a 2 dimensional space. Each monad is for a specific type of data: stock price, live stock, live stock weight, insurance policy, etc... Each monad represents a specific type of data. Let's say that's the X-axis. Each monad is also for a given instance of that data. For example, a price is uniquely identified as cusip/date, and each cow is uniquely identified by its id. Each transaction has a transaction id, etc... We can call this the Y-axis. So, given the X and the Y, we can uniquely identify every monad.

At this point, we can right a rule that tests that monad. So, we can test every indivisible unit, and then we can test the summation of these units using other rules that perhaps only group the indivisible monad rules. At the end, all this adds up to a single high level rule that tells you whether your data is valid or not. Along the way, your rules may change data, or take you on a different path, but that's another blog entry.

It's actually extremely elegant. I apologize if my description skewed the beauty or confused the matter. The more I think about this, the more I believe in it. Now you may be wondering: am I not describing an expert system built using a rule engine. Somewhat true, in fact I am describing an over simplified rule engine. The big difference is that instead of allowing each rule to be a self contained entity, this model assigns a rule to a particular type, and the rule outcome to the type and instance of the data object. So, instead of having a bunch of free for all rules, you have a controlled rule structure, and a controlled firing. This is one thing that I found to be very brittle in rule engines. The writer of the rules does not know when the rules will fire and has to be very careful in writing the rule filter. Now, some will argue that that's the whole point. Well, try selling a none-deterministic system to your business manager who is weary of the whole thing to begin with.

Anyways, as you can see I am a big fan of externalizing the ever changing business rules from the underlying system framework. This is an interesting idea, and really should be developed further.

Cromwell's rule

2006-02-19T11:18:00.000-08:00

Cromwell's rule is derived from a letter Oliver Cromwell wrote to the synod of the Church of Scotland on August 5th, 1650 in which he said "I beseech you, in the bowels of Christ, consider it possible that you are mistaken." The rule is, very simply, do not take anything for absolute; totality is wrong.

I am not Christian, in fact, I am very much Jewish, and certain parts of my anatomy will certainly show that, but I have a very great respect for that statement. The world is entering a very dangerous period. Everyone, and I mean EVERYONE, Jewish, Christian, Muslim, Buddhist, Zoroastrian, must consider the possibility that they are mistaken.

Very recently a small Danish newspaper published a set of cartoons making fun of Mohammed. They did it to show that the media is censoring itself on criticizing Mohammed. So, what happened. Well, the Muslim world went bizerk. They burned consulates, killed people, robbed banks, oh yes, robbed banks, marched, protested, burned flags, fired oozies in the air, chanted death to Israel, etc... An Iranian newspaper even created a competition for the best holocaust cartoon. Undoubtedly, numerous entries were submitted.

Everyone considers their position to be superior. No side understands or cares about the other. One side will tell you that the other does not understand or care. The other side will reverse the statement and tell you the same. Don't believe me. Here is an example. The greatest pain a person can feel is the passing of a loved one. That is the greatest fear and the greatest pain. A part of you dies as well. The more you love someone, the more you will understand this statement. On the other hand, one hundred Palestinian children being killed by a stray bomb will produce a quickly passing feeling of regret for their unfortunate passing.

Every culture, at every point in history that believed in totality brought about the greatest suffering up to that point in history. Consider Hitler, Stalin, Mussolini, Ferdinand and Isabella of Castile, Lenin, Mao, Christian Crusades, and so forth. This could be a very long list, but a few of these names should prove my point.

Someone has to be correct right? Maybe. This question seems extremely complicated. I don't know.

What I do know is that any culture or any person that believes something in such an absolute is dangerous and no matter how well meaning is horribly wrong. Nothing is simple and clear cut. Everything has multiple sides, and therefore any totality is always wrong no matter how well meaning.

I think Queen Margrethe II of Denmark said it best:
"There is, as said, something moving about people, whom to this degree surrender to a faith. However there is also something frightening about such a totality, which also is a side of Islam. There must be shown counter-play [interplay of an alternative / sparring], and once in a while you have to run the risk of getting a less flattering label stuck upon you. Because there is certain things before which one should not be tolerant."

Frustration

2006-02-12T10:22:00.000-08:00

I am extremely frustrated with the keyboard and mouse, specifically the mouse. I find the mouse to be a very outdated device that simply doesn't keep up with me. Good hackers don't even use the mouse. A lot of times, I find myself simply memorizing all the possible shortcuts and deal only with the keyboard. The reason is exactly that, shortcuts. Its faster to type out some cryptic combination like holding a key down while pressing a set of other keys rather then to nagivate the mouse to a menu, and perform a few clicks. I think the problem is that the mouse is not an extension of the hand. It's unnatural. The eyes and brain move infinitely faster. Perhaps, if the hand was the mouse. It takes such a long time to move the mouse from one side of the screen to another. The mouse is made slow on purpose because it is not a very precise device. Imagine how hard it would be to navigate it, if the mouse flew across the screen. The human hand, fingers are a lot more precise. In theory, I think the hand acting as a mouse can move much faster. There is also a possibility of interacting more naturally with the computer if your hands can be involved. For example, physically taking and moving windows rather then performing some sort of click and hold motion. There are also 2 hands and 10 fingers providing a potential for even fuller interaction. There is no reason that the mouse has to be a single item. What about 10 different mouses, or perhaps there is no mouse at all, but your hands can physically interact with the computer in a virtual plane.

Perhaps part of my frustration stems from that fact that it takes a very long time to do anything on the computer. Imagine building a simple web based program. There are a couple of screens, lets say 5 queries, some fancy interaction such as ajax, perhaps some column sorting, expanding, maybe some data pop-ups, some drill downs. A little program like this will probably take one person a few days, maybe even a week to fully build out and test. That's extremely slow. This system might not even be what the business wants causing further delays. There has to be a faster way. There is nothing even remotely complicated or interesting about these little programs. These programs are simply a test of how fast one can type and how easy they can define the architecture to minimize typing in the first place. We are developing stupid little programs at such a low level. It is extremely inefficient, not to mention boring.

The solution so far has been to build tools on top of tools. IDE's are attempting to simplify certain actions, visual development tools are attempting to convert languages into workflow models. IDE's are replacing development. They are simply making it faster. Workflow based languages are creating another layer of complexity. Instead of writing an if statement you move
a picture of an if statement into a plane, and then configure it by filling out a form. This is an interesting idea, and has the capacity to produce very powerful systems rather quickly, but I am not a true believer. I don't have a solution, I am just really frustrated.

Soft Asset Management

2006-01-26T19:48:00.000-08:00

Each organization struggles to harness and manage 3 simple assets.

Ideas
The rank and file engineers have a lot of very good ideas that never see the light of day. The reason for this tends to be a layer of dead managerial weight separating any developer from any person capable and willing to make a decision. The middle management is also extremely terrified of anything that's perceived as rocking the boat or anything that causes them to stick out ( such as championing an idea). The problem is that a lot of these people are simply afraid of showing how incompetent they really are. What needs to happen is a way to send ideas directly to the senior management or some sort of governing committee that can review the ideas and has the power to fund them. I've seen one organization implement this very well. They've setup a committee consisting of senior management to review all incoming ideas. The idea is emailed directly to your regional director. (There is one regional director per continent.) The director would then take your idea to the committee for review. At the end of the year, the committee chooses a handful of ideas to pursue and those people receive awards such as trips to Paris, trophy, envelops, and recognition - due to a large assembly that you get to speak in front of.

Prior Art
"Prior Art" is a term used in patent law to describe inventions that someone has already patented. Each organization produces a lot of innovation: software, libraries, fixes, procedures, etc... The ironic thing is that each group in the organization tends to develop its own solution to a problem probably faced by every group. In a couple of places, I've seen organizations attempt to avoid some duplication, but never very successfully. At the core, the problem is that each group has no idea what the other is doing. Additionally, there is rarely a single architectural vision or a way to search for existing solutions. Basically, the proposal is to setup an internal open-source community. The community would know of all projects going on in the organization, and would manage certain projects submitted to it such as general projects like scheduling systems, or monitoring libraries. Everyone should be aware of what the other is doing. If someone in the organization is working on a security model, and I am about to start writing my own password management system, well, hopefully I know about the other initiative.

Knowledge
Each engineer accumulates a wealth of knowledge. The most relevant and long lasting learning occurs from your peers and not from a 2 day intensive class. Although, its great to miss 2 days of work and get catered lunch. What organizations struggle to do is share individually acquired knowledge. In fact, a number of organizations, because of politics discourage the practice or do it with a heavy managerial hand. For example, in one company I've worked for, the management decided that the way to share knowledge is to hold bi-weekly developer meetings. Their solution was that each manager or director would bring some of their people to the meeting. In some cases, the directors brought only managers. The end result was that the meeting consisted of 30% developers and 70% managers/directors. The funny thing is that the topics started technical and quickly moved to managerial such as billing. I spoke up in one of these meetings. I believe I said something to the degree that this is mockery, and if both my manager and my director were not sitting next to me, I wouldn't be here. Well, I am no longer with that particular company. Some ways to promote knowledge sharing is to get developers talking and debating. For example, the organization can create a developer community and hold weekly meetings. During each meeting, one of the developers can present something useful and interesting to discuss such as programming patterns, algorithms, and standards. Additionally, within a team, discussion should be encouraged. Team leaders should promote a culture of learning and challenge such as emailing logic problems. The only reward is the recognition that the person is better then everyone else. The organization should also implement code review and design review. The reviews will enforce a level of competency which is sometimes missing during coding and designing.

Metaverse

2006-01-16T10:25:00.000-08:00

The word meta in Greek means "about", "beyond", and in English is used as a word prefix to "... indicate a concept which is an abstraction from another concept." The word verse is "... a single metrical line in a poetic composition; one line of poetry. "

The word metaverse was coined by Neal Stephenson's in his book Snow Crash in 1992. Neal uses the word to describe a virtual world that allows people to connect into and physically participate in a virtual world. People physically jack-in. The world is part virtual reality, part internet, part story.

At the moment there are a few games that are starting to broach this subject of metaverse. Games like everquest, for example, bring together all players into a single universe. People now hear of things like weddings occurring in cyberspace or real-estate being sold for real money. There is also a commodity market developing for game items such as magical weapons. Virtual items are being sold and bought with real money. Of course, one can argue that money is as real as the magical cloak that can make the owner appear or disappear or protect the wearer from all sorts of attacks.

The major difference between the cyberworlds these games create and the metaverse, is that the metaverse is not a game but real life. It has stores and restaurants, bars, and clubs. It has side walks, and bus lines and a very sophisticated police infrastructure. There is a class system, and a stable real-estate market. To get there, we need a basic virtual reality system, and a simple meta language like HTML. It should be relatively simple to create stores and shelves, and chairs, the humans will act as humans. The imaging can be performed locally, with only meta language being passed around. You should be able to walk into a store on the internet and have the same or better experience as walking into a real store.

Check this out: http://sketchup.google.com/

Patent Madness

2006-01-11T18:36:00.000-08:00

I am currently in the process of trying to patent an idea. The idea is relatively simple, but this post is not about that. Instead, it is about patent law. In my search, I got the opportunity to speak with a highly paid patent lawyer ($460 per hour) from an established firm. He gave me a breakdown of what is required to patent an idea.

For technology related ideas there are 2 types of patents, a full blown utility patent, and a provisional patent. The provisional patent is good for a year, but requires that you file for a full patent within that year. It's relatively cheap, and doesn't go through the rigor of a full patent. In fact, there is no rigor, whatever you file is good. The problem, as explained, by the lawyer, is if there is a dispute within that year, the language of the provisional application is closely scrutinized. Basically, if its written by the lay man, me, it won't stand up in court, and therefore is useless. The only advantage is a marketing gimmick, because it lets you say "Patent pending" for the duration of that year, and if it does stand up in court, your patent is for 21 years rather than 20.

To file a full patent with this law firm, you need to do the following:
$400 = initial 1 hour consultation with a junior lawyer and some feedback from a senior partner
$1500-$2000 = a professional search to see if your patent is already taken
$5000-$10000 = to write up the application form, the price ranges depending on complexity.
+ cost of filing, for small entity, (adds up to a couple of hundred)

He also mentioned that almost everyone gets rejected on their first submission. Your patent lawyer and the US Patent Office than negotiate on how broad your patent should be.
At the end of the day, you can have a patent for just under $20,000. The industry average, as explained by another lawyer, is $15,000 to get a patent. The cost fluctuates between $10,000 and $20,000 depending on who you get as the US patent representative, and how general your patent is. The expensive lawyer mentioned that the Amazon single click patent was very expensive. Another lawyer also added that the international patent costs even more than US. The filing fee alone is $4,000.

US Patent fees (Most of us are small entities)

World intellectual Property Organization

The patent process also takes on the average of 2 years.

If you do it yourself, you may get a patent for much less, probably around $2 -3k, but without the lawyer babel, its not worth the paper its printed on. Basically, as the highly paid lawyer explained to me, patents are written by lawyers for lawyers. Patents are only useful in court. He also kindly explained to me that the patent process is not meant for the small guy. Even if you get a patent, you may not be able to afford to defend it. Another lawyer chirped in that if you do have a patent and a requirement to defend, there are certain companies that will pay the defense fees for a share of the patent. This same lawyer also said that a patent is only necessary if you know how to make money from it. For example, license your idea, sell the patent, protect your idea so as to control the market. Patents are not required in a lot of cases, also, if you've already built the software and somehow released into public knowledge, in court, you do get some leniency during a patent dispute.

Grid

2005-12-30T07:26:00.000-08:00

There is shift in the tech community towards grids. Oracle came out with its 10G database, now Sun has released the Sun Grid, and is offering $1 per CPU hour to run apps on their grid. At the moment, grids are still a novelty. Berkeley introduced the grid to masses through the search for aliens (SETI). Their system, BOINC, is a very nice and simple grid, but is very University geared. There are a number of vendors and a couple of enterprise strength open-source solutions.

Vendor Tools
http://www.entropia.com
http://www.datasynapse.com/
http://www.avaki.com
http://www.platform.com/
http://www.ud.com
http://www.gridsystems.com/

Open-Source
http://www.zetagrid.net
http://www.globus.org
http://www.cs.wisc.edu/condor/
http://gridengine.sunsource.net/
http://boinc.berkeley.edu/
http://www.globus.org/cog/java/

The other interesting development is that people are talking about Grids in the same breath as web-services. Personally, I am not a big fan of web-services, and has always considered web-services a solution to a very specific and narrow problem space. A grid can be thought of as a single computer with a whole lot of CPU's. I guess running apps on the grid is really just a service, and web-services, in theory, is attempting to standardize that communication. A service oriented world is interesting, but I am not convinced the world works that way.

On top of all this bable, I am proposing to build a PSG (Pretty Simple Grid) as an open-source project. The grid, unlike all other grids, will be geared for small teams whether in large organizations or not. The grid must be simple to install and simple to use. It's features will be limited to what is required and what doesn't require much setup. It will be written in java; client download-able via internet; the client will run as service, screen-saver, etc... The goal is to allow individual technology teams to leverage the grid in their offices. Most of us working for the Man probably work in pretty small teams, but are a part of a large entity that is impossible to convince to install a large grid. A small grid used solely in the small team with their 10 pc, is simple to install and provides a lot of value.

Are great developers dopamine addicts?

2005-12-06T19:03:00.000-08:00

The below post is not complete, and not fully developed. I believe there is some sort of connection between dopamine and great developers. But, dopamine may not be the cause, but an effect. In any case, enjoy the post but don't judge the author.

Dopamine is natural chemical produced by the body. Based on medical studies, researchers have discovered that during periods of excitement and satisfaction, dopamine levels in the brain increase. One such study, discussed in the Wall Street Journal, shows that the reason people enjoy shopping is because of the excitement of trying on something new, or experiencing something new. For example, they've found that people tend to buy more things when they shop in a new environment such as another city.

Great developers tend to be people that are constantly searching for new challenges. They are in a constant pursuit of the known. Give a great developer a non-trivial project, and ask them what they think of it. A great developer will tell you that the project is fun and interesting, a bad developer will complain that the project is hard. Give a great developer a simple project that they've already mastered, and ask them what they think of it. A great developer will complain of boredom, and the lameness of the project. A bad developer will seem happy to have received an easy task.

So, where does dopamine come in? I am thinking that the reason great developers are in a constant pursuit of the known is because the pursuit is exciting; pursing the unknown gives them pleasure. It's the excitement of the chase. Once the chase is over, the excitement is over, the dopamine level decreases, and the developer becomes sad and bored until the next challenge.

Distributed Computing

2005-10-21T16:18:00.000-07:00

http://www.sun.com/service/sungrid/overview.jsp

http://boinc.berkeley.edu/

more to come...

Autonomous Computing

2005-10-16T09:10:00.000-07:00

Autonomous Computing is a very popular buzzword at IBM. The term describes a system that has self-configuration, self-healing, self-optimization, and self-protection. (copied from IBM)

A system that has all these attributes is excellent. The system does not require configuration. If you put a system on a production machine, it will automatically recognize that its in production and identify production configuration settings. In case of an error, the system will attempt to identify where the error originated from, fix the problem, and correct side affects by re-processing business logic, for example. The system is also able to monitor itself and perform tuning. For example, the system would recognize that certain data-structures tend to be of specific sizes, and initialize the data structures with the necessary size rather than continuously performing costly re-size operations. The system might also identify certain processing that can be done in parallel, and automatically split the processing. The system will also attempt to survive. If the production server is inadvertently stopped, the system will migrate to a different server, re-configure, and continue. In another scenario, if the database fails, the system will switch to a different storage medium.

Great, absolutely great; really hard to do. At interesting problem arises when building non-deterministic systems. They are very hard to test. More specifically, it is very hard to know exactly how a system will react in a scenario that hasn't been considered. For example, a system might rerun certain jobs numerous times causing data corruption, or inadvertently switch servers causing data fragmentation. The system might fix a data error by tweaking variables that could cause data problems without generating any errors. The bottom line is, for risk critical systems, non-deterministic machines have a potential to cause more harm then good. This might be why the business community has been weary for AIish technologies.

I am a great believer in non-deterministic systems. I think there is a great benefit to them. The problem is how to introduce them in a way that makes them more deterministic. The probable answer might be to build more complex systems. Another answer might be more descriptive languages. Each function might have attributes telling the system what could be done with this function. If the function modifies data, then it is not idempotent, etc... The system almost has to understand what its limits are, and work within the given confines.

SOX - Security Policy

2005-09-26T19:35:00.000-07:00

I was asked to solve an interesting problem today. The problem is to hide the database password from everyone but the production system that uses it. The db account information is currently stored in properties files. The properties files are in plain view on the production boxes, along with the version control system, etc... The goal is to have the password reside in a single spot, and in such a way, where it is still accessible by a couple of production system across the globe, but is not known by anyone but the senior manager and the dba. hmmmm. It should also be possible to change the password by modifying it in one spot, and have every system automatically start using the new password. hmmm again.

I thought about this for a bit, and came up with using public/private key cryptography. The idea is to put a private key on each machine that needs to use the production db account. The private key will only be accessible by the system account. The system properties file will contain a guest database account that will have access to a password table. The password table will contain a crypted account information that was encrypted by the dba or the manager using the associated public key. So, the dba crypts the db account using the public key, writes the crypto into the table. Each system has a guest account to read the table and has access to the private key which will decrypt the account. The system will then drop the guest connection, and re-create db pool using the decrypted production db account. The solution sounds good, but has a major flaw. It requires a guest account on a production database system. The guest account might not sound very dangerous, but it allows the hooligan to start from within the database rather than have to figure out how to even connect to it.

Heuristic

2005-09-15T11:25:00.000-07:00

manifesto

More to come...

Humility

2005-09-09T21:12:00.000-07:00

I saw something today that I would not wish upon my worst enemies. I saw humanity at its core. There was nothing that could have been done. Nothing in the world could have changed it.
Our bodies are extremely fragile, and it all ends as quickly as it starts.

The Rabbi said it was meant to be. Our lives are pre-ordained, he said. It is what it is. Fate.

He went on to say that we come in to this world with our hands closed, and leave this world with our hands open. In the beginning we are selfish, and want for ourselves, at the end we take nothing except for who we were.

There was nothing to do. I stood at the edge watching as her husband shoveled dirt. It was an unbearable site, but it had to be done. And, he had to do it. To be in that position is absolutely unthinkable, the absolute misery. But it had to be done. It was very important.

That's it. That was the end. There was nothing that anyone could have done. How can our lives be so fragile, and we spend them so recklessly. The Rabbi said that what we take from this world is who we were, what we accomplished, our respect, our dignity, humility.

Time goes by so quickly. It rushes by, going quicker and quicker. I feel it now; every day time goes faster and faster. Hours roll into days, weeks, months, years. Years go by as fast as a minutes. Ideas, moments, events, opportunities, gone as quickly as they appear. Some are forgotten and lost, others remain as a memory, a feeling, wrapping themselves around us, forming who we are, what we shell take.

I need to take life more seriously, or perhaps, the goal is to take it less seriously. Be a person said the Rabbi, that's it.

Steganography

2005-08-21T08:13:00.000-07:00

There was a post on slashdot about Steganography. Now I am not much of a slashdot reader, but chaos prevailed, and I was on slashdot. The concept is great. The ability to hide data within other data such as an image. There are a couple of algorithms available out there, but it seems none are mature enough for general usage.

There is a good article here, at least an article that rates high on google.
http://www.guillermito2.net/stegano/ideas.html

The guy makes a very good point that all data out there is not random, seems random, but in fact there is order to chaos. Encryption algorithms tend to go against the grain. Its like dumping a pink elephant in the middle of Time Square and asking what's out of the ordinary. Most encrypted files beckon for decryption. They sit out there in plain site with nothing hiding them but the encryption algorithm.

Encryption if very complicated, Steganography seems even more complicated. I am wondering if its possible to use the actual image as data; versus adding something to the image, or modifying part of the data stream like the header or certain insignificant bits. Given a file and an image, would it be possible to produce a mapping such that the file maps to the image, and given the image and the key its possible to produce the file. The key is obviously very complex and very specific to the file. The key is meaningless by itself, and so is the image. Of course, the weakness is that the key must be protected. The beautify is that there is no encrypted file sitting somewhere. Instead there are trillions of images sitting in the public domain. No algorithm defined, but it should be possible. For example, given a neural net, it is possible to get the correct result. The image is the outcome, the file is the inputs, the net then figures out the weights on the nodes to get the right result. The weights then make up the key, assuming that reverse is true, which I am not so sure.

Software Complexity

2005-08-02T19:17:00.000-07:00

I spend a lot of time developing large systems. I find myself continously wrestling with the fact that in order to make a system easy to use, it needs to be exceedingly complex. You could almost draw a direct parallel. As the system becomes easier to use, its complexity increases by the same factor. The side affect is that maintaining a large system becomes that much more complex. Each system has certain drawbacks that were probably rationalized as functionality, or perhaps functionality that later become a drawback. For example, lets say a system maintains some internal data cache. The data needs to be refreshed every day.

Solution 1 is to enforce that the system is restarted every day causing the internal cache to be refreshed.

Solution 2 is to build a timer that automatically refreshes the internal cache every midnight. The timer locks the system for that period and refreshes the cache.

Solution 3 is to build something dynamic that is able to identify a slow period, then lock the system and perform the refresh.

Solution 4 is to build a partial cache refresh, were only changed items are refreshed. The system automatically scans the data source every few minutes, identifies the changes and inserts them into the cache, locking only the data source.

Solution 5, the external data source notifies the system when data changes occur, and the system performs a specific cache refresh, locking only the items that changed.

Solution 6 incompasses solution 5, except instead of locking, a seperate cache is built and swapped in during a slow interval, etc...

There are a lot of solutions to this problem. In fact, there are a lot of solutions without even considering AI where you can start having nueral nets try to predict slow intervals, optimal times for refresh, or which data is likely to change, etc... Each solution makes the system more flexible and much smarter but with an ever increasing complexity cost. Solution 6 will probably yield the most flexible system with the least amount of downtime with perhaps some AI thrown in. But, Solution 6 will also be very complex with quite a bit of code envolved in making this work. There is going to be a seperate system that needs to be aware of data changed, a method of notification, ability to modify and lock parts of the cache, ability to identify slow periods or periods where the specific data is not used, etc.. Solution 6, if built correctly, will probably require little day to day maintenance but once it fails will be very complex to troubleshoot and fix.
So, what's the point of all this? I don't know. I like to make complex systems simple, but that's a loosing battle. Perhaps, the curve is not really a a straight line but more of a bell curve, where after a certain point, the system complexity drops. The system becomes so complex and so smart that it is actually easy to maintain.

Sarbanes-Oxley

2005-07-24T19:03:00.000-07:00

The organization, I am currently consulting for, is in the heat of trying to comply with Sarbanes-Oxley. I hear this name mentioned at least twice a week, if not more, as a reason for a number of wrong and potentially dangerous decisions. The act, through legal wording, goes through great detail to specify how auditing shall be done, how its paid for, who is going to do it, what the deliverable will be, etc... The report explicitly assumes that auditing is a major fix to all the accounting problems we are having. As long as the auditors do their job, everything will be fine. The report also touches upon record keeping, and briefly mentions, I believe in one sentence, the requirement to maintain safe access to a production system.
It is very import to question what auditors say. Some of my friends are these auditors; they are overworked, just out of college, being promised partnerships, and the sky. The act is designed for auditing companies, if not written by them, not the companies that are being audited. The requests the auditors are making are going to cause more harm then good. Yes, the requests sound excellent on paper, but will create a huge mess on the ground. The implementation of this Act is also being unnecessary rushed, probably for political reasons. DO NOT RUSH THIS! Fixing the mess will be much more expensive.
I am also having some trouble finding the exact places that outline all this requests mentioned by the Auditor. Perhaps, the Act simply gives the auditor control, and outlines that the auditor will be partial and correct. The Act then simply assumes that the auditor is right in his statements, and the company is required to comply.
If you take away anything, question everything. The auditors are following subject lines rather than diving down and understanding the text.
Compliance is, of course, required by law, but it should be done in a way that develops a limber organization that is able to adapt and involve. Rather than, an organization stuck in a paper trial and an ever evolving mountain of red tape.

Distributed Groove

2005-07-11T19:49:00.000-07:00

I spent my day troubleshooting JGroups. For those unfamiliar, JGroups is a open source middleware system. The problem was that sometimes when a client system is restarted, the client would fail to see the cluster and instead become a coordinator of its own cluster. After a bit of research, the problem ended up being connected with long garbage collection delays on the real coordinator. This caused the server to not respond to heart beat requests, which in turn caused the client to think that it was alone and therefore become its own coordinator in its own cluster.

To solve the problem, I started looking through the JGroups source code and I stumbled upon a reference to the Lamport timestamp algorithm. I remember studing the algorithm in school. The basic premise is an ability to understand logical order of messages in a distributed environment based on a concept of time. His paper goes into quite a bit of detail, some of it awfully trivial, and other awfully complicated. This brought me to his website, where I discovered a stock pile of research papers.

He covers a lot of very interesting concepts, most of it some what over my heard. The bizantine systems are very interesting: the ability to write systems that can react to any type of error. He also goes in to describe a truly parallel garbage collection algorithm, something that would be quite nice in Java.

It's very interesting. Most of the code being written these days, myself included, is written to get it out quickly, and cheaply. The code does what it supposed to do, but is by no means very efficient or bullet proof. It does the work, but is at best a temporary solution. Definately not elegant. Then I get to see all these research papers dealing explicitely with the elegance of programming. It is a nice feeling: raw computer science.

First Entry

2005-05-23T21:34:00.000-07:00

For those of you wondering what Pons Asinorum means, the exact translation is "Bridge of Asses". The phrase comes from Euclid's fifth proposition from book one of elements. The proposition states that the two angles of an isosceles triangle are equal. Euclid meant the phrase as the first test of intelligence and a bridge to the harder problems that follow.

The reason I choose this name is two fold. On the one hand, this blog will be used as a dumping ground of half baked ideas and half baked knowledge. This blog is a means to explore the ideas and knowledge in further detail to perhaps derive something more interesting and useful. So, the first part of the bridge is my own test of intelligence. The second part, is the bridge to something else that hopefully the ideas and knowledge combined will produce.