Greenfield Enterprise Architecture for an IT BU

Let's say you had the power to have a completely greenfield development for an entire Enterprise Architecture for an IT BU - what would it look like?

Well, which IT BU you might ask? Does it really matter?

You need a big database. What kind of enterprise architecture can you have without a database?

Next, we need a bunch of ETL to load data from source systems, because there is always a source system to source from. We can either code it or buy it - doesn't matter which.

And there shall be database performance problems with the data-load.

Next, we need some engine code. Let's call it "The System". And, It shall be in Java. And it shall have memory problems regardless of whether it's 32bit or 64bit or how much memory you allocate to it.

And next, there shall be a web-ui, for which, countless hours will be spent on things that will never be used. And there shall be an excel download link on every page - because that's the only feature the users seem to care about.

And there shall be data quality problems, and performance problems, and scalability problems, and extensibility, and bulk upload requirements, and usability issues, and technical debt, and The Business will cry out for Salvation.

Surely, there must be another way.

I have spent the last 5 years as an Enterprise Architect at a Tier 1 Investment Bank designing systems that solve Big and Expensive problems. There are a few observations I would like to make for things that worked and things that didn't.

1. User's are smart and love Excel and are better at coding than the H1B coder you got for the 2 for 1 sale from a body shop.
2. Build functions not systems - and expose your functions to your users. Allow the users to create a managed ecosystem around the functionality.
3. Make sure your functions work natively in Excel - think COM C# library.
4. Use elastic infrastructure like HDFS, and compute clouds, and data-grids, etc... - don't build dedicated systems, build services that have clean inputs and outputs and can run on scalable hardware like compute clouds.
5. The database and ETL has been my Achilles heel. The database schema is too rigid for the fast pass of change. Alternatively, the rigidness is required given how central the data is too everything. I have yet to really embrace the nosql movement, given the lack of ACID qualities. There are some promising developments in the form of Impala, which is a closer to a pure MPP database running on commodity elastic hardware. Perhaps an interim medium can be found between a strict data-model of a traditional database and a loose schema of a nosql database.

To be continued....

Cross Language

Java to C to C++/CLI to C# to Java via IKVM to HTTP Rest to Java Servlet


and it all works!!!

Platform Building

As architects, the most natural thing to do is to build systems. Building a system is a lot like building a sky-scraper. There is always a foundation and strong central core. Stuff around the core is a frame, and everything else is just eye candy - windows, walls, carpets, furniture, etc... The interesting distinction is that construction architects don't erect sky scrappers in the middle of Sahara, while system architects do. In all cases, all systems start in a vaccum and proceed from there. Once the system is built, the most rudimentary of neurons is spent on deciding how to integrate the system into an existing eco environment. In most cases, this means that the author slaps together a messaging bus or maybe some kinda of REST, WS, or maybe something new fan dangled like proto-buffers, etc... The system is still a big giant monolith with some little shitty input and output.

Now what if we decided to build a platform instead of a system - what does that mean:

1. User interface split into a Bloomberg like approach - I can jump to any screen directly by entering the right tag and parameters
2. Analytic & Calculators - segregated engines that can be called as libraries or services
3. Commons Core - all the technical generic plumbing libraries and services (calendar, security, etc...)
4. Modular controllers - business logic segregated by type
5. Physical services - compute farm, data storage, distributed locking schemes like Zookeeper, etc...

But you also need orchestration of all of this:
1. Workflows, schedulers, coordinators

Business Language

I've been toying with a new type of architecture - or more specifically another approach to solving an age old problem of ad-hoc reporting.

The question usually goes something like this:

We need a flexible, ad-hoc reporting solution which will allow us to do what ever we want whenever we want. It has to be dynamic and easy to use, and should give us the complete power to answer any business question. But we also want the ability to create presentable formatted reports and link together any type of data we so chose.

The answers vary from business vendors like SAP Business Objects or Oracle Business Intelligence tools, to smaller vendors, to various products pieced together like SSAS cubes with SSRS reporting. But, in all cases, the solution is always lacking that flexible and dynamic nature. With all tools, there is always a heavy IT presence. Some of the tools attempt to simplify this by creating an abstraction - such as the Business objects universe concept. This is effectively a meta-model around a physical representation. But this is very limiting as it restricts the use-case to a relational structure. Cubes are a very cool technology but also very limiting and the truth of the matter is that the users care about small data sets such as current days data or even a small subset of current days data - and in a number of cases the cube becomes too burdensome.

So the alternative approach was to actually give a new language to the business user directly. Goldman did something like this with their slang language. Imagine a language which is simple enough and descriptive enough for the user's to use and lacks all the very technical details of an actual technical language like Java or C#. We should accept that user's already do a fair amount of programming with the tools at hand such as VBA, and combination of Excel/Access. At times, it is very impressive some of the things these guys produce and at other times it is quite scary.

So, let's say there is a functional language that allows the user to manipulate the various available artifacts - i.e. back-end systems. Some functions retrieve data based on the supplied parameters and other functions can perform actions like calculations or data changes. Before you know it - you've provided a set of very powerful tools and what emerges is that the users start tinkering with the functions to build more dynamic and abstract features that were not previously imagined. What I came to realize is that revolutions are slow things that require tinkering, and at some point, something emerges which could not at all be previously imagined but once created makes perfect sense and is the most natural. The user's need tools to tinker, and that tinkering actually creates tremendous value and innovation. Innovation drives investment and growth, etc...

So, back to the language. The language should of course work within Excel, but should not be restricted to excel. Ideally, we bridge the language with perhaps a more functional cousin like F# and create a natural set of functions with a pure functional language. At the heart of this language is a data set - to borrow from any database driver - a TDS class - a tabular data set. Let's say the language only understands data sets not objects. Everything is a table of data or perhaps, a table of tables. Some functions create tables and some functions manipulate tables. It is also interesting to see how the roles of systems and enterprise architectures change. Systems become components which are forced to expose "functions" which are manipulated from this functional language. This in turn forces systems to become more dynamic loosely coupled modules as it's not always known how an external function may want to manipulate the components of a given system. Looked broader at a higher level, the systems start to make up an overall platform - with the functional language acting as a glue that binds it all.

I think this approach gives the most flexibility - coupled with some of the more traditional options like cubes and canned reporting - perhaps even a vendor product. The approach of course is more technical and requires some learning but it strengthens the bond between user and system and makes a user an investor as they are now in affect developing the system - or platform. It is a wonderful thing to create something that you're proud of that adds value - that feeling can be given to the user to strengthen the bond to the platform and the overall enterprise architecture.

0% unemployment

I'd like to once again deviate from my regular technological themes and instead discuss philosophy or rather some shade of philosophic economics. I was watching CSPAN the other night and a show came on discussing poverty in America. There was a panel of speakers including heavy weights in the topic along with some celebrity personalities sprinkled in. Some of the speakers espoused what is almost close to pure socialism that it was the responsibility, nay a requirement for corporations to hire people that they don't need and, which in turn, will create jobs for the poor people, and will help them get out of poverty. Now what the corporations are doing instead is hording the cash and rewarding their management ranks for driving efficiency - i.e. doing more with less people.

I am in technology and in laymen terms that means that my job is to build systems, which create automation, which in turn, create efficiencies or in other wards reduces headcount. I am even horrified when management hires too many people for one project as it destroys efficiency, makes the process tremendously burdensome and bureaucratic, and this eventually leads to either a failed system or a very poor system. These projects always cost more and deliver much less.

So - how do you achieve 100% employment - or more specifically, some higher number like 95% as in every society there will be some group of people which either cannot or won't work. In America, the official unemployment rate is just under 10%, but there is probably another 5% or more of unemployed which do not collect unemployment benefits. This means America has 85% employment - looked from this angle, that's not bad at all. But, it would be important to understand the distribution of the income across the 85%. Is the income completely skewed - i.e is it a nice bell curve or a steep hill with a cliff drop off.

If the bell curve is nice and even that would mean that we have a strong middle class and no fear of a revolution. On the other hand, if the curve is more of a cliff, then there will be tremors produced from the mountain, which could erupt and bring down everything around it. The problem is that individuals are rational, but people are not. Our political process is also looking more and more weak with the least worthy striving for notice. Politics and education should be the epitome of the most intellectual of us all - rather finance is where we spend our energies.

I am confusing various topics - but they are all related. Our wealth and the most brilliant talent seems to be focusing either on financial engineering or entertainment be it social type websites or some form of media.

It is very easy to get into a negative feedback loop and it is not easy to get out of it as the decisions and choices are counter intuitive and not at all obvious.

One should always better oneself - and perhaps the forces of good and right will prevail over the forces of ignorant.

This is not the most fluid of posts as I don't yet fully understand how all the pieces fit together.

Serializers

https://github.com/eishay/jvm-serializers/wiki/

http://code.google.com/p/thrift-protobuf-compare/wiki/Benchmarking

Zion Business Intelligence

How about a hypothetical scenario -

You are an Enterprise Architect in a Global Investment Bank and tasked with finding a solution to the data fragmentation problem.

And this is how it happened -
This particular bank has many data files flowing whether which way between various groups in the organization. Product controllers take a feed from front office systems, and market risk another feed, and credit analysts take a third feed, except the 3 feeds are generated differently, and have slightly different points of view, and in some cases, even different trade populations. But then, someone comes along and says they need to see 3 numbers - 1 from market risk, 1 from credit analysts, and 1 from product controllers side by side. And you say - that's impossible because it will take months just to analyze whether the data populations are the same, and even if there were, the numbers are produced at different levels, and even if they weren't, there is no way to show them on the same report. And even if there was, this is a 6 month project, and who is going to pay for it. And then they say, "aren't you an architect", and you hesitantly say "yes", and then they say "so go fix it."

Rabbit Hole
So, how to fix it. Core data primarily generates from front office systems and then flows through the rest of the organization. You can think of it as streams of water branching off into many smaller sub-streams, and then further branching off, until eventually the stream is to weak to branch of and just sips into the ground.

Well, easy breezy you say. Pervasive BI, data warehousing, Online Analytical Processing, bottom up data-warehousing, top down, bus architecture, centralized architecture, federation, hub and spoke, relational, dimensional, operational stores, data marts, Inmon, Kimball, conformed dimensions, Boyce-Codd normal form, 3NF, .....

We just need to take this mix - shake it up, and we'll get ourselves a fancy enterprise data architecture or perhaps something strong enough, maybe with a cherry - so we can forget the whole thing ever happened.

Blue pill or the red pill
What to do.

Well, Kimball likes the bus architecture, so, let's give that a go. The bus architecture consists of a bottom up or was it top down approach where you basically start off with a bunch of data marts, which then flow into a data warehouse. The data marts are primarily operational store type structures, while the data-warehouse is a pure data-warehouse, star-schema and all.
The problem, of course, is that you effectively already have a ton of little data marts all over the place, which don't conform. Ah, that's the problem, we need to have conformed dimensions. Right, and we do that how exactly? The other problem, of course, is that this is the same data being treated in a slightly different way, maybe with a slightly different trade population or attribute set or granularity or perhaps with a different temporal point of view. Seems awfully wrong to have a bunch of data marts storing the same data, which you then have to reconcile all together.

Right you say, let's go the other way, Inmon likes top down, so, let's create a big data warehouse, which then populates the data marts. So, who is going to build this monstrosity exactly? Well, can't be the individual business groups, because they are not stupid enough to take on a project like this, so, it would have to be some central group away from any particular business line, and close to senior management, 'cause this is going to cost a lot of money. So, the group is created, except they don't know what they are doing - very technical guys, but don't understand the business at all. And if by some miracle they do, they can't keep up with it. If this project actually succeeds, which is highly unlikely, and actually have the right amount of data, which is frankly impossible, it will still fail, because the business moves just too damn fast, and at the end of the day, they will never ever be able to fully understand and own what they are storing. So, after some time, and a whole lot of money, this will be dramatically killed off.

What about federation you say? We just need a magical vendor, and all our problems will disappear. You see this vendor will create an abstraction on top of our asylum, and this way, we will present a clear simple view shielding the end user from the underlying complexity - easy, breezy. I suppose that could work if it was actually possible to build enough complexity into this thing to actually bridge something which is fundamentally diverging and is actually able to perform at the required speed, and is actually supportable. So, let's just kill that idea for now, before we embarrass ourselves much further. Perhaps, if works for you readers out there - if you happen to like this federation thing - check out Composite, Inc - it's all the rage these days.

Red pill
Reality it is then. Well, seems to me we need a new philosophy. Let's call it Zion. There are a few core principles of this theory:

1. Each piece of data has 1 and only 1 owner.
2. The owner is the only one that can change this piece of data.
3. Each piece of data has a natural key and a surrogate key

The distinction between this theory and all the above is that this one says that first you need to understand what data is before you start deciding on how to deal with it. And the first question to answer is who is responsible for it? Responsible is not the same as which IT group owns the data-warehouse, but responsible, as in which business group is responsible for this data. If you need to change it, who best knows how to change it? How to evolve it? What it means?

If you answer this question, now you know who is going to build the data store for this type of data - and this will become the golden source for this data.

To be continued....

Social Networking

There is something interesting occurring in the world. People are exposing the most mundane bits of their life via twitter, friendster, youtube, etc...

What is interesting is where all of this is going. I think the next step will be that teens will record their life continuously and post bits of it for the rest of the world to see.

There are some examples of this already occurring:
http://research.microsoft.com/en-us/projects/mylifebits/

http://qik.com/

It needs to be effortless, continously recording, maybe an addon device behind the ear, connected to an iphone in your pocket.

After something interesting happens, you can then open the iphone, scan back, cut out the relevant section and share it via some social networking site.

Connecting to Sybase IQ 12 from Analysis Services 2008

I've been struggling with this task for awhile now. Since MSSAS 2000, you can only use the .NET provider or the OLE DB provider.

If you have the Sybase 12 client installed, you will have the "Sybase ASE OLE DB Provider". If you also install the Sybase IQ client, you'll additionally have the "Sybase Adaptive Server Anywhere OLE DB Provider 9.0"

On the ODBC side, you should have the "Sybase IQ", "Adaptive Server Anywhere 9.0", and "Sybase ASE ODBC Driver".

1. Create an ODBC connection using "Sybase IQ" data source.
a. Give it a name in the Data source name field in the ODBC tab
b. Provide your username and password in the Login tab.
c. Enter the server field in the Database tab. You'll need to get this information from your DBA. For me, it was <hostname>_<instance name>
d. For the network tab, select only TCP/IP, and enter: host=<host>;port=<port>

2. Open Analysis Services, create a new data source using "Native OLE DB\Sybase Adaptive Server Anywhere OLE DB Provider 9.0".
a. In the server or file name entry put the name of your ODBC data source that you created in step 1.
b. Press Test Connection, everything should work.

3. In the data source views section, create a new data source view given your data source that you just created. It seems importing the tables directly fails with some arcane error.
a. Don't select any tables, just click next, until an empty view is created.
b. You can now create named queries like "select * form <table name>"

Healthcare

Let's ignore the red versus blue and blue versus green and all the other colors of the rainbow. Let's also ignore all the stupid nonsense coming out of news talk shows.

What's at stake seems to be a lot more fundamental.

Is it or is it not the role of Government to protect the life of its citizens. Must the Government protect the "inalienable rights" of man - life, liberty and the pursuit of happiness? Or is such responsibility more fickle and has dependencies on array of variables such as politics, and budgets and costs, and personal gains?

Let's see if we can look at this problem at an even more basic way. Life is fundamentally unpredictable - complexity theory. Tomorrow we can either be hit by the bus, get cancer, both, or neither. In the case of neither, we, humans are conditioned to ignore the perils until either one or both of the above occurs. Now, is it not the role of a stable, evolved society to protect its weak and young? What does it say about our society when we concisely ignore the weak and the young? Are we as a society egotistical, thinking like every other great society before us, that we can do no wrong, strong survive, weak perish, one must accept the law of nature?


Some statistics on the makeup of America, total uninsured by various categories:
http://www.kff.org/uninsured/upload/7451_04_Data_Tables.pdf

http://facts.kff.org/chartbooks/State Variation and Health Reform.pdf

XML/A via Flex

XML/A stands for XML for analysis and is an industry standard for using XML to retrieve data from an OLAP cube.
The spec is available at http://www.xmlforanalysis.com/xmla1.1.doc

<SOAP-ENV:Body>
 <cxmla:ExecuteResponse xmlns:cxmla="urn:schemas-microsoft-com:xml-analysis">
     <cxmla:return>
           <root xmlns="urn:schemas-microsoft-com:xml-analysis:rowset" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:EX="urn:schemas-microsoft-com:xml-analysis:exception"> 
               <xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema" targetNamespace="urn:schemas-microsoft-com:xml-analysis:rowset" xmlns="urn:schemas-microsoft-com:xml-analysis:rowset" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:sql="urn:schemas-microsoft-com:xml-sql" elementFormDefault="qualified">
                   <xsd:complexType name="row">
                       <xsd:sequence>
                           <xsd:element minOccurs="0" name="_x005b_Counterpart_x005d_._x005b_All_x0020_Counterparts_x005d_._x005b_ABN_x0020_AMRO_x0020_Bank_x0020_N.V._x005d_" sql:field="[Counterpart].[All Counterparts].[ABN AMRO Bank N.V.]"/>
                           ...
                       </xsd:sequence>
                  </xsd:complexType>
               </xsd:schema>
               <row>
                   <_x005b_Counterpart_x005d_._x005b_All_x0020_Counterparts_x005d_._x005b_ABN_x0020_AMRO_x0020_Bank_x0020_N.V._x005d_ xsi:type="xsd:double">10</_x005b_Counterpart_x005d_._x005b_All_x0020_Counterparts_x005d_._x005b_ABN_x0020_AMRO_x0020_Bank_x0020_N.V._x005d_>
                   ...
               </row>
           </root>
      </cxmla:return>
 </cxmla:ExecuteResponse>
</SOAP-ENV:Body>

This is a sample response. In this case, I've used Pentaho Mondrian as the cube provider.
This XML can be rather easily walked with Adobe Flex.

var message:XML = ...
var soapEnv:Namespace = message.namespace("SOAP-ENV");
var cxmla:Namespace = new Namespace("cxmla", "urn:schemas-microsoft-com:xml-analysis");
message.addNamespace(cxmla);

var xsd:Namespace = new Namespace("xsd", "http://www.w3.org/2001/XMLSchema");
message.addNamespace(xsd);
        
var sql:Namespace = new Namespace("sql", "urn:schemas-microsoft-com:xml-sql");
message.addNamespace(sql);
    
var body:XMLList = message.soapEnv::Body;
var executeResponse:XMLList = body.cxmla::ExecuteResponse;
var schema:XMLList = executeResponse..xsd::schema;
var complexType:XMLList = schema.xsd::complexType.(@name="row")
var elements:XMLList = complexType.xsd::sequence.xsd::element;

The above will return a XMLList of elements within the complex type row.

A slight change, will produce a list of actual data rows.

var rows:XMLList = executeResponse..rowset::row;

Problem's for your CIO

1. Business does not know itself.
Every organization struggles with the problem of data. Either it's too much data, or not enough data.

In the case of too much data, the analysts and the IT teams struggle to make sense from the barrage of data. A lot of times, IT is forced to archive data, and by so remove it from analysts access. In other cases, large amounts of data is simply stored, but no viable reporting is available, or only a small segment is reported on.

In the case of not enough data, local analysts are unable to get complete pictures because data is stored in multiple systems, and a lot of times, is localized to regions. This means that organizations struggle to get complete pictures. Imagine what that means for financial risk: What is my exposure to Bear Sterns? I don't know, it will take a few days to compile the report from various places. Don't believe me, this was the case at Lehman, and is the case at a Global Asiatic Bank. I would also bet, is the case, in almost very other institution.

2. Business doesn't trust IT to deliver.
The business simply does not not trust IT to deliver the solutions it needs. When solutions are delivered, they become large monolithic entities locking the organization in, and becoming an expensive constant expense. Any change to the systems, becomes an expense at times larger than the cost to build the system in the first place. In other times, IT simply looks like a dog chasing it's own tail. They move very fast, and have short time-lines, but the end products are not even remotely close to what the business wanted. IT seems to always have an excuse. I was once in a 3 hour meeting. The business analyst wanted account numbers. He wanted a system where he can debit one account and credit another account. For that, he needed account numbers, and amounts within those accounts. The developer for the system was trying to explain that the analyst didn't need accounts, but only had to enter the transactions and attributes of the money movement. The system can then aggregate those transactions in what ever way the analyst wanted. You see, the system was designed generically, and adding account numbers would intrude on that design. And, so they went, back and forth. Neither understanding the other. The end result, nothing. No account numbers. It was deemed that this feature wasn't go live critical, and can be addressed in the next phase.

The fault is actually neither IT nor the business. The problem is more basic. The business is very agile. It's able to move quickly and easily. If you've ever read a legal document or saw financial models, you'll understand. They can be very complex and very nuanced. Most technical architectures are not build to be nuanced. They are built to solve concrete Boolean problems. IT wants the problem defined in concrete terms, but the business is only able to articulate it's current understanding of the world. Unfortunately, tomorrow it will be a new understanding. And so, IT builds a monolith, because, that's what it knows how to build, the business changes, and the organization suffers.

This really saddens me. Technology is capable of solving so many problems, but instead it's relegated to addressing the most basic of operational problems, and even there, we struggle. Consider all the technical possibilities, neural networks, expert systems, machine learning, data mining ....

3. Business data is not clean.
So much of data is simply bad. Imagine how complex some of the systems are, and then imagine the possibility of an error. An error that doesn't generate an exception, perhaps, rounding, perhaps, a logic error, perhaps, an unforseen condition. The output data becomes bad, and so it flows through the system. A lot of times, enterprise architectures don't even do basic reconcilliation. Reconcilliaton requires active design, time and thought. Most architectures are organically produced, and are feature driven rather than any thoughtful design. The end result is frankenstein architectures and garbage data.

4. There are a lot of manual processes.
How many times have you created a new system, which actually creates more manual processes than it removes. New systems sometimes require users to enter data in multiple places, verify multiple places, etc... Some firms have massive operational groups, hundreds of people. Their sole job is to do what systems fail to do. Enter data in multiple places, reconcile, data entry, data messaging, normalization, etc... This is a horrible model. It's extremely error prone, not too mention expensive.

5. Heavy reliance on people.
I believe that machines and people have their roles. Unfortunately, a lot of jobs done by people to day, should be done by machines. Machines and people need to find a harmony. Some things we do very well, other things, machines. Sifting through large amounts of data should be the role of machines. Alerting us to unforseen circumstences, should be machines. Allowing us access to data, should be the role of machines.

6. Presentation
So few firms give through to the presentation tier. In most cases, it's an after thought. Some UI's are desktop, other's web, other's, something else. The end user is required to memorize what features are available where, what report has what, etc... God forbid, we standardize and unify the disparate systems. The user should have a single place to go, a single way to do something. The learning curve to learn how to use everything should either not exist, or be extremely small.

Lehman Brothers

The last 4 months have been a crazy whirlwind. I was an employee of Lehman Brothers during it's demise. For those who know, that weekend went fast, without a single doubt, each of us expected a normal resolution; maybe a industry consortium bailout, maybe the Fed will extend a loan, maybe a last minute buyer. None of us expected a bankruptcy filing. That Monday felt like being shell shocked. Everything you worked for, all your hard work, gone, abandoned. All those meetings about tiny little minutia's of detail. All that time, coming up with the perfect design, the clean integration, the cleanest build file, the good interface - puff.

Some of the guys have worked their whole lives at the Firm. Think about it, you spend the last 20 years of your life building something. Sometimes you work late, sometimes even on weekends. You meet tough deadlines, you build out the system, you have projects, and Jira items, and future ideas for improvements, and then one Sunday evening, it all disappears in a single 20 second news-clip.

What can I say about that Monday, a lot of guys were afraid. Single earners, mortgage, kids, car payments, private school, dance lessons, ... It was never a problem before, the Firm was known for it's bonuses. The irony of having more money, is that your expenses rise proportionately. Some of the guys were hit very hard. A lot of the bonus is paid in Lehman Stock, part of the 401K is in Lehman, the firm also promoted personal investing in the stock. All in all, if you were with the firm for 20 years, you have a major percent of your worth tied with the company, and then it's gone.

The next few weeks gyrated between moments of team comradery and private lonely introspection. Every person came to work on time and stayed for the full day. Most of the guys, myself included, kept to the Lehman dress code of full business attire. I think it was a out of respect for the Firm, or perhaps, just routine.

Barclays purchase of the firm provided a rare glimmer of hope, a possibility of normality. It wasn't so much as having your job saved, which is of course very important, as a possibility of saving what you've built. Someone using what you've worked so hard to create. As time dragged on, it became clear that Barclays had no intention of taking our system, so, a glimmer of hope slowly changed into despair with a dash of anger. Barclays, of course, wound up laying off most of us. The system I've worked on was trashed.

Today, I am at another company, assigned the task of comparing Lehman system's to my new employer systems. This is how Barclay's employees must have felt when they found out their company was buying Lehman. It's us or them. And, so I do what is requested of me. I criticize and destroy the very systems I promoted, just a few months ago.

I have the deepest respect for the Firm. I have never worked for a company like that. They really tried to create something more then the sum of themselves. I shall miss it.

Intentional Software

After working on the Drools Flex Editor, I started thinking a lot more of what it would take for it to be usable by non-developers, i.e. domain experts. This narrative of thought brought me to heavyweights such as Charles Simonyi, founder of Intentional Software and
Gregor Kiczales, founder of AOP, latest paper. It also led me to a lot of other concepts such as the law of leaky abstractions, history of software, Lagom Process, along with the more obtuse concepts such as the omega number, and people like Gregory Chaitin.

And of course, what would our industry be without it's acronyms: MDA, DSL, BPM, UML, DDD, SOA. There are plenty of others, but I think I've made my point.

That's a lot of information to digest especially if you looked up Chaitin, which would have led you to our Founding Fathers: Leibniz , Turing, and Godel. Now ask your self: what kind of a thought narrative can take a person from Adobe Flex to Godel. It would almost be funny.

Now, let's step back for a moment. A lot of people argue that software development should be reduced to visual tools. A counter argument can be that mathematicians do not use visual tools to draw up their equations. They use blackboards and chalk, the most primitive of tools. Excel has been praised as the most successful intent based system. But, if you look at it, it's not visual at all. At best, it's a basic grid, with cell co-ordinates, and a blank text input box allowing manipulation of cells. Another argument is that business people are somehow not smart enough to program. It takes a special kind of mind to generate code. The fallacy with this statement is that business people already code, just not in the typical "tech" way, but rather in their own domain. Their interaction with excel, the domain expertise, the manipulation of that domain expertise - can all be considered coding. They manipulate their symbols to achieve their goals. The only thing separating technology developers and domain experts is which domain they are experts in.

There was a quote from Charles Simonyi that went something like this: If we don't expect business people to learn how to code, why do we expect coders to learn the business. Each path is extremely inefficient and is rife problems. So, instead, let's allow each group to focus on what they do best. Developers should stick to technology, business people should stick to business.

So, we have established that a business person is capable of performing some form of "development" to encode their domain expertise into a set of steps, "their intent". It is also probably safe to assume that a business person understands concepts like if-then-else, for-each and standard algebra. It is also safe to assume that they know nothing of JSP, Servlets, JMS, EJB, transactions, XA, JDBC, SQL, Java, Class, public/private, encapsulation, polymorphism, design pattern, singleton, facade, heap, stack, binary search tree, NP-complete, and on, and on, and on, .... So, where does this leave us? I think it means that software development stops being the pure domain of developers, and instead is split between developers and business people.

If we look at a typical business system, we can see that it has inputs(JMS, GUI, etc...), a concrete data representation model in the form of a database schema, complex output in the form of varied reports, processes that criss-cross the system that have some triggers such as external events (JMS, Schedule, User, etc...) . There is also business logic in the form of calculations, business steps, if-blocks, etc... sprinkled through the system. Some of it lives embedded in the report logic, others in the processes, and some, perhaps, even implicit in the data storage or data format.

I think we can start to take steps to separate the domains. Process flows attempt to separate the process logic from the system logic. Web Services attempts to expose the individual services and by so reduce the hard linking between services. Business Intelligence is attempting to expose the data to the users and allow ad-hoc manipulation. Proliferation of domain specific languages, online compilers, rule engines is a sign of the desire to separate the system from the business rules. Hibernate, JDO, etc... are attempting to isolate the system from the underlying data stores, and map out the data definitions. Ontology's are attempting to bridge the interaction of a human defined relationships and a system. Mashups - i.e. http://www.programmableweb.com/, Yahoo Pipes, are yet more examples of technical concepts being exposed to non-technical people. All these things, in my opinion, are converging on the same topic of intentional programming.

Alpha Release of Drools Flex Editor - 0.5

Orange Mile is proud to announce the long awaited alpha release of a Drools Rule Editor in Flex.

http://code.google.com/p/drools-flex-editor/

The current release includes all the Flex pieces without the rather basic server side code for rule compilation, and code completion. This will be available in the final 1.0 release.

This is the beginning of having rich enough components available within the system that can allow the user/admin to directly manipulate the business rules without having the long development cycle.

Orange Mile is Expanding

We are proud to welcome Venu to the Orange Mile Team. Venu will also act as a contributing author to the blog.

Although, Orange Mile started as a small math with a single avout, it has since expanded, grown and matured.

We have become a devout following. At times, looked down upon by the saecular world, but always, pursuing our dreams.

Entry written in the style of Anathem.

Orange Mile Security Release - 1.1

Orange Mile is proud to release version 1.1 of Dynamic Rule Based Security.

The new features include:
1. A complete example based on Spring Security - see orangemile-security-test.war
2. isGranted JSTL Tag

http://code.google.com/p/dynamic-rule-security

Doom and Gloom and the Economy

Although, this is a tech blog, I strongly believe that a good developer must be an Erudite. Given this, I feel justified in writing about the economy in this blog.

We are living through historic times. Mid last year, I started getting very scared and started blogging about randomness and the economy.
http://orangemile.blogspot.com/2007/05/black-swan.html
http://orangemile.blogspot.com/2007/07/supply-of-money.html

You will notice that in the Supply of Money entry, I actually wrote about the likely hood of the collapse of world economy due to the unsustainable supply of fiat money.

What's interesting about those entries and that time period is why did my mind shift to the arcane topics of money supplies, carry trades, fiat money, when the blog entries before and after clearly deal with the arcane topics of technology. Perhaps, my subconscious started to pick up on the feelings of uneasiness in the global market; hiccups if you will. I can't possibly attribute those entries to knowledge, because I am simply not qualified to speak of money supplies, fiat currency, and carry trades.

So, what is happening today is a global loss of confidence. What is interesting is that companies, specifically, banks are hoarding cash rather then people. I would argue that if people started hoarding cash than we're all doomed. The global economy would come to a scretching halt. Chinese economy will collapse, probably throwing that country into either Marshall law or revolution. America and Europe will fall into severe and prolong depression taking the rest of the civilized world with it. Africa will fall into an even lower level of sustainability with probably wide ranging civil wars due to lack of food and an acute demand for natural resources such as diamonds, gold, and oil. India's economy will also take a severe beating, but I think they will remain a loose democracy. If they position themselves well, they may end up being the next superpower.

Right now, the US government is printing money at an ever faster clip, giving it away, almost for free, and nationalizing large areas of the financial industry. Money is flooding the global economy. What's interesting is that we are actually in the period where money is actually disappearing. As the perceived value of assets fall, money disappears. The US Government then tries to fill in the gap of lost money, by providing more money to the institutions whose wealth disappeared; hoping against all hope that the newly provided money will be used to create more money by the institutions. Let's recap how money gets created. A person decided to do something on credit, let's say by a house. They go to the bank and say give me 300k to buy a house. The banks gives you the money, and you go buy a house. The thing is that the 300k is actually some other depositor's money. Money the bank doesn't actually have. What's happening now is that the bank thought it had 300k loan asset. But instead, the 300k is really only a 200k asset. This means that if you default, the bank looses 100k of someone Else's money. If enough loans do this, the bank won't be able to cover the loses, confidence in the bank erodes, people start to retrieve their deposits, and of course, after some interval, the bank simply runs out of money to give out to depositors. This is why the FDIC was created. This is a standard Ponzi scheme. In other words, if the asset side of the bank balance sheet starts to reduce, they will reduce the amount of new loans they can give out, and by so, reduce lending, which will probably drive the interest rates up because there are less institutions lending. This actually means the opposite of what I said earlier, money isn't destroyed, the rate of creation just reduces. The bank industry is structured as a very calibrated entity, with a minor hiccup in cash flows or perceived cash flows destabilizing the entire industry. The Fed is trying to erase the perceived losses from the Bank's balance sheet, and by so, start up the loan process. Another interesting thing is how many industries rely on having a continuous supply of new loans. Imagine a ponzi scheme applied to the car industry. If Ford borrows money to pay it workers, hoping that in the future it will sell enough cars to pay back the loan, except it doesn't, so it borrows more to keep going. At some point it actually needs to borrow from Person A to pay Person B, and on and on. This should end at some point when there is no-one else willing to lend to the said company. But, unfortunately, for most companies there is always someone willing to lend. This is partly due to the obscurity/opaqueness of the financial industry. Now, our current scenario, where the said company can't get a loan not because of the financial condition of the company, but because of the financial condition of the lender; all lenders.

What the Fed is trying to do now is fight the deflationary path. Money is becoming a scares resource, not because there is not enough of it, but because banks are hoarding it. A lot of people are also talking about hyper-inflationary model. I don't see this happening, even if I believed it for awhile. In a hyper-inflationary model, you have too much money. This is unlikely because the Fed can always mop up the money supply; and because the world is dollar denominated, at least for the foreseeable future. I think what is more likely is nationalization of a number of areas, and banks, a significant increase of money available to the banks, guarantee of bank assets, forced reduction of inter-bank rate, and a forced reduction of the interest rate payed by home owners.

Interesting reads:
http://en.wikipedia.org/wiki/Fractional_reserve_banking
http://en.wikipedia.org/wiki/Credit_default_swap

There is another animal that hasn't received much news: credit default swaps. This instrument is a form of insurance against default. The problem is that this is not based on anything, and the current amount of outstanding CDS is a few times larger than all money combined ever produced. This means that the government needs to do its darnest to make sure those CDS contracts never come due, because if they do, all financial institutions will file for bankruptcy, governments will default, end of the world, etc....

So, what is the government to do. Money must be made cheaper to a point where it is basically free. This will allow the banks to start to give out loans, this would spur the market for re-financing, which should save some borrowers. At the same time, the government will probably start with the regulation. We will see a period of a slow down - recession, in which the disaster of the day, will become a distant memory, and we will start up with the next bubble, maybe energy, maybe the housing sector again to a lesser form, although, this will probably be regulated to the gills. My guess energy or commodities. But the bubble won't start until the people regain confidence, which will take a few years.

I repeat, the Fed must make money cheap. After a recovery, they will attempt to make money more expensive again to starve off another bubble, but they will tread very lightly to avoid any more panic. This means that rates will stay cheap, or only very gradually over a long interval will start to go up.

Of course, there is another animal in this picture: US treasury bonds backed by our taxes. I don't fully understand this animal and it's relationship to the money supply, but hopefully, in the next few blog entries...

Release of Microsoft Analysis Services 2005 Automation SDK

I am proud to announce the long awaited release of the Microsoft Analysis Services 2005 Automation SDK.

http://code.google.com/p/mssas-automation/

The library allows a java developer to automate the creation and modification of a MSSAS 2005 cube. The design consists of codifying most of the XMLA 1.1 specification into java pojo's via Jibx binding framework. On top of this core library, it then becomes trivial to codify specific design patterns or utilities to automate or speed up the creation/modification of a cube.

How not to be a turkey - a dead turkey!

The idea is to build a little web app that will scan the common news sources nightly, and compile a score for different words on how negative or positive the topic is described. For example, regarding the economy, the system should pick up speeches from the Fed, congress discussions, etc... The idea behind all this is from the Black Swan Book. The theory goes that the night before Thanksgiving, the turkey should have the highest confidence in the goodness of humans.

To achieve this, I will need an NLP mood analyzer, or in other words, Sentiment Analysis. Some open source tools to accomplish this are:

NPL Libraries:

• http://www.opencalais.com/ (Reuters Web Service)
• RapidMiner
• JavaNPL
• LingPipe
• Jane16
• http://garraf.epsevg.upc.es/freeling

Knowledge Understanding

• http://commons.media.mit.edu/en/
• http://www.opencyc.org/
• http://wordnet.princeton.edu/
• http://framenet.icsi.berkeley.edu/

News Sources

• http://cookbook.daylife.com/
• http://thomas.loc.gov/home/c110query.html (Government Enrolled Bills)

Ekman's research on universal facial expressions
[happy, sad, anger, fear, disgust, surprise]

Frustration – Repetition of low-magnitude anger
Relief – Fear followed by happy
Horror – Sudden high-magnitude fear
Contentment – Persistent low-level happy