Scene 1

Scene 1:
The following conversation took place between two co-workers over an instant messaging product called Sparc.
The setting is a corporate office with many cubicles. The two co-workers have been asked to design
an enterprise, grid enabled architecture. They received a single 8 by 11 Visio diagram of the architecture.
They were also directed to leverage a FpML (financial markup language) as a messaging protocol between enterprise
systems.

Ben is an existentialist with a bend on fatalism. Mike is generally an optimist, unless Ben gets too fatalistic.

[4:45 PM] Ben: Boss wants to discuss FpMl
[4:52 PM] Mike: As long as we don't have to use it internally ....
[5:00 PM] Ben: I love it when the architecture is dictated from above. It makes designing so much easier.
[5:04 PM] Mike: And simpler, too. The whole system fits into a picture with a few boxes in it
[5:05 PM] Ben: it's pretty cool.
[5:05 PM] Ben: What's wrong with using FpML internally? You suck!
[5:06 PM] Ben: It's a nice intermediary format that allows system to communicate in a well defined language. Honestly Mike, what kind of an architect do you call yourself?
[5:06 PM] Mike: Hey, we've got enough on our plate writing our own database, our own operating system and our own programming language. I just don't need to have to use FpML as well as all that. I think it could jeopardize the entire project!
[5:08 PM] Ben: come on, that's crap. We can introduce this conversion in our custom db level, or even add it natively into our custom language.
[5:08 PM] Ben: think about it seamless integration with FpML, beautiful. Too bad FpML only covers derivatives, haha
[5:09 PM] Ben: I am sure we can work through that. We just need to work with the FpML working group to add a few parts to their spec.
[5:10 PM] Ben: 10-4?
[5:11 PM] Mike: Yeah, OK, it's taking me a while to think of a witty reply. 10-4
[5:11 PM] Ben: sorry to rush you, take your time. I just thought you were ignoring me because you are working.
[5:13 PM] Mike: Does FpML support images? In case we need to attach screenshots when we're reporting errors in market data?
[5:14 PM] Ben: Not yet, but I think we should bring this up when we discuss with them about expanding their specification to support other products.
[5:15 PM] Mike: I think whatever solution we go with, it's vital that we can scavenge unused cycles from people's mobile phones
[5:15 PM] Ben: and PDA's
[5:16 PM] Mike: and pacemakers
[5:16 PM] Ben: and watches
[5:16 PM] Mike: and elevators
[5:16 PM] Ben: maybe we can work something out where we can use the employee’s home machines.
[5:17 PM] Mike: Or people with Bluetooth devices in their briefcases as they wander past the building
[5:17 PM] Ben: good thinking, what about fax machines?
[5:17 PM] Mike: I wouldn't like to see any proposal signed off until we've really considered all these factors
[5:18 PM] Ben: I am glad at least you and me are on the same page.
[5:18 PM] Ben: We need to write up a document, someone will sign off, and then we can proceed with the development.
[5:19 PM] Ben: I think this conversation is sufficient as design.
[5:19 PM] Mike: Especially once we've deleted the Sparc logs
[5:19 PM] Ben: man, if someone has a sniffer, we're doomed.
[5:20 PM] Mike: That would be sad - especially as you introduced this product into CompanyX
[5:21 PM] Ben:
[5:21 PM] Mike: Do I have to 10-4 the smileys ?
[5:21 PM] Ben: no worries, I believe encryption is on.
[5:22 PM] Ben: no, don't worry about the smileys
[5:24 PM] Mike: I don't think we should restrict ourselves to FpML, either. We should have a meta-markup framework where we can just plug-in any standard that comes along - in case we need to support fPml or FPml or fPmL later on
[5:24 PM] Ben: I love it. Consider it added to the spec.
[5:24 PM] Mike: The spec which I hope is written in specML ?
[5:25 PM] Ben: should we look into whether we can leverage specML along side FpML as the messaging protocol?
[5:25 PM] Mike: Absolutely
[5:26 PM] Ben: http://www.mozilla.org/rhino/
[5:26 PM] Ben: I think we should use rhino tool to build out entire framework
[5:26 PM] Ben: think about it, we can release partial code, no reason to compile.
[5:27 PM] Mike: I've used Rhino before (indirectly) - it's built into JWebUnit
[5:27 PM] Ben: so, what do you think of using it as our core language?
[5:30 PM] Mike: Might be a bit low-level. I want something high-level that maps 4 boxes on a diagram into a fully built-out, productionised, resilient, performant, scaleable, internationalised system that runs on everything from a supercomputer to a Beowulf cluster to a digital watch.
[5:32 PM] Ben: do you have a copy of our entire conversation; I think it would make a wonderful blog entry.

Commodity, Speed of Light, EAB

This post was actually going to be about technology commodity. I actually even wrote a part of it. I had all kinds of things in there - a definition of the word commodity from the wikipedia, a reference to Karl Marx, processing grid, service grid, data grid, etc... It was going to be a pretty good post before I erased it.

What the hell? Well, I had no point, I was just writing the obvious. Let me start over. Any enterprise architecture is going to be distributed, but that's not enough. Systems need to communicate and share data. Some systems may provide a service to other systems. Some systems may be in-charge of routing messages. Other systems may be in charge of doing calculations, other systems provide auxiliary services like calendar, or caching. The point is that a whole lot of systems are going to be communicating. In fact, in some cases, that communication will be very heavy and may become a liability. An Enterprise Architecture Bottleneck(EAB). You gotta love acronyms. They make everything sound so much more impressive.

In order to reduce EAB, your system will need to reduce the amount of data being transferred, figure out a faster transfer method, go faster than the speed of light, or all of the above. For the sake of simplicity, let's assume the last point is currently not feasible. For the second item, you can buy a bigger pipe, but you are still stuck to a certain latency. The cost to transfer a bit from NY to London will always be bound to the speed of light. So, can the system reduce the amount transferred. I think it's possible if the system is aware of the data patterns and can remove any unnecessary or redundant information. For example, let's say me and you are carrying a conversation. Certain things are obvious, other things can be deduced by you without me saying anything. For other things, I may only need to say a few things for you to understand much more, and in other cases, you may already know certain things, because I've already mentioned them. What does this all mean? The sending system will need to analize the data stream and learn to reduce the load. Of course, this assumes that the sender and receiver have agreed on some transfer protocol.

Well, this post is still pretty obvious, but maybe a bit more interesting than another tirade on processing grids.

Business Process Modeling

I was discussing a system process with a business client, and during the conversation it hit me that the client has no idea what the business process is. At some points during the development of the system, a business specification was written, perhaps even by this same person, but overtime, the system evolved, and the process forgotten. Now, my business client has no idea what the system is actually doing. To him, the system is a black box, one big scary unknown. He has a general idea of what it does - it takes data from some place, does some filtering and calculation, populates a few tables, does some more calculations and filtering and eventually generates some results. The business analyst is then charged with the verification of the results - the irony is that the business analyst isn't completely sure of how the number was generated. He has a certain level of trust, but overtime that trust erodes and he is left feeling uneasy and unsure.

Later that day, I sat in a meeting where a discussion was taking place about a new business system. It was a usual meeting, many people spoke about many things, most of the conversations strayed far from the actual deliverable or even reality, nothing was agreed, a few people tried to cock fight each other, and, overall, the meeting reversed progress - usual meeting. Anyways, during the meeting, specifically during one of the cock fights, one of the analysts spoke up and said something very profound and interesting: this was followed by a cock block, but that's a different story. So this interesting statement was that he believed that quality analysis testing should not end when the system goes to production but should be an ongoing part of the system. He believed that it was important to have a stable validation layer in the system in order to provide basic sanity checks that the system is performing as expected in an endless parade of changing data. My team-members rose up in anger against him, some claimed he was a heretic, others threatened ex-communication. I sat silent listening and wondering.

Each system is basically a workflow. Once you remove some of the techy parts, you end up with a business process. In fact, at some point this system was a very clean visio diagram. Each box was then blown up into a class diagram, and then some crack smoking code monkey (developer) defecated all over it - an enterprise system is born. This workflow is then overlaid with data. The workflow reacts differently to different pieces of data, but its still functionally a flow - actually more of a graph. The graph is a mix of generalized technical aspects and business logic. The problem these days is that the business logic is sprinkled all over the system making it very hard to re-create exactly what happened.

So, I wonder if it would be possible to overlay an actual system with a meta-system. Would it be possible to create a set of, let's say annotations, to add along side code and possible some additional hooks to allow another system to walk the system code to generate the graph, and overlay the graph with the business documentation - sprinkled throughout the code. The end result can be a self-documenting system. No, I am not talking about javadoc, or external specification. I am talking about a tool for the business user to verify what a given system is doing. Because the documentation and the code are living side by side, perhaps are even the same thing, the business user can be confident in what they are seeing.

The second part is that a lot of data centric systems live and die by the data they are receiving. Garbage in, garbage out they say. Well, I am not quite sure this statement needs to be true. After a long deep thought, I agreed with the business analyst and took a stand to support him. I think he is right, QA should not end once the system is in production. Each system should be built to be in a constant state of testing itself. The point isn't to test the code, the point is to test the data. The data is the most important thing. As developers and architects we treat data as a second citizen. What comes in to the system should be checked, what happens in the system should be checked, and what comes out of the system should be checked. It would help if the checks are a hypothesis test. The analyst proposed having a parallel testing dataset. He figured that a constant check against a constant may provide a basic sanity check or at least maybe raise some red flags if the data is too far from the norm. Of course, this type of test is context specific, but I think the basic principle has value. Data isn't just data, it's the most important thing. When the business analyst receives the end result, and the end result is wrong, the analyst spends hours trying to narrow down what went wrong. Sometimes the problem is the inputs, sometimes the problem is the business logic, other times, he just doesn't know.

I wanted to get this post out, but overall I am still thinking through a lot of these concepts. I think there is something conceptually there, but its a bit foggy.