Sunday, February 26, 2012

Platform Building

As architects, the most natural thing to do is to build systems. Building a system is a lot like building a sky-scraper. There is always a foundation and strong central core. Stuff around the core is a frame, and everything else is just eye candy - windows, walls, carpets, furniture, etc... The interesting distinction is that construction architects don't erect sky scrappers in the middle of Sahara, while system architects do. In all cases, all systems start in a vaccum and proceed from there. Once the system is built, the most rudimentary of neurons is spent on deciding how to integrate the system into an existing eco environment. In most cases, this means that the author slaps together a messaging bus or maybe some kinda of REST, WS, or maybe something new fan dangled like proto-buffers, etc... The system is still a big giant monolith with some little shitty input and output.

Now what if we decided to build a platform instead of a system - what does that mean:

1. User interface split into a Bloomberg like approach - I can jump to any screen directly by entering the right tag and parameters
2. Analytic & Calculators - segregated engines that can be called as libraries or services
3. Commons Core - all the technical generic plumbing libraries and services (calendar, security, etc...)
4. Modular controllers - business logic segregated by type
5. Physical services - compute farm, data storage, distributed locking schemes like Zookeeper, etc...

But you also need orchestration of all of this:
1. Workflows, schedulers, coordinators

Sunday, January 15, 2012

Business Language

I've been toying with a new type of architecture - or more specifically another approach to solving an age old problem of ad-hoc reporting.

The question usually goes something like this:

We need a flexible, ad-hoc reporting solution which will allow us to do what ever we want whenever we want. It has to be dynamic and easy to use, and should give us the complete power to answer any business question. But we also want the ability to create presentable formatted reports and link together any type of data we so chose.

The answers vary from business vendors like SAP Business Objects or Oracle Business Intelligence tools, to smaller vendors, to various products pieced together like SSAS cubes with SSRS reporting. But, in all cases, the solution is always lacking that flexible and dynamic nature. With all tools, there is always a heavy IT presence. Some of the tools attempt to simplify this by creating an abstraction - such as the Business objects universe concept. This is effectively a meta-model around a physical representation. But this is very limiting as it restricts the use-case to a relational structure. Cubes are a very cool technology but also very limiting and the truth of the matter is that the users care about small data sets such as current days data or even a small subset of current days data - and in a number of cases the cube becomes too burdensome.

So the alternative approach was to actually give a new language to the business user directly. Goldman did something like this with their slang language. Imagine a language which is simple enough and descriptive enough for the user's to use and lacks all the very technical details of an actual technical language like Java or C#. We should accept that user's already do a fair amount of programming with the tools at hand such as VBA, and combination of Excel/Access. At times, it is very impressive some of the things these guys produce and at other times it is quite scary.

So, let's say there is a functional language that allows the user to manipulate the various available artifacts - i.e. back-end systems. Some functions retrieve data based on the supplied parameters and other functions can perform actions like calculations or data changes. Before you know it - you've provided a set of very powerful tools and what emerges is that the users start tinkering with the functions to build more dynamic and abstract features that were not previously imagined. What I came to realize is that revolutions are slow things that require tinkering, and at some point, something emerges which could not at all be previously imagined but once created makes perfect sense and is the most natural. The user's need tools to tinker, and that tinkering actually creates tremendous value and innovation. Innovation drives investment and growth, etc...

So, back to the language. The language should of course work within Excel, but should not be restricted to excel. Ideally, we bridge the language with perhaps a more functional cousin like F# and create a natural set of functions with a pure functional language. At the heart of this language is a data set - to borrow from any database driver - a TDS class - a tabular data set. Let's say the language only understands data sets not objects. Everything is a table of data or perhaps, a table of tables. Some functions create tables and some functions manipulate tables. It is also interesting to see how the roles of systems and enterprise architectures change. Systems become components which are forced to expose "functions" which are manipulated from this functional language. This in turn forces systems to become more dynamic loosely coupled modules as it's not always known how an external function may want to manipulate the components of a given system. Looked broader at a higher level, the systems start to make up an overall platform - with the functional language acting as a glue that binds it all.

I think this approach gives the most flexibility - coupled with some of the more traditional options like cubes and canned reporting - perhaps even a vendor product. The approach of course is more technical and requires some learning but it strengthens the bond between user and system and makes a user an investor as they are now in affect developing the system - or platform. It is a wonderful thing to create something that you're proud of that adds value - that feeling can be given to the user to strengthen the bond to the platform and the overall enterprise architecture.

0% unemployment

I'd like to once again deviate from my regular technological themes and instead discuss philosophy or rather some shade of philosophic economics. I was watching CSPAN the other night and a show came on discussing poverty in America. There was a panel of speakers including heavy weights in the topic along with some celebrity personalities sprinkled in. Some of the speakers espoused what is almost close to pure socialism that it was the responsibility, nay a requirement for corporations to hire people that they don't need and, which in turn, will create jobs for the poor people, and will help them get out of poverty. Now what the corporations are doing instead is hording the cash and rewarding their management ranks for driving efficiency - i.e. doing more with less people.

I am in technology and in laymen terms that means that my job is to build systems, which create automation, which in turn, create efficiencies or in other wards reduces headcount. I am even horrified when management hires too many people for one project as it destroys efficiency, makes the process tremendously burdensome and bureaucratic, and this eventually leads to either a failed system or a very poor system. These projects always cost more and deliver much less.

So - how do you achieve 100% employment - or more specifically, some higher number like 95% as in every society there will be some group of people which either cannot or won't work. In America, the official unemployment rate is just under 10%, but there is probably another 5% or more of unemployed which do not collect unemployment benefits. This means America has 85% employment - looked from this angle, that's not bad at all. But, it would be important to understand the distribution of the income across the 85%. Is the income completely skewed - i.e is it a nice bell curve or a steep hill with a cliff drop off.

If the bell curve is nice and even that would mean that we have a strong middle class and no fear of a revolution. On the other hand, if the curve is more of a cliff, then there will be tremors produced from the mountain, which could erupt and bring down everything around it. The problem is that individuals are rational, but people are not. Our political process is also looking more and more weak with the least worthy striving for notice. Politics and education should be the epitome of the most intellectual of us all - rather finance is where we spend our energies.

I am confusing various topics - but they are all related. Our wealth and the most brilliant talent seems to be focusing either on financial engineering or entertainment be it social type websites or some form of media.

It is very easy to get into a negative feedback loop and it is not easy to get out of it as the decisions and choices are counter intuitive and not at all obvious.

One should always better oneself - and perhaps the forces of good and right will prevail over the forces of ignorant.

This is not the most fluid of posts as I don't yet fully understand how all the pieces fit together.

Sunday, October 31, 2010

Zion Business Intelligence

How about a hypothetical scenario -

You are an Enterprise Architect in a Global Investment Bank and tasked with finding a solution to the data fragmentation problem.

And this is how it happened -
This particular bank has many data files flowing whether which way between various groups in the organization. Product controllers take a feed from front office systems, and market risk another feed, and credit analysts take a third feed, except the 3 feeds are generated differently, and have slightly different points of view, and in some cases, even different trade populations. But then, someone comes along and says they need to see 3 numbers - 1 from market risk, 1 from credit analysts, and 1 from product controllers side by side. And you say - that's impossible because it will take months just to analyze whether the data populations are the same, and even if there were, the numbers are produced at different levels, and even if they weren't, there is no way to show them on the same report. And even if there was, this is a 6 month project, and who is going to pay for it. And then they say, "aren't you an architect", and you hesitantly say "yes", and then they say "so go fix it."

Rabbit Hole
So, how to fix it. Core data primarily generates from front office systems and then flows through the rest of the organization. You can think of it as streams of water branching off into many smaller sub-streams, and then further branching off, until eventually the stream is to weak to branch of and just sips into the ground.

Well, easy breezy you say. Pervasive BI, data warehousing, Online Analytical Processing, bottom up data-warehousing, top down, bus architecture, centralized architecture, federation, hub and spoke, relational, dimensional, operational stores, data marts, Inmon, Kimball, conformed dimensions, Boyce-Codd normal form, 3NF, .....

We just need to take this mix - shake it up, and we'll get ourselves a fancy enterprise data architecture or perhaps something strong enough, maybe with a cherry - so we can forget the whole thing ever happened.

Blue pill or the red pill
What to do.

Well, Kimball likes the bus architecture, so, let's give that a go. The bus architecture consists of a bottom up or was it top down approach where you basically start off with a bunch of data marts, which then flow into a data warehouse. The data marts are primarily operational store type structures, while the data-warehouse is a pure data-warehouse, star-schema and all.
The problem, of course, is that you effectively already have a ton of little data marts all over the place, which don't conform. Ah, that's the problem, we need to have conformed dimensions. Right, and we do that how exactly? The other problem, of course, is that this is the same data being treated in a slightly different way, maybe with a slightly different trade population or attribute set or granularity or perhaps with a different temporal point of view. Seems awfully wrong to have a bunch of data marts storing the same data, which you then have to reconcile all together.

Right you say, let's go the other way, Inmon likes top down, so, let's create a big data warehouse, which then populates the data marts. So, who is going to build this monstrosity exactly? Well, can't be the individual business groups, because they are not stupid enough to take on a project like this, so, it would have to be some central group away from any particular business line, and close to senior management, 'cause this is going to cost a lot of money. So, the group is created, except they don't know what they are doing - very technical guys, but don't understand the business at all. And if by some miracle they do, they can't keep up with it. If this project actually succeeds, which is highly unlikely, and actually have the right amount of data, which is frankly impossible, it will still fail, because the business moves just too damn fast, and at the end of the day, they will never ever be able to fully understand and own what they are storing. So, after some time, and a whole lot of money, this will be dramatically killed off.

What about federation you say? We just need a magical vendor, and all our problems will disappear. You see this vendor will create an abstraction on top of our asylum, and this way, we will present a clear simple view shielding the end user from the underlying complexity - easy, breezy. I suppose that could work if it was actually possible to build enough complexity into this thing to actually bridge something which is fundamentally diverging and is actually able to perform at the required speed, and is actually supportable. So, let's just kill that idea for now, before we embarrass ourselves much further. Perhaps, if works for you readers out there - if you happen to like this federation thing - check out Composite, Inc - it's all the rage these days.

Red pill
Reality it is then. Well, seems to me we need a new philosophy. Let's call it Zion. There are a few core principles of this theory:

1. Each piece of data has 1 and only 1 owner.
2. The owner is the only one that can change this piece of data.
3. Each piece of data has a natural key and a surrogate key

The distinction between this theory and all the above is that this one says that first you need to understand what data is before you start deciding on how to deal with it. And the first question to answer is who is responsible for it? Responsible is not the same as which IT group owns the data-warehouse, but responsible, as in which business group is responsible for this data. If you need to change it, who best knows how to change it? How to evolve it? What it means?

If you answer this question, now you know who is going to build the data store for this type of data - and this will become the golden source for this data.

To be continued....

Friday, May 21, 2010

Social Networking

There is something interesting occurring in the world. People are exposing the most mundane bits of their life via twitter, friendster, youtube, etc...

What is interesting is where all of this is going. I think the next step will be that teens will record their life continuously and post bits of it for the rest of the world to see.

There are some examples of this already occurring:
http://research.microsoft.com/en-us/projects/mylifebits/

http://qik.com/

It needs to be effortless, continously recording, maybe an addon device behind the ear, connected to an iphone in your pocket.

After something interesting happens, you can then open the iphone, scan back, cut out the relevant section and share it via some social networking site.

Thursday, September 17, 2009

Connecting to Sybase IQ 12 from Analysis Services 2008

I've been struggling with this task for awhile now. Since MSSAS 2000, you can only use the .NET provider or the OLE DB provider.

If you have the Sybase 12 client installed, you will have the "Sybase ASE OLE DB Provider". If you also install the Sybase IQ client, you'll additionally have the "Sybase Adaptive Server Anywhere OLE DB Provider 9.0"

On the ODBC side, you should have the "Sybase IQ", "Adaptive Server Anywhere 9.0", and "Sybase ASE ODBC Driver".

1. Create an ODBC connection using "Sybase IQ" data source.
a. Give it a name in the Data source name field in the ODBC tab
b. Provide your username and password in the Login tab.
c. Enter the server field in the Database tab. You'll need to get this information from your DBA. For me, it was <hostname>_<instance name>
d. For the network tab, select only TCP/IP, and enter: host=<host>;port=<port>

2. Open Analysis Services, create a new data source using "Native OLE DB\Sybase Adaptive Server Anywhere OLE DB Provider 9.0".
a. In the server or file name entry put the name of your ODBC data source that you created in step 1.
b. Press Test Connection, everything should work.

3. In the data source views section, create a new data source view given your data source that you just created. It seems importing the tables directly fails with some arcane error.
a. Don't select any tables, just click next, until an empty view is created.
b. You can now create named queries like "select * form <table name>"

Tuesday, August 18, 2009

Healthcare

Let's ignore the red versus blue and blue versus green and all the other colors of the rainbow. Let's also ignore all the stupid nonsense coming out of news talk shows.

What's at stake seems to be a lot more fundamental.

Is it or is it not the role of Government to protect the life of its citizens. Must the Government protect the "inalienable rights" of man - life, liberty and the pursuit of happiness? Or is such responsibility more fickle and has dependencies on array of variables such as politics, and budgets and costs, and personal gains?

Let's see if we can look at this problem at an even more basic way. Life is fundamentally unpredictable - complexity theory. Tomorrow we can either be hit by the bus, get cancer, both, or neither. In the case of neither, we, humans are conditioned to ignore the perils until either one or both of the above occurs. Now, is it not the role of a stable, evolved society to protect its weak and young? What does it say about our society when we concisely ignore the weak and the young? Are we as a society egotistical, thinking like every other great society before us, that we can do no wrong, strong survive, weak perish, one must accept the law of nature?


Some statistics on the makeup of America, total uninsured by various categories:
http://www.kff.org/uninsured/upload/7451_04_Data_Tables.pdf

http://facts.kff.org/chartbooks/State Variation and Health Reform.pdf