Cromwell's rule is derived from a letter Oliver Cromwell wrote to the synod of the Church of Scotland on August 5th, 1650 in which he said "I beseech you, in the bowels of Christ, consider it possible that you are mistaken." The rule is, very simply, do not take anything for absolute; totality is wrong.
I am not Christian, in fact, I am very much Jewish, and certain parts of my anatomy will certainly show that, but I have a very great respect for that statement. The world is entering a very dangerous period. Everyone, and I mean EVERYONE, Jewish, Christian, Muslim, Buddhist, Zoroastrian, must consider the possibility that they are mistaken.
Very recently a small Danish newspaper published a set of cartoons making fun of Mohammed. They did it to show that the media is censoring itself on criticizing Mohammed. So, what happened. Well, the Muslim world went bizerk. They burned consulates, killed people, robbed banks, oh yes, robbed banks, marched, protested, burned flags, fired oozies in the air, chanted death to Israel, etc... An Iranian newspaper even created a competition for the best holocaust cartoon. Undoubtedly, numerous entries were submitted.
Everyone considers their position to be superior. No side understands or cares about the other. One side will tell you that the other does not understand or care. The other side will reverse the statement and tell you the same. Don't believe me. Here is an example. The greatest pain a person can feel is the passing of a loved one. That is the greatest fear and the greatest pain. A part of you dies as well. The more you love someone, the more you will understand this statement. On the other hand, one hundred Palestinian children being killed by a stray bomb will produce a quickly passing feeling of regret for their unfortunate passing.
Every culture, at every point in history that believed in totality brought about the greatest suffering up to that point in history. Consider Hitler, Stalin, Mussolini, Ferdinand and Isabella of Castile, Lenin, Mao, Christian Crusades, and so forth. This could be a very long list, but a few of these names should prove my point.
Someone has to be correct right? Maybe. This question seems extremely complicated. I don't know.
What I do know is that any culture or any person that believes something in such an absolute is dangerous and no matter how well meaning is horribly wrong. Nothing is simple and clear cut. Everything has multiple sides, and therefore any totality is always wrong no matter how well meaning.
I think Queen Margrethe II of Denmark said it best:
"There is, as said, something moving about people, whom to this degree surrender to a faith. However there is also something frightening about such a totality, which also is a side of Islam. There must be shown counter-play [interplay of an alternative / sparring], and once in a while you have to run the risk of getting a less flattering label stuck upon you. Because there is certain things before which one should not be tolerant."
Sunday, February 19, 2006
Sunday, February 12, 2006
Frustration
I am extremely frustrated with the keyboard and mouse, specifically the mouse. I find the mouse to be a very outdated device that simply doesn't keep up with me. Good hackers don't even use the mouse. A lot of times, I find myself simply memorizing all the possible shortcuts and deal only with the keyboard. The reason is exactly that, shortcuts. Its faster to type out some cryptic combination like holding a key down while pressing a set of other keys rather then to nagivate the mouse to a menu, and perform a few clicks. I think the problem is that the mouse is not an extension of the hand. It's unnatural. The eyes and brain move infinitely faster. Perhaps, if the hand was the mouse. It takes such a long time to move the mouse from one side of the screen to another. The mouse is made slow on purpose because it is not a very precise device. Imagine how hard it would be to navigate it, if the mouse flew across the screen. The human hand, fingers are a lot more precise. In theory, I think the hand acting as a mouse can move much faster. There is also a possibility of interacting more naturally with the computer if your hands can be involved. For example, physically taking and moving windows rather then performing some sort of click and hold motion. There are also 2 hands and 10 fingers providing a potential for even fuller interaction. There is no reason that the mouse has to be a single item. What about 10 different mouses, or perhaps there is no mouse at all, but your hands can physically interact with the computer in a virtual plane.
Perhaps part of my frustration stems from that fact that it takes a very long time to do anything on the computer. Imagine building a simple web based program. There are a couple of screens, lets say 5 queries, some fancy interaction such as ajax, perhaps some column sorting, expanding, maybe some data pop-ups, some drill downs. A little program like this will probably take one person a few days, maybe even a week to fully build out and test. That's extremely slow. This system might not even be what the business wants causing further delays. There has to be a faster way. There is nothing even remotely complicated or interesting about these little programs. These programs are simply a test of how fast one can type and how easy they can define the architecture to minimize typing in the first place. We are developing stupid little programs at such a low level. It is extremely inefficient, not to mention boring.
The solution so far has been to build tools on top of tools. IDE's are attempting to simplify certain actions, visual development tools are attempting to convert languages into workflow models. IDE's are replacing development. They are simply making it faster. Workflow based languages are creating another layer of complexity. Instead of writing an if statement you move
a picture of an if statement into a plane, and then configure it by filling out a form. This is an interesting idea, and has the capacity to produce very powerful systems rather quickly, but I am not a true believer. I don't have a solution, I am just really frustrated.
Perhaps part of my frustration stems from that fact that it takes a very long time to do anything on the computer. Imagine building a simple web based program. There are a couple of screens, lets say 5 queries, some fancy interaction such as ajax, perhaps some column sorting, expanding, maybe some data pop-ups, some drill downs. A little program like this will probably take one person a few days, maybe even a week to fully build out and test. That's extremely slow. This system might not even be what the business wants causing further delays. There has to be a faster way. There is nothing even remotely complicated or interesting about these little programs. These programs are simply a test of how fast one can type and how easy they can define the architecture to minimize typing in the first place. We are developing stupid little programs at such a low level. It is extremely inefficient, not to mention boring.
The solution so far has been to build tools on top of tools. IDE's are attempting to simplify certain actions, visual development tools are attempting to convert languages into workflow models. IDE's are replacing development. They are simply making it faster. Workflow based languages are creating another layer of complexity. Instead of writing an if statement you move
a picture of an if statement into a plane, and then configure it by filling out a form. This is an interesting idea, and has the capacity to produce very powerful systems rather quickly, but I am not a true believer. I don't have a solution, I am just really frustrated.
Thursday, January 26, 2006
Soft Asset Management
Each organization struggles to harness and manage 3 simple assets.
Ideas
The rank and file engineers have a lot of very good ideas that never see the light of day. The reason for this tends to be a layer of dead managerial weight separating any developer from any person capable and willing to make a decision. The middle management is also extremely terrified of anything that's perceived as rocking the boat or anything that causes them to stick out ( such as championing an idea). The problem is that a lot of these people are simply afraid of showing how incompetent they really are. What needs to happen is a way to send ideas directly to the senior management or some sort of governing committee that can review the ideas and has the power to fund them. I've seen one organization implement this very well. They've setup a committee consisting of senior management to review all incoming ideas. The idea is emailed directly to your regional director. (There is one regional director per continent.) The director would then take your idea to the committee for review. At the end of the year, the committee chooses a handful of ideas to pursue and those people receive awards such as trips to Paris, trophy, envelops, and recognition - due to a large assembly that you get to speak in front of.
Prior Art
"Prior Art" is a term used in patent law to describe inventions that someone has already patented. Each organization produces a lot of innovation: software, libraries, fixes, procedures, etc... The ironic thing is that each group in the organization tends to develop its own solution to a problem probably faced by every group. In a couple of places, I've seen organizations attempt to avoid some duplication, but never very successfully. At the core, the problem is that each group has no idea what the other is doing. Additionally, there is rarely a single architectural vision or a way to search for existing solutions. Basically, the proposal is to setup an internal open-source community. The community would know of all projects going on in the organization, and would manage certain projects submitted to it such as general projects like scheduling systems, or monitoring libraries. Everyone should be aware of what the other is doing. If someone in the organization is working on a security model, and I am about to start writing my own password management system, well, hopefully I know about the other initiative.
Knowledge
Each engineer accumulates a wealth of knowledge. The most relevant and long lasting learning occurs from your peers and not from a 2 day intensive class. Although, its great to miss 2 days of work and get catered lunch. What organizations struggle to do is share individually acquired knowledge. In fact, a number of organizations, because of politics discourage the practice or do it with a heavy managerial hand. For example, in one company I've worked for, the management decided that the way to share knowledge is to hold bi-weekly developer meetings. Their solution was that each manager or director would bring some of their people to the meeting. In some cases, the directors brought only managers. The end result was that the meeting consisted of 30% developers and 70% managers/directors. The funny thing is that the topics started technical and quickly moved to managerial such as billing. I spoke up in one of these meetings. I believe I said something to the degree that this is mockery, and if both my manager and my director were not sitting next to me, I wouldn't be here. Well, I am no longer with that particular company. Some ways to promote knowledge sharing is to get developers talking and debating. For example, the organization can create a developer community and hold weekly meetings. During each meeting, one of the developers can present something useful and interesting to discuss such as programming patterns, algorithms, and standards. Additionally, within a team, discussion should be encouraged. Team leaders should promote a culture of learning and challenge such as emailing logic problems. The only reward is the recognition that the person is better then everyone else. The organization should also implement code review and design review. The reviews will enforce a level of competency which is sometimes missing during coding and designing.
Ideas
The rank and file engineers have a lot of very good ideas that never see the light of day. The reason for this tends to be a layer of dead managerial weight separating any developer from any person capable and willing to make a decision. The middle management is also extremely terrified of anything that's perceived as rocking the boat or anything that causes them to stick out ( such as championing an idea). The problem is that a lot of these people are simply afraid of showing how incompetent they really are. What needs to happen is a way to send ideas directly to the senior management or some sort of governing committee that can review the ideas and has the power to fund them. I've seen one organization implement this very well. They've setup a committee consisting of senior management to review all incoming ideas. The idea is emailed directly to your regional director. (There is one regional director per continent.) The director would then take your idea to the committee for review. At the end of the year, the committee chooses a handful of ideas to pursue and those people receive awards such as trips to Paris, trophy, envelops, and recognition - due to a large assembly that you get to speak in front of.
Prior Art
"Prior Art" is a term used in patent law to describe inventions that someone has already patented. Each organization produces a lot of innovation: software, libraries, fixes, procedures, etc... The ironic thing is that each group in the organization tends to develop its own solution to a problem probably faced by every group. In a couple of places, I've seen organizations attempt to avoid some duplication, but never very successfully. At the core, the problem is that each group has no idea what the other is doing. Additionally, there is rarely a single architectural vision or a way to search for existing solutions. Basically, the proposal is to setup an internal open-source community. The community would know of all projects going on in the organization, and would manage certain projects submitted to it such as general projects like scheduling systems, or monitoring libraries. Everyone should be aware of what the other is doing. If someone in the organization is working on a security model, and I am about to start writing my own password management system, well, hopefully I know about the other initiative.
Knowledge
Each engineer accumulates a wealth of knowledge. The most relevant and long lasting learning occurs from your peers and not from a 2 day intensive class. Although, its great to miss 2 days of work and get catered lunch. What organizations struggle to do is share individually acquired knowledge. In fact, a number of organizations, because of politics discourage the practice or do it with a heavy managerial hand. For example, in one company I've worked for, the management decided that the way to share knowledge is to hold bi-weekly developer meetings. Their solution was that each manager or director would bring some of their people to the meeting. In some cases, the directors brought only managers. The end result was that the meeting consisted of 30% developers and 70% managers/directors. The funny thing is that the topics started technical and quickly moved to managerial such as billing. I spoke up in one of these meetings. I believe I said something to the degree that this is mockery, and if both my manager and my director were not sitting next to me, I wouldn't be here. Well, I am no longer with that particular company. Some ways to promote knowledge sharing is to get developers talking and debating. For example, the organization can create a developer community and hold weekly meetings. During each meeting, one of the developers can present something useful and interesting to discuss such as programming patterns, algorithms, and standards. Additionally, within a team, discussion should be encouraged. Team leaders should promote a culture of learning and challenge such as emailing logic problems. The only reward is the recognition that the person is better then everyone else. The organization should also implement code review and design review. The reviews will enforce a level of competency which is sometimes missing during coding and designing.
Monday, January 16, 2006
Metaverse
The word meta in Greek means "about", "beyond", and in English is used as a word prefix to "... indicate a concept which is an abstraction from another concept." The word verse is "... a single metrical line in a poetic composition; one line of poetry. "
The word metaverse was coined by Neal Stephenson's in his book Snow Crash in 1992. Neal uses the word to describe a virtual world that allows people to connect into and physically participate in a virtual world. People physically jack-in. The world is part virtual reality, part internet, part story.
At the moment there are a few games that are starting to broach this subject of metaverse. Games like everquest, for example, bring together all players into a single universe. People now hear of things like weddings occurring in cyberspace or real-estate being sold for real money. There is also a commodity market developing for game items such as magical weapons. Virtual items are being sold and bought with real money. Of course, one can argue that money is as real as the magical cloak that can make the owner appear or disappear or protect the wearer from all sorts of attacks.
The major difference between the cyberworlds these games create and the metaverse, is that the metaverse is not a game but real life. It has stores and restaurants, bars, and clubs. It has side walks, and bus lines and a very sophisticated police infrastructure. There is a class system, and a stable real-estate market. To get there, we need a basic virtual reality system, and a simple meta language like HTML. It should be relatively simple to create stores and shelves, and chairs, the humans will act as humans. The imaging can be performed locally, with only meta language being passed around. You should be able to walk into a store on the internet and have the same or better experience as walking into a real store.
Check this out: http://sketchup.google.com/
The word metaverse was coined by Neal Stephenson's in his book Snow Crash in 1992. Neal uses the word to describe a virtual world that allows people to connect into and physically participate in a virtual world. People physically jack-in. The world is part virtual reality, part internet, part story.
At the moment there are a few games that are starting to broach this subject of metaverse. Games like everquest, for example, bring together all players into a single universe. People now hear of things like weddings occurring in cyberspace or real-estate being sold for real money. There is also a commodity market developing for game items such as magical weapons. Virtual items are being sold and bought with real money. Of course, one can argue that money is as real as the magical cloak that can make the owner appear or disappear or protect the wearer from all sorts of attacks.
The major difference between the cyberworlds these games create and the metaverse, is that the metaverse is not a game but real life. It has stores and restaurants, bars, and clubs. It has side walks, and bus lines and a very sophisticated police infrastructure. There is a class system, and a stable real-estate market. To get there, we need a basic virtual reality system, and a simple meta language like HTML. It should be relatively simple to create stores and shelves, and chairs, the humans will act as humans. The imaging can be performed locally, with only meta language being passed around. You should be able to walk into a store on the internet and have the same or better experience as walking into a real store.
Check this out: http://sketchup.google.com/
Wednesday, January 11, 2006
Patent Madness
I am currently in the process of trying to patent an idea. The idea is relatively simple, but this post is not about that. Instead, it is about patent law. In my search, I got the opportunity to speak with a highly paid patent lawyer ($460 per hour) from an established firm. He gave me a breakdown of what is required to patent an idea.
For technology related ideas there are 2 types of patents, a full blown utility patent, and a provisional patent. The provisional patent is good for a year, but requires that you file for a full patent within that year. It's relatively cheap, and doesn't go through the rigor of a full patent. In fact, there is no rigor, whatever you file is good. The problem, as explained, by the lawyer, is if there is a dispute within that year, the language of the provisional application is closely scrutinized. Basically, if its written by the lay man, me, it won't stand up in court, and therefore is useless. The only advantage is a marketing gimmick, because it lets you say "Patent pending" for the duration of that year, and if it does stand up in court, your patent is for 21 years rather than 20.
To file a full patent with this law firm, you need to do the following:
$400 = initial 1 hour consultation with a junior lawyer and some feedback from a senior partner
$1500-$2000 = a professional search to see if your patent is already taken
$5000-$10000 = to write up the application form, the price ranges depending on complexity.
+ cost of filing, for small entity, (adds up to a couple of hundred)
He also mentioned that almost everyone gets rejected on their first submission. Your patent lawyer and the US Patent Office than negotiate on how broad your patent should be.
At the end of the day, you can have a patent for just under $20,000. The industry average, as explained by another lawyer, is $15,000 to get a patent. The cost fluctuates between $10,000 and $20,000 depending on who you get as the US patent representative, and how general your patent is. The expensive lawyer mentioned that the Amazon single click patent was very expensive. Another lawyer also added that the international patent costs even more than US. The filing fee alone is $4,000.
US Patent fees (Most of us are small entities)
World intellectual Property Organization
The patent process also takes on the average of 2 years.
If you do it yourself, you may get a patent for much less, probably around $2 -3k, but without the lawyer babel, its not worth the paper its printed on. Basically, as the highly paid lawyer explained to me, patents are written by lawyers for lawyers. Patents are only useful in court. He also kindly explained to me that the patent process is not meant for the small guy. Even if you get a patent, you may not be able to afford to defend it. Another lawyer chirped in that if you do have a patent and a requirement to defend, there are certain companies that will pay the defense fees for a share of the patent. This same lawyer also said that a patent is only necessary if you know how to make money from it. For example, license your idea, sell the patent, protect your idea so as to control the market. Patents are not required in a lot of cases, also, if you've already built the software and somehow released into public knowledge, in court, you do get some leniency during a patent dispute.
For technology related ideas there are 2 types of patents, a full blown utility patent, and a provisional patent. The provisional patent is good for a year, but requires that you file for a full patent within that year. It's relatively cheap, and doesn't go through the rigor of a full patent. In fact, there is no rigor, whatever you file is good. The problem, as explained, by the lawyer, is if there is a dispute within that year, the language of the provisional application is closely scrutinized. Basically, if its written by the lay man, me, it won't stand up in court, and therefore is useless. The only advantage is a marketing gimmick, because it lets you say "Patent pending" for the duration of that year, and if it does stand up in court, your patent is for 21 years rather than 20.
To file a full patent with this law firm, you need to do the following:
$400 = initial 1 hour consultation with a junior lawyer and some feedback from a senior partner
$1500-$2000 = a professional search to see if your patent is already taken
$5000-$10000 = to write up the application form, the price ranges depending on complexity.
+ cost of filing, for small entity, (adds up to a couple of hundred)
He also mentioned that almost everyone gets rejected on their first submission. Your patent lawyer and the US Patent Office than negotiate on how broad your patent should be.
At the end of the day, you can have a patent for just under $20,000. The industry average, as explained by another lawyer, is $15,000 to get a patent. The cost fluctuates between $10,000 and $20,000 depending on who you get as the US patent representative, and how general your patent is. The expensive lawyer mentioned that the Amazon single click patent was very expensive. Another lawyer also added that the international patent costs even more than US. The filing fee alone is $4,000.
US Patent fees (Most of us are small entities)
World intellectual Property Organization
The patent process also takes on the average of 2 years.
If you do it yourself, you may get a patent for much less, probably around $2 -3k, but without the lawyer babel, its not worth the paper its printed on. Basically, as the highly paid lawyer explained to me, patents are written by lawyers for lawyers. Patents are only useful in court. He also kindly explained to me that the patent process is not meant for the small guy. Even if you get a patent, you may not be able to afford to defend it. Another lawyer chirped in that if you do have a patent and a requirement to defend, there are certain companies that will pay the defense fees for a share of the patent. This same lawyer also said that a patent is only necessary if you know how to make money from it. For example, license your idea, sell the patent, protect your idea so as to control the market. Patents are not required in a lot of cases, also, if you've already built the software and somehow released into public knowledge, in court, you do get some leniency during a patent dispute.
Friday, December 30, 2005
Grid
There is shift in the tech community towards grids. Oracle came out with its 10G database, now Sun has released the Sun Grid, and is offering $1 per CPU hour to run apps on their grid. At the moment, grids are still a novelty. Berkeley introduced the grid to masses through the search for aliens (SETI). Their system, BOINC, is a very nice and simple grid, but is very University geared. There are a number of vendors and a couple of enterprise strength open-source solutions.
Vendor Tools
http://www.entropia.com
http://www.datasynapse.com/
http://www.avaki.com
http://www.platform.com/
http://www.ud.com
http://www.gridsystems.com/
Open-Source
http://www.zetagrid.net
http://www.globus.org
http://www.cs.wisc.edu/condor/
http://gridengine.sunsource.net/
http://boinc.berkeley.edu/
http://www.globus.org/cog/java/
The other interesting development is that people are talking about Grids in the same breath as web-services. Personally, I am not a big fan of web-services, and has always considered web-services a solution to a very specific and narrow problem space. A grid can be thought of as a single computer with a whole lot of CPU's. I guess running apps on the grid is really just a service, and web-services, in theory, is attempting to standardize that communication. A service oriented world is interesting, but I am not convinced the world works that way.
On top of all this bable, I am proposing to build a PSG (Pretty Simple Grid) as an open-source project. The grid, unlike all other grids, will be geared for small teams whether in large organizations or not. The grid must be simple to install and simple to use. It's features will be limited to what is required and what doesn't require much setup. It will be written in java; client download-able via internet; the client will run as service, screen-saver, etc... The goal is to allow individual technology teams to leverage the grid in their offices. Most of us working for the Man probably work in pretty small teams, but are a part of a large entity that is impossible to convince to install a large grid. A small grid used solely in the small team with their 10 pc, is simple to install and provides a lot of value.
Vendor Tools
http://www.entropia.com
http://www.datasynapse.com/
http://www.avaki.com
http://www.platform.com/
http://www.ud.com
http://www.gridsystems.com/
Open-Source
http://www.zetagrid.net
http://www.globus.org
http://www.cs.wisc.edu/condor/
http://gridengine.sunsource.net/
http://boinc.berkeley.edu/
http://www.globus.org/cog/java/
The other interesting development is that people are talking about Grids in the same breath as web-services. Personally, I am not a big fan of web-services, and has always considered web-services a solution to a very specific and narrow problem space. A grid can be thought of as a single computer with a whole lot of CPU's. I guess running apps on the grid is really just a service, and web-services, in theory, is attempting to standardize that communication. A service oriented world is interesting, but I am not convinced the world works that way.
On top of all this bable, I am proposing to build a PSG (Pretty Simple Grid) as an open-source project. The grid, unlike all other grids, will be geared for small teams whether in large organizations or not. The grid must be simple to install and simple to use. It's features will be limited to what is required and what doesn't require much setup. It will be written in java; client download-able via internet; the client will run as service, screen-saver, etc... The goal is to allow individual technology teams to leverage the grid in their offices. Most of us working for the Man probably work in pretty small teams, but are a part of a large entity that is impossible to convince to install a large grid. A small grid used solely in the small team with their 10 pc, is simple to install and provides a lot of value.
Tuesday, December 06, 2005
Are great developers dopamine addicts?
The below post is not complete, and not fully developed. I believe there is some sort of connection between dopamine and great developers. But, dopamine may not be the cause, but an effect. In any case, enjoy the post but don't judge the author.
Dopamine is natural chemical produced by the body. Based on medical studies, researchers have discovered that during periods of excitement and satisfaction, dopamine levels in the brain increase. One such study, discussed in the Wall Street Journal, shows that the reason people enjoy shopping is because of the excitement of trying on something new, or experiencing something new. For example, they've found that people tend to buy more things when they shop in a new environment such as another city.
Great developers tend to be people that are constantly searching for new challenges. They are in a constant pursuit of the known. Give a great developer a non-trivial project, and ask them what they think of it. A great developer will tell you that the project is fun and interesting, a bad developer will complain that the project is hard. Give a great developer a simple project that they've already mastered, and ask them what they think of it. A great developer will complain of boredom, and the lameness of the project. A bad developer will seem happy to have received an easy task.
So, where does dopamine come in? I am thinking that the reason great developers are in a constant pursuit of the known is because the pursuit is exciting; pursing the unknown gives them pleasure. It's the excitement of the chase. Once the chase is over, the excitement is over, the dopamine level decreases, and the developer becomes sad and bored until the next challenge.
Dopamine is natural chemical produced by the body. Based on medical studies, researchers have discovered that during periods of excitement and satisfaction, dopamine levels in the brain increase. One such study, discussed in the Wall Street Journal, shows that the reason people enjoy shopping is because of the excitement of trying on something new, or experiencing something new. For example, they've found that people tend to buy more things when they shop in a new environment such as another city.
Great developers tend to be people that are constantly searching for new challenges. They are in a constant pursuit of the known. Give a great developer a non-trivial project, and ask them what they think of it. A great developer will tell you that the project is fun and interesting, a bad developer will complain that the project is hard. Give a great developer a simple project that they've already mastered, and ask them what they think of it. A great developer will complain of boredom, and the lameness of the project. A bad developer will seem happy to have received an easy task.
So, where does dopamine come in? I am thinking that the reason great developers are in a constant pursuit of the known is because the pursuit is exciting; pursing the unknown gives them pleasure. It's the excitement of the chase. Once the chase is over, the excitement is over, the dopamine level decreases, and the developer becomes sad and bored until the next challenge.
Friday, October 21, 2005
Sunday, October 16, 2005
Autonomous Computing
Autonomous Computing is a very popular buzzword at IBM. The term describes a system that has self-configuration, self-healing, self-optimization, and self-protection. (copied from IBM)
A system that has all these attributes is excellent. The system does not require configuration. If you put a system on a production machine, it will automatically recognize that its in production and identify production configuration settings. In case of an error, the system will attempt to identify where the error originated from, fix the problem, and correct side affects by re-processing business logic, for example. The system is also able to monitor itself and perform tuning. For example, the system would recognize that certain data-structures tend to be of specific sizes, and initialize the data structures with the necessary size rather than continuously performing costly re-size operations. The system might also identify certain processing that can be done in parallel, and automatically split the processing. The system will also attempt to survive. If the production server is inadvertently stopped, the system will migrate to a different server, re-configure, and continue. In another scenario, if the database fails, the system will switch to a different storage medium.
Great, absolutely great; really hard to do. At interesting problem arises when building non-deterministic systems. They are very hard to test. More specifically, it is very hard to know exactly how a system will react in a scenario that hasn't been considered. For example, a system might rerun certain jobs numerous times causing data corruption, or inadvertently switch servers causing data fragmentation. The system might fix a data error by tweaking variables that could cause data problems without generating any errors. The bottom line is, for risk critical systems, non-deterministic machines have a potential to cause more harm then good. This might be why the business community has been weary for AIish technologies.
I am a great believer in non-deterministic systems. I think there is a great benefit to them. The problem is how to introduce them in a way that makes them more deterministic. The probable answer might be to build more complex systems. Another answer might be more descriptive languages. Each function might have attributes telling the system what could be done with this function. If the function modifies data, then it is not idempotent, etc... The system almost has to understand what its limits are, and work within the given confines.
A system that has all these attributes is excellent. The system does not require configuration. If you put a system on a production machine, it will automatically recognize that its in production and identify production configuration settings. In case of an error, the system will attempt to identify where the error originated from, fix the problem, and correct side affects by re-processing business logic, for example. The system is also able to monitor itself and perform tuning. For example, the system would recognize that certain data-structures tend to be of specific sizes, and initialize the data structures with the necessary size rather than continuously performing costly re-size operations. The system might also identify certain processing that can be done in parallel, and automatically split the processing. The system will also attempt to survive. If the production server is inadvertently stopped, the system will migrate to a different server, re-configure, and continue. In another scenario, if the database fails, the system will switch to a different storage medium.
Great, absolutely great; really hard to do. At interesting problem arises when building non-deterministic systems. They are very hard to test. More specifically, it is very hard to know exactly how a system will react in a scenario that hasn't been considered. For example, a system might rerun certain jobs numerous times causing data corruption, or inadvertently switch servers causing data fragmentation. The system might fix a data error by tweaking variables that could cause data problems without generating any errors. The bottom line is, for risk critical systems, non-deterministic machines have a potential to cause more harm then good. This might be why the business community has been weary for AIish technologies.
I am a great believer in non-deterministic systems. I think there is a great benefit to them. The problem is how to introduce them in a way that makes them more deterministic. The probable answer might be to build more complex systems. Another answer might be more descriptive languages. Each function might have attributes telling the system what could be done with this function. If the function modifies data, then it is not idempotent, etc... The system almost has to understand what its limits are, and work within the given confines.
Monday, September 26, 2005
SOX - Security Policy
I was asked to solve an interesting problem today. The problem is to hide the database password from everyone but the production system that uses it. The db account information is currently stored in properties files. The properties files are in plain view on the production boxes, along with the version control system, etc... The goal is to have the password reside in a single spot, and in such a way, where it is still accessible by a couple of production system across the globe, but is not known by anyone but the senior manager and the dba. hmmmm. It should also be possible to change the password by modifying it in one spot, and have every system automatically start using the new password. hmmm again.
I thought about this for a bit, and came up with using public/private key cryptography. The idea is to put a private key on each machine that needs to use the production db account. The private key will only be accessible by the system account. The system properties file will contain a guest database account that will have access to a password table. The password table will contain a crypted account information that was encrypted by the dba or the manager using the associated public key. So, the dba crypts the db account using the public key, writes the crypto into the table. Each system has a guest account to read the table and has access to the private key which will decrypt the account. The system will then drop the guest connection, and re-create db pool using the decrypted production db account. The solution sounds good, but has a major flaw. It requires a guest account on a production database system. The guest account might not sound very dangerous, but it allows the hooligan to start from within the database rather than have to figure out how to even connect to it.
I thought about this for a bit, and came up with using public/private key cryptography. The idea is to put a private key on each machine that needs to use the production db account. The private key will only be accessible by the system account. The system properties file will contain a guest database account that will have access to a password table. The password table will contain a crypted account information that was encrypted by the dba or the manager using the associated public key. So, the dba crypts the db account using the public key, writes the crypto into the table. Each system has a guest account to read the table and has access to the private key which will decrypt the account. The system will then drop the guest connection, and re-create db pool using the decrypted production db account. The solution sounds good, but has a major flaw. It requires a guest account on a production database system. The guest account might not sound very dangerous, but it allows the hooligan to start from within the database rather than have to figure out how to even connect to it.
Thursday, September 15, 2005
Friday, September 09, 2005
Humility
I saw something today that I would not wish upon my worst enemies. I saw humanity at its core. There was nothing that could have been done. Nothing in the world could have changed it.
Our bodies are extremely fragile, and it all ends as quickly as it starts.
The Rabbi said it was meant to be. Our lives are pre-ordained, he said. It is what it is. Fate.
He went on to say that we come in to this world with our hands closed, and leave this world with our hands open. In the beginning we are selfish, and want for ourselves, at the end we take nothing except for who we were.
There was nothing to do. I stood at the edge watching as her husband shoveled dirt. It was an unbearable site, but it had to be done. And, he had to do it. To be in that position is absolutely unthinkable, the absolute misery. But it had to be done. It was very important.
That's it. That was the end. There was nothing that anyone could have done. How can our lives be so fragile, and we spend them so recklessly. The Rabbi said that what we take from this world is who we were, what we accomplished, our respect, our dignity, humility.
Time goes by so quickly. It rushes by, going quicker and quicker. I feel it now; every day time goes faster and faster. Hours roll into days, weeks, months, years. Years go by as fast as a minutes. Ideas, moments, events, opportunities, gone as quickly as they appear. Some are forgotten and lost, others remain as a memory, a feeling, wrapping themselves around us, forming who we are, what we shell take.
I need to take life more seriously, or perhaps, the goal is to take it less seriously. Be a person said the Rabbi, that's it.
Our bodies are extremely fragile, and it all ends as quickly as it starts.
The Rabbi said it was meant to be. Our lives are pre-ordained, he said. It is what it is. Fate.
He went on to say that we come in to this world with our hands closed, and leave this world with our hands open. In the beginning we are selfish, and want for ourselves, at the end we take nothing except for who we were.
There was nothing to do. I stood at the edge watching as her husband shoveled dirt. It was an unbearable site, but it had to be done. And, he had to do it. To be in that position is absolutely unthinkable, the absolute misery. But it had to be done. It was very important.
That's it. That was the end. There was nothing that anyone could have done. How can our lives be so fragile, and we spend them so recklessly. The Rabbi said that what we take from this world is who we were, what we accomplished, our respect, our dignity, humility.
Time goes by so quickly. It rushes by, going quicker and quicker. I feel it now; every day time goes faster and faster. Hours roll into days, weeks, months, years. Years go by as fast as a minutes. Ideas, moments, events, opportunities, gone as quickly as they appear. Some are forgotten and lost, others remain as a memory, a feeling, wrapping themselves around us, forming who we are, what we shell take.
I need to take life more seriously, or perhaps, the goal is to take it less seriously. Be a person said the Rabbi, that's it.
Sunday, August 21, 2005
Steganography
There was a post on slashdot about Steganography. Now I am not much of a slashdot reader, but chaos prevailed, and I was on slashdot. The concept is great. The ability to hide data within other data such as an image. There are a couple of algorithms available out there, but it seems none are mature enough for general usage.
There is a good article here, at least an article that rates high on google.
http://www.guillermito2.net/stegano/ideas.html
The guy makes a very good point that all data out there is not random, seems random, but in fact there is order to chaos. Encryption algorithms tend to go against the grain. Its like dumping a pink elephant in the middle of Time Square and asking what's out of the ordinary. Most encrypted files beckon for decryption. They sit out there in plain site with nothing hiding them but the encryption algorithm.
Encryption if very complicated, Steganography seems even more complicated. I am wondering if its possible to use the actual image as data; versus adding something to the image, or modifying part of the data stream like the header or certain insignificant bits. Given a file and an image, would it be possible to produce a mapping such that the file maps to the image, and given the image and the key its possible to produce the file. The key is obviously very complex and very specific to the file. The key is meaningless by itself, and so is the image. Of course, the weakness is that the key must be protected. The beautify is that there is no encrypted file sitting somewhere. Instead there are trillions of images sitting in the public domain. No algorithm defined, but it should be possible. For example, given a neural net, it is possible to get the correct result. The image is the outcome, the file is the inputs, the net then figures out the weights on the nodes to get the right result. The weights then make up the key, assuming that reverse is true, which I am not so sure.
There is a good article here, at least an article that rates high on google.
http://www.guillermito2.net/stegano/ideas.html
The guy makes a very good point that all data out there is not random, seems random, but in fact there is order to chaos. Encryption algorithms tend to go against the grain. Its like dumping a pink elephant in the middle of Time Square and asking what's out of the ordinary. Most encrypted files beckon for decryption. They sit out there in plain site with nothing hiding them but the encryption algorithm.
Encryption if very complicated, Steganography seems even more complicated. I am wondering if its possible to use the actual image as data; versus adding something to the image, or modifying part of the data stream like the header or certain insignificant bits. Given a file and an image, would it be possible to produce a mapping such that the file maps to the image, and given the image and the key its possible to produce the file. The key is obviously very complex and very specific to the file. The key is meaningless by itself, and so is the image. Of course, the weakness is that the key must be protected. The beautify is that there is no encrypted file sitting somewhere. Instead there are trillions of images sitting in the public domain. No algorithm defined, but it should be possible. For example, given a neural net, it is possible to get the correct result. The image is the outcome, the file is the inputs, the net then figures out the weights on the nodes to get the right result. The weights then make up the key, assuming that reverse is true, which I am not so sure.
Tuesday, August 02, 2005
Software Complexity
I spend a lot of time developing large systems. I find myself continously wrestling with the fact that in order to make a system easy to use, it needs to be exceedingly complex. You could almost draw a direct parallel. As the system becomes easier to use, its complexity increases by the same factor. The side affect is that maintaining a large system becomes that much more complex. Each system has certain drawbacks that were probably rationalized as functionality, or perhaps functionality that later become a drawback. For example, lets say a system maintains some internal data cache. The data needs to be refreshed every day.
Solution 1 is to enforce that the system is restarted every day causing the internal cache to be refreshed.
Solution 2 is to build a timer that automatically refreshes the internal cache every midnight. The timer locks the system for that period and refreshes the cache.
Solution 3 is to build something dynamic that is able to identify a slow period, then lock the system and perform the refresh.
Solution 4 is to build a partial cache refresh, were only changed items are refreshed. The system automatically scans the data source every few minutes, identifies the changes and inserts them into the cache, locking only the data source.
Solution 5, the external data source notifies the system when data changes occur, and the system performs a specific cache refresh, locking only the items that changed.
Solution 6 incompasses solution 5, except instead of locking, a seperate cache is built and swapped in during a slow interval, etc...
There are a lot of solutions to this problem. In fact, there are a lot of solutions without even considering AI where you can start having nueral nets try to predict slow intervals, optimal times for refresh, or which data is likely to change, etc... Each solution makes the system more flexible and much smarter but with an ever increasing complexity cost. Solution 6 will probably yield the most flexible system with the least amount of downtime with perhaps some AI thrown in. But, Solution 6 will also be very complex with quite a bit of code envolved in making this work. There is going to be a seperate system that needs to be aware of data changed, a method of notification, ability to modify and lock parts of the cache, ability to identify slow periods or periods where the specific data is not used, etc.. Solution 6, if built correctly, will probably require little day to day maintenance but once it fails will be very complex to troubleshoot and fix.
So, what's the point of all this? I don't know. I like to make complex systems simple, but that's a loosing battle. Perhaps, the curve is not really a a straight line but more of a bell curve, where after a certain point, the system complexity drops. The system becomes so complex and so smart that it is actually easy to maintain.
Solution 1 is to enforce that the system is restarted every day causing the internal cache to be refreshed.
Solution 2 is to build a timer that automatically refreshes the internal cache every midnight. The timer locks the system for that period and refreshes the cache.
Solution 3 is to build something dynamic that is able to identify a slow period, then lock the system and perform the refresh.
Solution 4 is to build a partial cache refresh, were only changed items are refreshed. The system automatically scans the data source every few minutes, identifies the changes and inserts them into the cache, locking only the data source.
Solution 5, the external data source notifies the system when data changes occur, and the system performs a specific cache refresh, locking only the items that changed.
Solution 6 incompasses solution 5, except instead of locking, a seperate cache is built and swapped in during a slow interval, etc...
There are a lot of solutions to this problem. In fact, there are a lot of solutions without even considering AI where you can start having nueral nets try to predict slow intervals, optimal times for refresh, or which data is likely to change, etc... Each solution makes the system more flexible and much smarter but with an ever increasing complexity cost. Solution 6 will probably yield the most flexible system with the least amount of downtime with perhaps some AI thrown in. But, Solution 6 will also be very complex with quite a bit of code envolved in making this work. There is going to be a seperate system that needs to be aware of data changed, a method of notification, ability to modify and lock parts of the cache, ability to identify slow periods or periods where the specific data is not used, etc.. Solution 6, if built correctly, will probably require little day to day maintenance but once it fails will be very complex to troubleshoot and fix.
So, what's the point of all this? I don't know. I like to make complex systems simple, but that's a loosing battle. Perhaps, the curve is not really a a straight line but more of a bell curve, where after a certain point, the system complexity drops. The system becomes so complex and so smart that it is actually easy to maintain.
Sunday, July 24, 2005
Sarbanes-Oxley
The organization, I am currently consulting for, is in the heat of trying to comply with Sarbanes-Oxley. I hear this name mentioned at least twice a week, if not more, as a reason for a number of wrong and potentially dangerous decisions. The act, through legal wording, goes through great detail to specify how auditing shall be done, how its paid for, who is going to do it, what the deliverable will be, etc... The report explicitly assumes that auditing is a major fix to all the accounting problems we are having. As long as the auditors do their job, everything will be fine. The report also touches upon record keeping, and briefly mentions, I believe in one sentence, the requirement to maintain safe access to a production system.
It is very import to question what auditors say. Some of my friends are these auditors; they are overworked, just out of college, being promised partnerships, and the sky. The act is designed for auditing companies, if not written by them, not the companies that are being audited. The requests the auditors are making are going to cause more harm then good. Yes, the requests sound excellent on paper, but will create a huge mess on the ground. The implementation of this Act is also being unnecessary rushed, probably for political reasons. DO NOT RUSH THIS! Fixing the mess will be much more expensive.
I am also having some trouble finding the exact places that outline all this requests mentioned by the Auditor. Perhaps, the Act simply gives the auditor control, and outlines that the auditor will be partial and correct. The Act then simply assumes that the auditor is right in his statements, and the company is required to comply.
If you take away anything, question everything. The auditors are following subject lines rather than diving down and understanding the text.
Compliance is, of course, required by law, but it should be done in a way that develops a limber organization that is able to adapt and involve. Rather than, an organization stuck in a paper trial and an ever evolving mountain of red tape.
It is very import to question what auditors say. Some of my friends are these auditors; they are overworked, just out of college, being promised partnerships, and the sky. The act is designed for auditing companies, if not written by them, not the companies that are being audited. The requests the auditors are making are going to cause more harm then good. Yes, the requests sound excellent on paper, but will create a huge mess on the ground. The implementation of this Act is also being unnecessary rushed, probably for political reasons. DO NOT RUSH THIS! Fixing the mess will be much more expensive.
I am also having some trouble finding the exact places that outline all this requests mentioned by the Auditor. Perhaps, the Act simply gives the auditor control, and outlines that the auditor will be partial and correct. The Act then simply assumes that the auditor is right in his statements, and the company is required to comply.
If you take away anything, question everything. The auditors are following subject lines rather than diving down and understanding the text.
Compliance is, of course, required by law, but it should be done in a way that develops a limber organization that is able to adapt and involve. Rather than, an organization stuck in a paper trial and an ever evolving mountain of red tape.
Monday, July 11, 2005
Distributed Groove
I spent my day troubleshooting JGroups. For those unfamiliar, JGroups is a open source middleware system. The problem was that sometimes when a client system is restarted, the client would fail to see the cluster and instead become a coordinator of its own cluster. After a bit of research, the problem ended up being connected with long garbage collection delays on the real coordinator. This caused the server to not respond to heart beat requests, which in turn caused the client to think that it was alone and therefore become its own coordinator in its own cluster.
To solve the problem, I started looking through the JGroups source code and I stumbled upon a reference to the Lamport timestamp algorithm. I remember studing the algorithm in school. The basic premise is an ability to understand logical order of messages in a distributed environment based on a concept of time. His paper goes into quite a bit of detail, some of it awfully trivial, and other awfully complicated. This brought me to his website, where I discovered a stock pile of research papers.
He covers a lot of very interesting concepts, most of it some what over my heard. The bizantine systems are very interesting: the ability to write systems that can react to any type of error. He also goes in to describe a truly parallel garbage collection algorithm, something that would be quite nice in Java.
It's very interesting. Most of the code being written these days, myself included, is written to get it out quickly, and cheaply. The code does what it supposed to do, but is by no means very efficient or bullet proof. It does the work, but is at best a temporary solution. Definately not elegant. Then I get to see all these research papers dealing explicitely with the elegance of programming. It is a nice feeling: raw computer science.
To solve the problem, I started looking through the JGroups source code and I stumbled upon a reference to the Lamport timestamp algorithm. I remember studing the algorithm in school. The basic premise is an ability to understand logical order of messages in a distributed environment based on a concept of time. His paper goes into quite a bit of detail, some of it awfully trivial, and other awfully complicated. This brought me to his website, where I discovered a stock pile of research papers.
He covers a lot of very interesting concepts, most of it some what over my heard. The bizantine systems are very interesting: the ability to write systems that can react to any type of error. He also goes in to describe a truly parallel garbage collection algorithm, something that would be quite nice in Java.
It's very interesting. Most of the code being written these days, myself included, is written to get it out quickly, and cheaply. The code does what it supposed to do, but is by no means very efficient or bullet proof. It does the work, but is at best a temporary solution. Definately not elegant. Then I get to see all these research papers dealing explicitely with the elegance of programming. It is a nice feeling: raw computer science.
Monday, May 23, 2005
First Entry
For those of you wondering what Pons Asinorum means, the exact translation is "Bridge of Asses". The phrase comes from Euclid's fifth proposition from book one of elements. The proposition states that the two angles of an isosceles triangle are equal. Euclid meant the phrase as the first test of intelligence and a bridge to the harder problems that follow.
The reason I choose this name is two fold. On the one hand, this blog will be used as a dumping ground of half baked ideas and half baked knowledge. This blog is a means to explore the ideas and knowledge in further detail to perhaps derive something more interesting and useful. So, the first part of the bridge is my own test of intelligence. The second part, is the bridge to something else that hopefully the ideas and knowledge combined will produce.
The reason I choose this name is two fold. On the one hand, this blog will be used as a dumping ground of half baked ideas and half baked knowledge. This blog is a means to explore the ideas and knowledge in further detail to perhaps derive something more interesting and useful. So, the first part of the bridge is my own test of intelligence. The second part, is the bridge to something else that hopefully the ideas and knowledge combined will produce.
Subscribe to:
Posts (Atom)