Masterminds of Programming Book

O'Rielly's new Masterminds of Programming book includes interviews of me and Tom Love (Objective-C), Falkoff (APL), Kurtz (BASIC), Moore (FORTH), Milner (ML), Chamberlin (SQL), Aho, Weinberger, and Kernighan (AWK), Geschke and Warnock (PostScript), Stroustrup (C++), Meyer (Eiffel), Wall (Perl), Jones, Hudak, Wadler, and Hughes (Haskell), van Rossum (Python), de Figueiredo and Ierusalimschy (Lua), Gosling (Java), Booch, Jacobson, and Rumbaugh (UML), Hejlsberg (Delphi). Its an honor to be part of that crowd!

My interview reiterates and expands on the brick analogy I've developed in this blog. My interests are more in the components that languages produce, not languages themselves. I have often described Objective-C as the soldering gun that helps to build and use Software-ICs. I had exactly that metaphor in mind when I invented it.

However OOP-style classes are very small granules; grains of sand when bricks (and even larger) components are needed. SOA services are very large-granuarity components. The trusted security components mentioned below are small (sub-SOA) components used to make them. In other words, if the enterprise is a city, SOA services are buildings, trusted security components are bricks to make the buildings, and OOP classes are sand, mud and straw used to make bricks.

The problem is that our industry has not yet started making real bricks, fully encapsulated (no dangling dependencies), standards-compliant interfaces, tested for compliance, and certified as trusted components. Rather we use mud bricks constructed at each building site from whatever mud and straw is at hand (JBOSS, java.net, Spring, etc) by whoever is part of that project. Since components aren't trusted, every SOA service must be individually tested and certified from the ground up since the qualiity of mud bricks depends entirely on the skill of whoever made them. It's a big problem.

Certification and testing is just one of the differences. As with mud vs real bricks, there are technical differences involved in tightening up encapsulation so that changes in underlying dependencies won't invalidate a trusted component. That is why I use OSGI as the basis for SoaKit, a SOA security components suite I've been working on for several years. I'll describe that in more detail when I get a moment.

I'll close here by relating this analogy to the Free and Open Source (FOSS) movement. FOSS is typically involved in producing low-level classes (mud and straw) for others to make higher level components from. FOSS components are free, like mud bricks, which anyone can build from the mud and straw at any construction site. Real bricks, on the other hand, are not free, although they are  made from exactly the same no-cost materials. The trust requirement leads to an obvious business model.

It may seem odd that anyone would choose to buy real bricks when mud bricks are available for free. Yet that is the norm in every industry but ours.

Malik's Laws of Home Construction

One of our architects posted a link to Malik's Laws of Service Oriented Architecture which argues that building reusable services/software is futile. I responded with this version that substitutes "brick" for "service" throughout:

Malik's Laws of Home Construction

• No one but you will build the bricks you need in time for you to use them
• If you build a brick that no one else asked for, you will have built it for yourself
• If you build a brick for yourself, you will optimize it for your own use
• It is therefore the optimal brick for you to use
• It is very unlikely to be the optimal one for anyone else to use
• No one besides you will use it
• You will not use anyone else's

And so forth. Notice that Malik is 100% right in the context of primitive (mud brick) construction and 100% wrong in the context of modern (real brick) construction.

Also notice that customers never clamor for a transition from the mud bricks they're used to. The statement of work invariably specifies more of the same, meanwhile complaining bitterly of labor costs, weathering, and roofs collapsing on their heads. As if this were a law of nature instead of a shortcoming of the mud brick approach to architecture.

The  transition from primitive to modern is a slow and evolutionary process that isn’t even mainly technological. Its mostly about trust building, which only begins when a pioneer takes the risk and their customer starts telling their friends. I’d like us to be that pioneer (and yes, I know about pioneers and arrows).

DNI Open Source Conference 2008

I attended the DNI Open Source Conference yesterday but left right after the keynore, as soon as I realized that "Open Source Intelligence" is not at all what we mean by "Open Source Software". We mean pipes. They mean contents. And I find DNI's meaning deeply disturbing.

Part of it was the keynote speaker's "double humped camel" analogy where the gap between humps was the budget cuts of the 1990s. He followed with a "moment of silence for 9/11 victims" which I realized was a triumphant celebration of his camel's second hump in politically-correct disguise.

DNI's meaning of "open source" is basically anything that's not nailed down (as distinct from "closed source" which is). He alluded to that meaning in "there’s real satisfaction in solving a problem or answering a tough question with information that someone was dumb enough to leave out in the open".

He's not talking about software. He's talking about sifting thru mountains of irrelevant information about people's daily lives to draw half-baked conclusions from anything they find there. Airline records. Credit card records. Speed cameras. Street cameras. Anything that's not nailed down.

And that scares the bejezus out of me. Obviously because of the police state implications but also because of the dubious quality of this information. Yes, its free, and worth every penny. Arithmetically better refining of exponentially lower-quality data is just not as effective as putting boots on the ground to develop quality intel resources.

Didn't 9/11 teach us what comes from relying on high-tech SIGINT at the expense of low-tech HUMINT? Especially with even more second-hump resources to get in each other's way?

Grumpf. Oh my country.

The Mud Brick Business

Mike Taylor wrote: Creating high-quality software is an interesting mix of art and science. At least 20 years ago innovative leaders like Brad Cox and Tom Love (inventors of Objective C) began describing a "software industrial revolution" in which the process of creating software would move from an art done manually by skilled craftsmen to an "industrialized" process that allowed high-quality systems to be built from well-tested reusable parts. This dream remains largely unfulfilled.

In 2007, Accenture CTO Don Rippert described "Industrialized Software Development" as one of eight major trends that Accenture has identified as likely to have major impact on IT over the next five years1.

This triggered this email thread on "The Mud Brick Business"

Its amazing how the Software Industrial Revolution and SoftwareIC metaphors keep turning up (but flattering nonetheless). Recently I’ve been leaning to a new metaphor that contrasts primitive (mud brick) and modern (real brick) architecture which I find useful for understanding why that “dream remains largely unfulfilled”.

The difference between mud brick craftsmanship (cut-to-fit services) and real bricks (trusted standards based products) has little to do with brick-making technology and more to do with trust, standards, business models, etc. An innovative mud brick worker can't just decide to quit hauling mud (selling services) and start making standard bricks (products). That requires a business model in which standard bricks (SoaKit components) is viable compared to selling services (making mud bricks). Mud bricks are “free”. Real ones are not.

Yet somehow that bridge got crossed in antiquity to the point that we take it for granted today. Its discouraging it is still out of reach in software to this day.

I later responded to a misunderstanding: Mud bricks are made from existing materials too (soil, straw, water) just as our mud brick workers use java.net for low-level stuff. Mainly using seldom contributing.

Our work is building custom homes with mud bricks, making whatever non-standard untrusted components we need along the way (mud bricks) from java.net raw materials, not taking the evolutionary leap to standard trusted components (real bricks) that anyone can understand, trust and reuse. The evolutionary leap isn’t to pre-fab houses; its to using pre-fab trusted components (real bricks) to build custom houses, just like real engineers/architects do it.

Concise description is tricky because software engineering has not evolved trusted components nor even a vocabulary for describing them, unlike housing where there are dozens if not hundreds of well-understood integration levels; a nearly infinitely fractal tree of integration technologies (I build the house, you build the water pump, they build the motor, somebody else makes axles, somebody else steel, somebody else digs the ore, etc). Those nouns are understood and thus trusted in ways that software nouns, terms like "Access Manager" and "Access Agent", are not.

The closest software engineering gets to a “real brick" is a computer application, with java classes a distant second. Two levels (more like 1.2), unlike real engineering which has thousands of well-understood levels. That’s where SoaKit comes in as a modest beginning. It adds one new encapsulation layer between classes and applications, with OSGI as the membrane between inside and outside.

A challenging issue for software architects is... Does a mud brick business need an architecture role? I think not. Architecture never existed until construction made the shift to using trusted components (real bricks). Before then, there was only a customer, some laborers, and someone to choose mud and straw and mediate customer needs to the laborers. Everything from there was just cut to fit. Architects only emerged when there were so many trusted components available that a specialist was needed to choose between them.

DOD needs bricks, not just java clay and sand (ala JBI/SCA/SoaKit)

Note Added: This is a old article that I wrote to address the SOA Blindness that is now prevalent within DOD; the belief that SOA-scale integration is all that DOD needs. Its tries to show why an intermediate level of integration can address a major unsolved problem; SOA security and interoperability.

JBI and SCA seek to provide this intermediate level, which is why I singled them out by name. But on digging deeper, they turned out to be far more complex than this problem requires.  I eventually adopted a much simpler approach based on plain threads and queues with OSGI for modularity. I call this approach SoaKit; a collection of pluggable components for addressing SOA security and interoperability. Such components are the bricks mentioned throughout this blog. Plain java class libraries are the clay and sand.

Java Business Integration (JBI) is a Sun initiative to provide a Enterprise Service Bus for SOA services. Software Components Architecture (SDA) is a newer and far more ambitious OASIS-sponsored initiative to meet the same goal in a language-independent manner. SoaKit was originally based on JBI but has since switched to a much simpler bus based on native Java threads and queues. The needs explanation remains sound although SoaKit is no longer based on either JBI or SCA.

Consider some project building some SOA service, a sensor application for example. DOD would like to deploy that service to many different environments ranging from the lab's firewall-protected LAN to hostile environments on platforms as diverse as submarines, aircraft, and land vehicles. Each platform has unique communication capabilities and each threat environment has unique security and interoperability requirements.

But if projects are responsible not only for the services' core functionality, but also for its communication, security and interoperability requirements, the ability to reuse services across threat environments, platforms, and communication infrastructures is lost. So long as projects are responsible for developing service's full functionality in programming languages such as Java or C++, services cannot be deployed to diverse environments without changing the code and testing it from the ground up.

To gain SOA's promise, we need a way to let service development projects focus on their core competencies (the project's functionality objectives; sensor functionality in this example), and address enterprise requirements (connectivity, threat environments, security, communication links) without without changing the product (the sensor service)'s core logic.

A relatively new standard is available that does exactly that, although its advantages are largely unknown, obfuscated into oblivion by astonishingly bad terminology. JBI (Java Business Integration) is actually an integration technology similar to SOA. I'll concentrate here on why JBI is so important without explaining the arcane terminology used by its devotees, as this is available elsewhere.

JBI is an integration technology, just as soldering irons are the integration technology for making circuits out of chips, resistors and so forth. The technology is wielded by a new class of developer that I'll call configurators to distinguish them from programmers. Just as programmers assemble JBI components from lower level objects in programming languages like Java, configurators assemble SOA services from pre-existing JBI components by using XML as the configuration language.

Our general-purpose sensor service is one example of a JBI component but there are many others. Anything that abides by the JBI specification can be used as a JBI component. JBI development environments (Glassfish is one of several examples) have many others that configurators use to deploy bare Java functionality to meet the requirements of each new operational environment. Attach a pair of LAN transport components and the service qualifies as a participant within a fire-walled SOA lab environment. Replace them with transport components suitable for an aircraft or submarine and the sensor component can participate as a service within a low-threat SOA environment. To deploy to higher-threat environments, just configure in encryption, signing and integrity components, which are all provided by the JBI environment. If the sensor must communicate with incompatible services, just mediation/translation components can be added just as easily.

JBI does this by defining an internal bus like the external bus that SOA services use to communicate. Configurators use XML to connect off-the-shelf JBI components via this bus to achieve the needed effect. The net effect is that new components (like that sensor package) can be developed and tested for functionality without concern for how the service might be deployed later. This allows sensor experts to concentrate on building sensors without concern for matters beyond their specialty, such as how to secure that sensor against every-changing threats, platforms and communication technologies.

The same advantages apply to testing. Without JBI, obtaining authority to operate, for thousands of SOA services involves testing not just the services functionality, but whether it complies with security, interoperability, and transport constraints. JBI means that the new service need only be tested for whether its core functionality is correct. The service can then be protected by pre-existing encryption, signing, verification, transport and even mediation components, each of which is developed and tested within its own independent development cycle.

SCA Capabilities

mrowley writes (as I understand him) that Security characteristics (authentication, authorization, confidentiality) are one of SCA's responsibilities.

That diverges from my view, where security is something you *compose* by developing/composing SCA components, as distinct from something that SCA *provides*. The latter requires BEA to get it right, whereas in my view, there is no "right" in a broad enterprise like DOD that lacks a consensus (a standard) for what "security" really means. You're stuck with building adapters (like Objective Gateway) as distinct from letting BEA do it for you.

This goes back to why I might seem obsessed with escaping whatever requirements there might be for WSDL-based type-checked components in favor of untyped (by WSDL) components, which really means the component does type checking on whatever comes its way.

This is the same old static- vs. dynamic argument from the Ada vs. Smalltalk wars. That ultimately boiled down to...the real world demands both. In our case, you really need WSDL-specified typing of SOA services as a whole (because the standard says so). But not for the JBI components you compose those services from. They must be defined as xsd:any so that they can be stored away and resassembled later to accept whatever SOA service/SOAP message you're doing that day.

Just as in my simple auditing example, which really applies to every security component (authentication, authorization, confidentiality, integrity, nonreputiation, ....), and arguably every JBI component that aspires to broad reuse.

Especially including BCs...I just hope the Sun soapBC gotchas haven't spread to the others.