Decision Time

Continuous optimization. This is a huge and unexplored area of supply chain that I am on task to work on. I haven't worked on it yet, and while tentatively we've planned on using Java (anything is better than C++!), I am 100% open to leapfrogging my performance.

The issues I have found with non-supported languages at Amazon are as follows:

• • Manager/Peer pressure. "You want to use WHAT?!" Having a cheering section to at the very least cheer you up helps.

  • Database interoperability. The DBAs don't allow you to connect to Oracle without using APLOracle ? Crap! One solution is to use MySQL, which it what I may consider.

  • Service and communication interoperability. Rarely any important system stands alone. You need to hit up remote cat, you need to capture and inject pubsub messages, you need to access HTTP/BSF services, etc.

  All of these are easily solvable, but not by just one person with a day job.

  I want to do this, we need to talk more obviously!

  PS: I've been talking about these things for a while, just not on a blog, and also I've been stuck with the 3 problems outlined above. I think the hardest problem (#1) for me won't be a big issue anymore, but I still need help.

For what it's worth, C++, Perl and Java were all disallowed at Amazon early on. Each of them succeeded in overcoming popular resistance by accumulating a critical mass of developers who eventually just pressed forward and started getting real work done in these languages.

Initially it was just ANSI C, our in-house web-templating language, and Emacs-Lisp for the original Customer Service app suite (which was quite popular for many years, actually). Oh, and the inevitable handful of scripting languages that are essential in Unix -- shell-script, awk, etc. All other languages were strictly forbidden.

Perl broke through first, and they had the easiest time doing it, because (a) there were many problems for which Perl was arguably the best option in 1995-1998, and (b) Perl is kiiind of soooort of a standard on Unix, so it was hard to keep it out.

I personally think Python would have been a smarter choice; it's the choice Google made, so obviously it doesn't cause grevious ills for highly-scalable problem spaces. But Perl had Python beat in the CGI space for quick-and-dirty internal web apps, for Unix integration, and overall popularity.

C++ came next. The "why can't I use C++?" griping started to ramp up heavily from mid-1998 to end-1998, and then it started appearing in big projects like Customer Master. Of course it *immediately* (as in, on the very same project) began causing portability/compatibility headaches -- we had to do a GCC-to-CXX migration for BEA compatibility, then migrate back later during PARCS. It was months of work and caused no end of bugs. But the C++ fans pressed forward fairly aggressively, once they realized they outnumbered the original Amazon architects and could brazenly flaunt their non-portable template tricks.

Java took much longer. It was used first in SCOS, and their framework was adopted by CS, but Swing didn't play well with the existing xterm-based network architecture, so it was incredibly sluggish through most of 1999, and I believe it was the infrastructure guys who put up the most resistance, saying there was no possible way we could afford the hardware costs to make Java scale. For all I know, they may have been right; I'm just recounting the sequence of events as I remember it.

I don't think Java transitioned to a first-class language at Amazon (where you could use it with impugnity) until... mid-2003, maybe? Pretty recently. And again, it was largely just a big mass of competent Java programmers who finally just forced the issue. All they needed to do was prove that it worked. They were carrying the pagers for their own systems, and they were hitting deadlines, and the systems were performing adequately for the most part, depending on who you asked.

Even today, though, it remains to be seen whether someone could sell the idea of running a Java 5 JVM instance on every essentially every online and production machine. Many people are still highly skeptical of Java's value-add, and take a "run it on your box, but not MY box" stance. I personally can sympathize with both sides of the argument, but generally speaking, I'd favor replacing C++ with Java.

I should make the important point that I trust Java and the JVM. Regardless of how I feel about Java's long-term viability, I think that being a Java shop (plus Ruby or Python for scripts, of course) would be a huge improvement over being a C++ shop, and could easily carry us for 4 to 6 years. Unfortunately, Java seems to be losing some ground to C++, for a very good reason: having a unified platform for our primary websites is far more important than the particular language choice; just ask anyone who has to deploy to three or more platforms today.

C++, Java and Perl all have a lot going for them: they're well-documented, stable, mature languages with plenty of libraries, big communities, and reasonable interoperability options. At least we're no worse off than the rest of the industry.

But my hope is that a bunch of us can agree to team up, follow the example set by the Perl, C++ and Java early-adopters in years past, and share the pain of driving a higher-quality language to Amazon-level production readiness.

I envision more than just incremental improvement. I think that if we make good choices, and bust our butts on it for a while, we could put ourselves in a position to do the really fancy stuff Amazon always dreams about, like building self-regulating systems with reinforcing feedback loops that need far less human intervention. Right now I just don't see much of that happening. I know it's a minority opinion at the moment, but I hold C++, Perl and Java largely accountable for our lack of progress on this front.

But I can guarantee a fair amount of mundane work up front -- we'll have to solve issues like your #2 and #3 before we can move on to more interesting problem domains. (FWIW, the Ruby folks here have quietly gone and solved at least #3 already. That's one of the side-benefits of using a cool high-level language: you can generally get stuff working really fast, with very little code, and it's more fun too.)

Problems #2 and #3 can be fun anyways. I don't dislike working on platform issues, especially when you are working towards a magnificent new future.

So if it is your hope some people will team up, any suggestions to those of us who have the projects and time to team up on?

Well, I'd like to give it a couple days, to give more people time to read this, think it over, and comment. In the meantime, I'm finally forcing myself to learn Erlang and Gambit Scheme, since they have uniquely scalable concurrency support that doesn't seem to exist in any other languages out there.

In both Gambit and Erlang, it's easy to fire up half a million to a million active threads on a single workstation, passing messages, each of them able to block on I/O at will without affecting any of the others. You Just Can't Do That with Java (or Ruby, or Python, or any "normal" multi-threaded or multi-processing language I'm aware of). It makes approaching the problem of thousands of concurrent connections a lot easier when the language has that level of concurrency support built-in.

If we were to choose any language other than Gambit or Erlang, it would be a prerequisite to be able to modify the language runtime to give it Erlang-style concurrency -- and get the language maintainers to accept the patch, so we don't have to maintain it in-house. Depending on the language, that could be nontrivial, to say the least.

Hence my initial predisposition towards Gambit. Although it would need a lot of library work, it has a surprisingly strong core set of language features and tools. It would be a good foundation to build on. But I need to study it for a few days to get a feel for it.

I learned scheme in University. While I wasn't 100% enthusiastic for it at the time, I didn't have the difficulty some other people did with the highly functional and recursive nature of it. I never understood why tail-recursion is better until a few years later.

But one thing we never touched on was macros. Does Scheme have macros? Is that why you are predisposed to Gambit Scheme? From the respective sites, it appears that erlang is a bit more, ahem, production ready than Gambit Scheme. Maybe I read the wrong site though?