Code's Worst Enemy

If you could snap your fingers and bring Perl 6 on Parrot's implementation up to the level of Rhino's, would you still choose JavaScript?

I wish he could snap his fingers to get rid of his anti-Ruby collaborators. Ha. :-)

But, I'm training for decent athlete status myself.

And I'm hoping for a quick release of that Emacs + JavaScript IDE / byte-code compiler. :-)

I think you make a lot of good points in your post. I also, however, think you're confusing the problem of codebase size with code duplication. I'm all for removing duplication from my code, and keeping it as small as possible in the meantime. But there's a point where making code smaller by using macros, terse languages and the like actually harms the maintainability of the code as much as code bloat does.

Just so you know where I'm coming from, I've worked professionally only in C++ and Java. I think they're fine tools for a lot of jobs, but they do leave a lot to be desired as you've pointed out. My personal projects, however, range from Perl, Python, Java, C++, and Assembly all the way up to my most recent favorite language, which is Ruby. Ruby offers a lot of extra power, and I've got a personal project I'm working on right now that's using JRuby. (I agree wholeheartedly with you about VMs, by the way.)

But that doesn't mean that I like Ruby just for its terseness. As a good example of what horrors you can inflict on people in the same of reducing code size, go take a look at this MUD written in Ruby in only 15 lines of code. Would YOU want to try and add a feature to that? Granted, it's an extreme example, but I think it illustrates the problem with your thesis. Yes, codebase size can be a problem. But reducing size by decreasing readability is just as big of a problem.

Anyone dumb enough to have curly braces as a requirement is never never going to consider Javascript to be a legitimate language for developing "real" apps

Did you consider Scala at all? Ramming JRuby down people's throats still sounds like a good option too ;)

I tried to "learn programming" via Java 1.1 in high school back in the day, self-taught. My plan was to build a MUD server and client and use XML for data storage. It was... fun. Years later, in college, I did it again in Java and then in C#, always getting to the point of no longer liking what I was doing at pretty much the same stage of functionality.

Recently, as a way of evaluating the true ninja-like qualities of Ruby, I decided to re-write it using Ruby and YAML.

My personal experience, to your point, was that the Ruby version came out 1) massively smaller in code size, 2) far easier to debug and pick up after long periods of awayness, 3) in at least 1/3 of the time it took previously and 4) had ass-loads more functionality in that time frame.

Now, you could argue that my first go was as a green programmer, but the college versions were written while I was earning a degree in software engineering and (WORSE) the Ruby version was written after having worked in Marketing for a few years. Marketing!

I used Smultron. Not really scalability concerns, there. It turned out that the benefits of an IDE were only valuable because what I was doing was needlessly complex enough that an IDE became relevant. IDEs are not features of a lanaguage, in my opinion, but necessities of a language due to a lack of other features - at least to some scale.

Good stuff. As a poet, I loved the camel metaphor.

Can you say a bit more about why, exactly, you're up at 500,000 lines of code? I can guess, but the idea of one programmer working along and getting to that amount of code is a bit startling.

Good! Let's go with JavaScript, also known (to some) as Lisp/Scheme in the sheeps clothing.

And you have the right word to sell it: "Interpreter Pattern". You cannot sell anything to Java Community without first saying a spell from GoF book.

What about SCALA?

I find SCALA to be a pretty easy language to get excited about. Great expressiveness and integration into the JVM.

What about something like JPype:
http://jpype.sourceforge.net/

For some stuff you can use Python when convenient (like string processing) and for other stuff you can use Java (like library support).

@Alec,

> Yes, codebase size can be a problem. But reducing size by decreasing readability is just as big of a problem.

Lots of people assume software's length and clarity are linearly correlated. But consider this:

"I think what is wanted with programming languages and programs intended for human consumption is something along the lines of "the greatest meaning-based compression" (or "greatest semantic compression", to be more succinct). If greatest possible compression were the goal, then all that would be needed to write succinct programs would be to run the final program through a compression program like gzip."

That comes from this post, succinctness = power. Since I originally found it from Steve Yegge's Ten Challenges, I think that's probably what he's going for.

Just tossing an idea - beside reducing the size of the dirt you could also ship it to someone else. I mean someone who would actually welcome it.

Hmm I think I'll stop being metaphorical now - how about
turning parts of the program as libraries and publishing them with some Open Source license?

Sir C.A.R. Hoare agrees: "The price for reliability is the pursuit of the utmost simplicity. It is a price which the very rich find most hard to pay." (cribbed from Dijkstra's EWD1304)

Count me in the minority, too. Code size is a huge problem, and like you suggest, programmers who aren't familiar with templates, generics, lambdas, and tuples (I'd say macros, too, but I'll leave that for lispers) will be missing useful tools for compressing their code.

OTOH, today you don't get checks for writing with languages that have strong support for those approaches. That is, unless you start your own company and sell it for a hefty sum because you got in before anyone else had a chance to finish ;)

Amazon has 100 million lines of code? Are you sure? Unbelievable!!! Windows is around 50 million, Solaris is ~ 35 million.

I agree that size can be a big problem (I'm working on a very large Java project) but I'm not convinced that 200,000 lines of Ruby code really is more maintainable than 500,000 lines of Java. We programmers have a lot of experience in dealing with large Java/C/C++ codebases but has anyone got 200,000 lines of Ruby?

Jeff Atwood recently discussed the problems with Bugzilla and its large Perl codebase. Sometimes the LoC is not the problem but the functional complexity of the system.

@Jamie,

> ...I'm not convinced that 200,000 lines of Ruby code really is more maintainable than 500,000 lines of Java...Sometimes the LoC is not the problem but the functional complexity of the system.

Since ruby allows for greater semantic compression, I think a ruby system done well would take much longer to reach 200 kloc, than a Java system would to reach 500 kloc.

The trick is to maximize semantic compression, not just LoC.

Most programmers are Javascript users these days. That should not be taken to imply any fondness for curly braces and semicolons. I love the dynamicism and the object model. Unfortunately, most people proposing changes to the language want to keep the syntax and scrap the object model.

Just taken a look at Rhino. NBL is shaping up nicely.

Before ruling out Scheme, try
SISC. http://sisc-scheme.org/

Maybe you did this already, but I
never saw you mentioning it in
your blog.

It is a *great* JVM Scheme
implementation, with good
documentation.

I also like the JVM and this
combination of Scheme and the JVM
is working very well for me.

SISCWeb (http://siscweb.sourceforge.net/)
is also very nice.

As opposed to everyone else here commenting on code size vs functional complexity vs blah, I really want to ask one question.

Can I join you in your venture of rewriting this game, or is this a solo effort?

Agreed that code size is a problem, and the language of choice can definitely lead to bloat.

That being said, I've worked with many smart folks - "good" programmers, who just don't raise their level of abstraction even when the language allows it. And that holds doubly so when time is "of the essence" - as it often feels when on a deadline.

Another vote for Scala. I like Javascript and Rhino sounds interesting, but Scala is a much more powerful, expressive language.

Suppose you structured your code in such a way that you only had to look at under 1000 LoC at a time in order to do useful work. Would the 500KLoC be as big a problem?

First of all I would agree that code-base size is as big a problem as you say. However, I would posit that the size of the problem be more of a function of the structure/design of the code, than just the number of LoC.

The latest system I put into production had roughly 500KLoC. Developers would have only a couple thousand LoC in their IDE open at a time. This came about as a result of a much architectural and design work.

[You can find some of the patterns and frameworks we used on my blog: http://udidahan.weblogs.us]

Thoughts?

Also, I have a different take on the 'millions of lines of code' problem:

If I leverage 'millions of lines of code' from Apache projects, Eclipse, and a few other third-party modules in a small app with a Maven POM, is that the same thing?

How about if I have a larger app, like a game, but the game engine, logic, etc. are modularized in a similar fashion?

Maybe millions of lines of code are inevitable. But as long as they're well-defined to the point you don't have to pull them all into your head at once, there's no problem.

I agree that reorganizing into new containers and shelves won't solve anything. Its all about defining good modules and keeping those modules current and relevant in the face of unanticipated changes that would cause them to otw grow strange appendages.

Why can't that be done in Java? Apache and Eclipse seem to pull it off well enough...

@Casey

I get the feeling that's why he chose the JVM. He had existing code as well as the multitude of code that makes up the JVM supporting libraries.

Steve, I'm stealing this quote: "Java is like a variant of the game of Tetris in which none of the pieces can fill gaps created by the other pieces, so all you can do is pile them up endlessly."

If you ever see it again (which is unlikely with my modest blog) you're welcome to retaliate.

I assume when measuring the size of your game, you didn't include the JDK. That suggests another way to reduce lines of code: write code for a more featureful platform. The lines don't disappear, but they become someone else's problem.

You know that the fearfulness of bloat applies equally well to documents in natural language, too?

Rhino is a Java implementation of an ECMA script interpreter, right?

How does it do speed-wise?

Are you envisioning a Java program that's just a thin wrapper around Rhino and the "real" program is written in ECMA?

Have you looked at Lua at all?

How about LISP on JVM? Have you looked at clojure?

http://clojure.sourceforge.net/

+1 Scala.

"Its a poor workman who blames his tools." -- Anonymous

Highly structured languages make complexity possible. Scripting languages fall over before you get anywhere near the same level of complexity. if you consider that a feature I suppose thats your business.

(And Yes, I've done a lot of Java and a lot of PERL in my day. And C, C++, Modula2... )

I think you probably missed the most important tool you could have used on this project, though now maybe its a bit late...


Effective Java

It seems to me that you've experience the power of Java and it's allowed you to create something much bigger than you imagined you could. It's probably bigger than you can manage to support. So, this is the point that you have to decide is it a multi-developer project or is it still just your little pet?

There was a lot of text and I skimmed a lot. Did you tell us what the name of your game is and where to find it when you get it back on line?

Wow, that was long...I read only the first couple of paragraphs. But I was pleased that I found the word "garden" when I searched the rest of the text, as I believe it is a useful metaphor: If you only "plant" and never "weed", you won't have a pleasing result.

"The Java community believes, with near 100% Belief Compliance, that modern IDEs make code base size a non-issue. End of story.
"

Actually, we believe that IDE's make us much more productive when we have to manage code bases of varying sizes. Big difference than making "code base size a non-issue". Code base size is always an issue in probably any language you write.

Finally, I don't think your 500,000 line Java game is a good example. Mainly because you are the only one maintaining it and you're the only one who has written the code. Try dealing with an OSS project where code contributions come from volunteers who may or may not stick around to maintain their code. Coming from that perspective an IDE is invaluable to data mine, debug, and refactor code for the OSS project leads. Reliable refactoring and code analysis are crucial.

Personally, I feel most of Java's problems lie not with the language itself but with the libraries and frameworks within the JDK and outside the JDK. Rails, is IMO, the only reason Ruby has gotten any traction. Its filling the productivity void that EE has generally failed on.

you people seem to have missed the coolest (and probably the fastest) JVM language of them all...

CLOJURE

google it.

functional programming, macros, software transactional memory, etc.

you can't lose.

+1 Scala

Right there with you. I have a seemingly smaller problem than you -- 250Klines of C# and a team of people -- but the code base is just not as malleable as I'd like it to be.

Over the last 6 months, I've started attacking it from multiple directions.

For one thing, I'm integrating an IronPython interpreter. Initially, this is just for application scriptability, but long-term, I'm hoping we can replace enormous chunks of C# with it in the upper layers. I picked IronPython because it's the closest thing to a mature language out there that runs on the .Net CLR.

On the flipside, I've started integrating F# (Microsoft's OCaml variant for .Net) in the lowest layers. Initially it's as the back end for some DSLs that we're using to simplify some messy and complex code, but I'd like to see it start flatly replacing C# in a lot of places. So far, the only real problem with F# is that writing F# libraries to be used from C#, while better than using C# itself, is more cumbersome than writing F# that only has to be called from itself.

The early returns are that this is helping. It seems like a multi-language strategy is at least helping to keep our code's growth rate contained. If it gets proven, I may be able to make a push for using the new languages to actually shrink the code base, which would be a nice win.

So what's the game??? I want to play.

I see where you're coming from, but I think you're seeing a symptom as the problem. Large code is monstrously hard to manage, but I don't know that the fact that it is large code is the issue.

Can you give more information about why exactly you think this is the problem with your code? How did you reach this conclusion exactly?

Let me prescribe you a better solution:

Java programmers will stop writing bloat codes if Author Stevey stops writing bloat article about bloat Java programs without giving any factual or statistical analysis to the cause (and why these other P* languages will solve the problem.)

Oh... let's not blame the author. Let's just blame English being a bloat and bulky language.

I believe the game in question is Wyvern, which was hosted at http://www.cabochon.com/

It's sort of a mud but with a graphical interface as well.

I think most technologies suffer in their success - as more people participate in the community, more competing (and complementary) thoughts/patterns/practices vie for our attention and we understand more about what problems the technology didn't solve well. We then focus on that and lose sight what problems the technology did solve well.

Good software designers need to carefully weigh both sides (instead of getting caught up in the 'throw it out' mindset) and calculate how to bridge the gap. On that very tactical point, I totally agree that selective application of dynamic languages in an existing JVM-based application is absolutely the right way to go and achieve "semantic compression" without sacrificing too much existing investment. Further, such a decision needs to weigh factors like talent availability - the code most likely will be maintained by a third party.

I'm a Rails novice (and thus, in the 'honeymoon' period), but thoroughly expect the community to go through the same contortions in 5(+/-) years. Look at Perl. Look at JavaScript. Extrinsic factors drastically changed the fortunes of both languages in the preceding 5 years.

Semi-related: http://icedjava.blogspot.com/2007/12/dear-java-thanks-for-complexity-of.html

Steve, you sure discounted Lisp fairly quickly considering that you have made no mention that other people will be using your code base. Are you unwilling to say that you yourself dislikes Lisp?

I am writing a pretty heavyweight game in Lisp, using real graphics middleware like Gamebryo. I fail to see what a lack of a VM provides, and I wonder if the speedup from using sbcl/corman common lisp will outweigh the immediate performance hit from your application's dependance on the jvm threading model.

I think the core premise (your code is 500k lines; unit tests would double it; there is no hope!) is wrong.

Proper modularization, which really requires a good set of unit tests, would fix your problem by making one 500k project into, say, ten 10k projects, each of which acts somewhat independently. At that point, working on any of the areas of the application is working on a single 10k application with 9 additional libraries being used (in addition to the many standard and third-party libraries you are already using).

There is a certain level of verbosity in Java, and much of the required syntax doesn't do a good job of "fading into the woodwork" leaving just your functional code. Likewise, the lack of defined language constructs makes cut-and-paste (which is what patterns are) necessary, and you are right that that is bad. So, point taken there. I just don't see that as a 2-5x differential in apparent code complexity. I'd gauge that more as a 10% differential, of course varying depending on the particulars of the project.

If dynamic typing is desired in your application hacking that into Java isn't the best use of your time. Dead stop there. If you want/need dynamic types, move off Java.

Still, a 500k application need not be overwhelming, if it was designed "well" in the first place.

So, was there something keeping this from being modularly designed?

I agree that code size or bloat is an issue, but what is being discussed is really compression. Think of a picture or a movie: Each as some discrete values that are displayed. You could create a file for each for each describing all attributes of each attribute. This then properly captures the Meta data such as shapes and faces. What people have found over the years is that the size of these files is large. In order to reduce the size, you can do one of two things: 1) Reduce the number of attributes saved (resolution, depth, etc). 2) Enable some compression method. Compression methods typically look for patterns, but there are other types as well. Sometimes you do both things at once. Can you take three different pictures and reduce all to a single bit and still tell the differences between the original images? Nope, not possible.

You can think of lines of code as the same thing. Yes, using macros or another language might reduce the lines of code, but it probably does not reduce the complexity. Not unless you are also reducing the functionality (as many scripty/webby languages tend to do).

Suppose you have two languages X and W. X has 20 tags and W has 100 tags (instead of tags, I could say functions or operations or syntax or…). You can presume that something written in W will be smaller than something written in X, particularly if W has all the tags of X. Does this make the program less complex? Not really. Instead of the developer having to learn 20 things, she now has to learn 100 and that does not include knowing anything about the program itself, i.e. what it is suppose to do.

So you say the previous example does not apply, then lets try this. Suppose you have languages X and Y. X has 20 tags and Y has 20 tags but lets you use macros. You can presume that something written in Y is smaller than something in X. But is it less complex to understand? Once again, macros are just adding more tags to language Y.

To make matters worse, when you start to combine multiple languages, especially if you count XML as a language (it is really a meta language), then not only does the developer have to learn all the used parts of each language used, but has to learn how to use one over another and the complexities of moving from one to another.

Am I Java Developer? Definitely. Is Java a perfect language for every situation? Definitely Not!! Not even in Most Situations. The same could be said for any language. Any language is going to have issues (proponents, detractors, snobs, and blind followers).

So I said I agreed that code bloat is an issue, yet I went and proved that solving it by using a different language (or yet another language) does not solve the problem. I think the real problem is complexity. The more functionality you have, the more complex it becomes. If you have 500K lines that are easy to read (which your game is not), then 150K lines probably does not make it any less complex. Not unless you copied/pasted each line of code 3 times.

Lastly, developers should stop looking for the “silver bullet” of development. There is no such thing. The next language is not going to solve your problems, nor is the next design book (GoF = GOOFS?), nor the next management methodology (agile…make me walk instead of run).

Writing and maintaining a multi-million-line Java program is easy enough,
if you manage to split it up into a few hundred sub-projects (libraries)
that each have a very small "surface area" (public interfaces) that hardly
ever change.

Of course, definining these interfaces is hard, and keeping them non-"leaky"
is even harder. (In a sense, all that easy and obvious interface abstractions
have already have been defined and are now part of the runtime environment
(or event the operating system) -- hardly anybody has to implement their own
network stack, file system, or process scheduler anymore.

There are many factors in why a specific language is picked for a software project, my interpretation of what you're saying is that the size of resulting code base is not given enough priority in the decision making process.

I understand that the blog is a rant and so is more emotional rather than about rational analysis but I do think it might be nice to have a follow up where you look at specifics of how Java bloat is eliminated by languages with different features.

When the company that I work for did this we actually ended up creating our own language with the feature list we wanted. Similar to you, one of our desires was to have it work on the JVM.

This language (called Cal) has now been open sourced and can be found here:
http://openquark.org/

I've hit a few painful things over the years, but more importantly I saw the light ages ago when I got to work on an incredibly elegant bit of code. The language has some effect, but more often it is using brute force to pound out your solution that produces fat and ugly code.

Bloat and artificial complexity don't have to be problems. They often stem from the project's organization structure and/or its processes. Even a single developer can cause a problem. If you had been more disciplined, I bet your 500,000 lines could have easily been implemented in half, if not a quarter of the size. I've been ranting about that for years, but as you say nobody takes size seriously as an issue.

That said, I do find Java to be much fatter than languages like Perl or C. Some days it feels so close to VB that it makes me shiver.

Paul.

Dare I mention Fortress?

I'm a mobile developer so I feel your pain. Code base is a real problem for us...as every byte we put out in code is a byte less that we can use for heap memory. Systems are getting easier, but it used to be a real problem. Now just imaging having to write a Java program in the least number of lines possible...and you start to see the problem with mobile dev. :-)

You know, Steve,

you kinda fail to explain why this is a problem in the larger scheme of things and how it fits into that scheme. For example, in the old days, car mechanics could take apart and fix an entire car by themselves. Same for television amateurs. But these days, the amount of computer technology and integration in such machines makes them into mere bystanders. You sound a bit like an old television amateur who bemoans the amount of complexity present day's devices represent, while not talking about the greater picture and at the same time quite enjoying the extra functionality or performance his device offers him.

The only thing that can beat complexity is simplicity; does expressiveness elicit simplicity or is it just a way to shrink code? You do not make it clear. If you can shrink your code down to even 20%, just wait a couple of years, and you'll be back to where you started. There's a basic tendency in human culture that bespeaks of this motion, and it is the desire for more and better. But in principle, size needs to be no enemy; the only requirement is that the amount of information you have to deal with at any level, is kept at a certain minimum. That is why OS X is so praised, and why Windows will for a long time be more popular than desktop Linux: the amount of options open to you at any point in time is so very limited and that is what makes it easy to use. The same holds for software. Compare the man page for 'rpm' with that of 'dpkg' and you'll see what I mean. Abstraction, small interfaces, limited options, modularisation, hierarchy, layering, you name it.

I am not speaking as a senior programmer, but as one of target audience, not yet familiar with scripting languages. But I also have a bit of a mental block regarding complexity: I cannot hold as many pieces of information 'in vision' and have a bit of a harder time digesting information than most people. Because of that, I'm perhaps a bit better qualified to speak of such things. And I will tell you this: most people - most programmers - do not appreciate the difficulty people can have with the digestion and navigation of information, and this applies to everyone. Introducing simplicity at every step and every boundary and every interface, and limited, self-contained areas of relevance for any given viewpoint, is the most important thing in making complex systems simple and manageable.

If you have a system of 7 layers each of which has seven more layers, and so on, into infinity, will it be hard to manage? No. You can easily understand the working of the system at any level from any viewpoint, and the levels/area's you cannot see are irrelevant to the understanding of that part of the system you can see at any time. You will need more people to do the management of course, and there may be more to manage than if you had not been so vigorous in modularizing, but it will be a lot easier to do.

Maybe my lack of experience with large projects is speaking through, but I just want to make a general point.

I said as much here recently:

http://beust.com/weblog/archives/000462.html

ie, big code bases talk to maintanancea a lot more than your pet language's typing features

But now I see Bob Lee responding to my non-existent comment, so looks like it got pulled. Oh well.

@Udi and Tom:

I'd conjecture that in a way, modularity/limited visibility can make the code "size" problem worse, for a very simple reason: You don't know what's in there that you aren't looking at.

It's a lot easier to make a small code base "modular", in the sense of orthogonality. If you suddenly realize that you should be abstracting out, say, your text-formatting routines into a text-formatting object or library, you can do that. If the project is small enough, you can probably just scan through the thing visually and find all the text-formatting code. Or you can go tell the three other guys who work on the code to start using your new library.

When a project has millions of lines of code - you just can't do that. Nobody knows where to find all the routines that use text formatting. Sure, search capabilities are helpful, but they're not all-powerful.

This is a bigger problem with large dev teams, where there are developers you probably don't even know exist who are, at this very minute, writing something that formats text. But even one-man teams can suffer from it, because you're not always going to remember something you wrote a few years back.

I had the recent joy of looking at code I wrote 15 years ago. The first thing I noticed was that I made extensive use of linked lists, but I repeated a lot of the code. At the time, it was intentional; function calls were too expensive. But now we have fast computers, so I resolved to write myself a linked-list helper library.

A little further into the project, I was looking through some other code, and - what do you know! - found a linked-list helper library. It seemed like just what I needed. And that's when I realized - it ought to, because *I* wrote that. I just hadn't looked at it in 15 years, and neither had anyone else. So if I can't even remember to use my own abstractions, how can I expect the rest of the company to?

There's an anti-pattern I keep seeing, but haven't found a great name for yet. I call it "stratification".

Basically, the more you write high-level libraries that wrap lower-level ones, the less you use the low-level libraries. Thus, the less you understand them. At some point, you forget they even exist. At that point, you will - inevitably - write an even-higher-level library that recreates the lower-level functionality ON TOP OF the high-level library.

And if you've ever scanned in a document so you could use fax software to transmit it to a remote bank of faxmodems which will make a voice call over an IP-based, packet-switched telephone network only to reach another faxmodem in the same data center which will take the fax, convert it to a PDF and send it via e-mail to your recipient, who reads it via a web-based mail interface that uses JavaScript to implement offline mail reading functionality... you know what I'm talking about.

Seriously, am I the only one who wants to have a go at this game, bugs or no bugs?
Come on, stick it back up! Post a link. Post a linnnnnk.
It's in Java, it's not like it's going to wipe my hard drive. Well, maybe, but that'd be damn impressive.

I don't see how you prove your main point, that code size is an issue. You spend a lot of time saying it is, but offer up little in the way of proof or examples. (EG: what was the bad thing that happened to you?) It's not hard to say "code size is a problem". It's hard to say why it's a problem and give examples of 1) code that is too big/bloated and 2) code that does the same thing more succinctly.

Simple code-size reduction is not a silver-bullet. C and C++ are capable of very compact code that is almost as hard to read as Perl code. (see the ioccc - http://www.ioccc.org/ )

What I think you're getting at is code size reduction while maintaining readability. An altogether different beast and one that requires good design as much as language choice.

None of that should be taken as disagreement. Huge code-bases are beasts and have their problems. I like the idea of what you're attempting to show. I just think you've failed to adequately prove your point.

Tangentially, I wonder how much reduction is possible. This drive to write smaller code sounds very similar to the ideas behind the infamous programmer-induced rewrite. Namely, the "I can do it better!" syndrome. A noble idea, and not always misplaced. But rewriting code because the original code is ugly/too big isn't a panacea either. Crufty, old code contains a lot of knowledge about the problem being solved, earned over many years (through many nasty bugs and hours of debugging them) and encoded in usually inelegant ways. Tossing that and rewriting from scratch means you have to go through that learning process again.

There's a balance between good and good enough and simply assuming any code base over some number of lines of code is bad is near-sighted to say the least. Large problems sometimes need large solutions.

The simplistic solution, breaking the program into smaller programs that work together to solve the larger problem, only masks the problem and pushes it to someone else's plate (the poor schmuck that has to get the programs to work together correctly).

+1 Scala. Steve, could you please comment on your thoughts about Scala compared to Mozilla Rhino?

Thanks.

Amen! As a (currently)Java programmer, this frustration is what's driving me towards my own JVM language. Luckily, it has no due dates or constituency, so I can be very, very "patient" with it.

It boggles my mind that bigger is seen as better -- maybe what we need is APL for the JVM? :-)

+1 Scala. My guess is that Stevey just prefers dynamic typing. But there's not much you'll need to do in practice that Scala's type system can't express. BTW, The Scala book is now out in PDF.

Great post, I agree in principle with the code bloat argument. However, I notice Steve doesn't count the lines of code comprising the code base of the OS, JVM, or the Java libraries he uses as part of the code bloat.

I believe this is because a stable and well-designed (or well ... designed in the case of the Java libraries) and well-tested component doesn't require the mental overhead of a poorly-designed component.

Now, it is true that these things tend to be services and infrastructure, not domain elements, and services tend to have much firmer interfaces over time than domain classes, which change as your understanding of the problem domain and customer changes. However, I suspect that in any application a good portion of the code are value objects and services, not domain code. Anything to make that portion smaller is almost certainly a good idea.

Perhaps that is the future: systems and infrastructure programming in C# or Java surrounding domain programming in your favorite scripting language, which is as mutable as domains tend to be.

So when are your tips for tech interviews coming up? I have one coming and could use some advice!

Could you explain why you think VMs are important and/or useful? I am curious about this.

I'm also going through a VM scripting language evaluation, but coming out with a different conclusion.

Coming from a Java background, Groovy is attractive in that it employs the same type system, scoping rules, library, etc. as Java. All my Java knowledge is portable into Groovy. Its a far more symbiotic fit with the VM than the other scripting alternatives. One can migrate from Java to Groovy over time, mix and match Java and Groovy is no problem.

Not sure when you gave Groovy a whirl. If it was before 2007, suggest you give it another look. There is now a well run team that has produced 2 major releases this year. Latest Groovy 1.5.1 brings a lot of functional and performance improvements.

Arghh!

Have any of you making all of your wonderful observations ever bothered to learn any of the dynamic languages mentioned? Do you think it's possible there are things happening in those languages that you can't do in Java and therefore can't know and make a clear argument about?

.............................

I am surprised no one has mentioned the central tension: typing or no typing. I am leery of writing a very large system in a language with no typing (e.g., Ruby, Lisp, etc). A compiler can help catch so many simple mistakes, which are often the ones we make. Two questions, then:

1- Do you think strong typing is useful for code reliability (like I do)?

2- Do you think strong typing is inherently an order of magnitude more verbose?

'Cause I agree Java is verbose and I'm not happy about that.

Dan

@df

Surely you know of Stevey's opinions on static type systems!

Wyvern was a blast to play and I've missed it. Use whatever language or approach you have to, just hurry up and get it back out there!

... pretty please?

If curly braces is all it takes, couldnt you just do a readtable hack in Common Lisp that changes (car x) to {car x}?

Sigh.

+1 Scala
even if it's statically typed, the compression rate is very high :
- Type inference => eliminates verbosity
- Pattern Matching => eliminates the need for Visitor DP and make you write decent recursive data structures
- closures => no more Kingdoms of Nouns
- implicit => an "equivalent" of Haskell type classes
- operators => just an example in the pi-calculus library: spawn < p | q | r >
this is object oriented code, spawn is a method that returns an object, < is a method, p is a process, | is a method, q is a process, r is a process, > is a method.
- support for currying
- support for mixins
- it also supports syntactic "tricks" that helps you write DSL (like the a.m(b) === a m b, à la smalltalk).
- There's no code generation like in Ruby last time I checked (even if there's an obscure package where you can build syntactic terms at runtime)
- No loss in performance
- a good type checker

I don't think that your opinion about code base size is necessarily a minority one,. As an experienced Java programmer, I've watched over the years as developers have come along with their non-emacs fancy graphical IDEs and they've cheerfully generated all sorts of convoluted and unnecessary constructs in a few button clicks, with the result that functionally mundane components are thousands of lines long, and virtually incomprehensible.

Having said that, I still think that at its heart, if you can remove all of the more idiotic and misjudged frameworks and libraries, that Java is still a good language and it is possible to produce elegant software using it. But, similarly to C++, it requires super-human discipline to do so.

I haven't used Java now for a long while: I simply don't require the functional richness and syntactic rigour that it provides. Whenever I want to bash out a few web pages or interact with a database table, I go straight to a scripting language, and it's done within a few minutes.

This is an interesting take on the subject:

http://dspace.mit.edu/bitstream/1721.1/6064/2/AIM-986.pdf

I'm not sure if Lisp still has any chances to become relevant again, but I think that the idea of having many layers of software, operating at different levels of abstraction and possible to understand seperately is very interesting. TeX is structured this way, and it certainly is big and complex.

Try Scala. It's compatible enough with Java that you wouldn't even have to start from scratch. You could just pick a section to rewrite in Scala and slowly convert.

"Surely you know of Stevey's opinions on static type systems!"

No, I don't. Not everyone who reads a blog post has read the entire blog or knows the person.

Anyway, no matter what his opinions, it seems centrally tied to this issue -- Java, code bloat -- although I'm not sure what the right position is.

It seems to me static typing helps large projects .. and causes code bloat.

Dan

Man, the people you work with like Javascript and not Ruby? Not that i'm some Javascript hater, but i didn't expect that!

Funny story. I "got religion" about code base size after a game i wrote grew to unmanageable levels, too! (Much smaller, but it was written in Hypercard(!) rather than Java, so there weren't as many tools to manage the size. It was one of my first projects, too!)

Ever since that point i sort of liked the dynamically typed, elegant, and powerful languages--Ruby is pretty much my spec language. For a long time the "real" language (Java/C/etc) people would sort of turn their noses up at that sort of thing (including Javascript, which i also liked--though not very much) and i just sort of nodded along and shrugged it off. I thought "Well, okay then. Maybe i'm just not interested in 'real' languages." Interestingly, i couldn't stand Perl either.

Then i found this blog and was like "Oh, so i'm not the only one who doesn't like 'real' languages!"

This post, here, made me think of my old game and the like.

Incidentally:

"I mean, they 'made sense' at some superficial level when I read the marketing brochures, but now that I've written a few interpreters and have dug into native-code compilers, they make a lot more sense. It's another rant as to why, unfortunately."

I hope you'll write that some day! I'd be interested--i'm much more on the "made sense at some superficial level" understanding, so getting a shortcut to a more complete understanding would be nice...

I can't believe I'm the _11th_ person to some along and suggest considering Scala. Now I'm beginning to worry that I'm part of a fad. Either that, or a group of people who understand language design and are seriously impressed with the design of Scala. For what it's worth, I am not jumping in just because everyone else is: I knew I was going to add a comment suggesting Scala even before I finished reading the post (unless Scala was what you picked).

One important suggestion. Regardless of what language you pick, make use of it's Java interoperability features to _convert_ your code base little by little rather than rewriting from scratch. Sure, you can refactor large chunks to make use of the more powerful features of the new language -- that's the whole point of the exercise. But I'm sure you've read Joel's article on rewriting from scratch (http://www.joelonsoftware.com/articles/fog0000000069.html) and it's one opinion of his I think is born out by the facts.

I'm not young, but I am new to OOP and have experienced cycles of frustration and confusion with eclipse (and its complexity) as well as a handful of other IDEs, Visual Studio notwithstanding.

The one question that wasn't answered by this blog is, "What does language/framework/IDE-specific bloat look like (examples?) and how do the solutions you propose solve this problem?" But to give you credit, I believe you forewarned readers that this question wouldn't be answered.

Wow - using Spring is considered bloat? I would say that you haven't really groked Spring. IMHO, managing large code bases becomes really challenging when code is not 1) testable 2) decoupled. Spring is a fantastic technology for enabling both of these. If start without Spring, adding it in usually means writing negative lines of code.

Thanks for the rant. It is a timely reminder that people who do research on programming languages and those who construct them in 'the wild' must revisit the 'size' issue, and do so time and again.

I have put 'size' in quote because I don't think it is sheer size. If it were, we'd be programming in APL. (Yes, I am old enough to know that it existed.) I conjecture you mean some form of 'complexity' as in 'N pieces are related to N pieces in my code base, and I can't recall how they are related when I maintain my code base.' And so on. I wish you took some more time nailing down what it really is about 'size' that bothers you. In other words, blog again and describe the symptoms, describe what is bad with big code base.

Two comments: We (PLT Scheme) is confronted with a similar growth of our code base (mzscheme, drscheme, various tools, user contributions) but somehow sheer size hasn't done 'it' to us. We have no desire to rewrite it in JavaScript ;-)

I am one of the few serious mainstream PL researchers who has stood up to 'type research' (as in types are all there is to a PL) for two decades now. Having said that, I still think that a knee-jerk reaction to static type systems is improper. When you maintain untyped code, it does become difficult what the three arguments to your method at line 123,765 represent: closures, objects, integers, and what 'invariants' they satisfy. I have watched a programmer struggle with just that issue, and it is a programmer whom I would trust with my life (as far as programming is concerned). If the types are written down (and checked and sound), you can save yourself some 15 minutes with every method you touch. Sure it adds a few token pieces of type info to the method header, but what's that compared to 20,000 methods touched times 1 - 15 minutes of 'manual type recovery.' And don't tell me about type inference for 'scripting languages'; I have also spent four dissertations and 20 years on this, plus a couple of years of trying to analyze Python, to know better. (It's kind of acceptable if the language is designed for it from the scratch.)

-- Matthias Felleisen

I used to think there was something wrong with me, because I never, ever, ever managed to use any IDE without giving up in disgust hours later. I plain could not stand them. To me they seemed like barriers to thinking rather than assistants.

Good post. Jeff Atwood posted a "sorta" conflicting view on the same day no less! Both make for interesting reads.

http://www.codinghorror.com/blog/archives/001022.html

"Yes, codebase size can be a problem. But reducing size by decreasing readability is just as big of a problem."

Yes, but if he wanted to shorten his code and make it less readable he could just use two letter names for classes, methods and variables.

Language choice is very important. Java forces you to insert so much visual noise into your code that you have to read everything twice before you understand what it does.

It's been said already, but I'll have to say it again.

Scala FTW

If you want to massively reduce the size of your Java code base and massively increase its maintainability then you need a language that supports functional programming. There are other benefits as well, e.g. concurrency is much easier.

As long as you're stuck with the JVM, your only technically-decent option is Scala but it has few users and little support.

If you can afford to ditch the JVM then you might like to look at more popular languages than Scala, like OCaml, Haskell, Erlang and F#.

I agree with your opinion about the code size.
But there are two types of code. First the code you write by hand. And then the code you can generate. I'm speaking of power plant simulators. You have a subset of well tested functions and build the simulator with data from the real plant.

Oh, and in my work I still use tcl/tk. Programs read and useable even after ten years.

Have a look at tclblend or jacl.

Regards

rene

Have you read http://www.equi4.com/moam/fourth ?

Groovy for me. I guess it has had a few rough spots in its past but now seems to have more momentum than the other JVM languages - at least in my customer base which is historically Java focused and predominantly pro-agile.

Have you looked at QBASIC?
I have been a professional QB programmer for many years, and can assure you that you have overlooked this gem.

It's use of SUBs is really quite fantastic.

Obligatory link: http://en.wikipedia.org/wiki/QBasic

Reminds me of Paul Graham's http://www.paulgraham.com/power.html. Might be something to this code size problem after all :). Enjoyed your article.

Steve, I need no convincing here -- Java has its moments, but terseness is not one of them, while ES4 is shaping up to be pretty nice.

That said, I think your argument would probably be more compelling to the general audience (ie, people who haven't been exposed to ES4, or other languages with similar features) if you provided some examples of the sort of code compression that's possible.

I first met Ruby after reading your NBL post. On queue I recoiled at the lack of curly braces (perhaps subconsciously primed by your post), at the def keyword, which doesn't have an equivalent in Java, variables starting with @ symbols, and at the pipes used to delimit block parameters because they're not a nice symmetrical pair. From where I sat on my high horse, I thought the lack of regularity (nested curlies) would forever leave a bad taste in the mouth.

Boy was I wrong.

A few days into playing with Ruby and I forgot all about curly braces. Now I wince whenever I see those redundant parentheses tacked on the end of every parameterless Java method call, not to mention the receding tails of closing curlies at the end of every method and class definition.

So you're right, the populace has been conditioned with curly braces, but I don't think it's a show-stopper for adoption of sensible languages. The trance might be real, but it doesn't take much to snap out of it.

I walked a similar path, found Scala, and have been in love with it for more than a year.

I've written a number of 100K+ LoC code bases including the Mesa spreadsheet (for NextStep, OS/2, and OS X) and the Integer spreadsheet (written in Java.) I understand the code bloat problem and have experienced it first hand. There's an order-of-magnitude productivity decrease when the 1 or 2 key developers in a project can't keep the code base in their head anymore. Your head must be darned large to keep a 500K code base in it.

In 1999-2001, I built a Java-based web framework that sidestepped the whole code bloat issue by allowing developers to define common web development tasks using much higher level semantics and the high level semantics where compiled down into Java code. A 3,000 line object description file would typically bloat to 200,000 lines of actual Java code. The nice thing is that the developer didn't need to maintain the Java code... it was byte-code for most purposes.

In 2004, I started doing Ruby/Rails work. I wrote a bunch of commercial applications in Rails. I found that, even with reasonable test coverage, changing up code bases of 10K LoC became challenging. Additionally, because of the weak performance aspects of Ruby and lack of good Unicode support, it was not useful for my projects.

In 2006, I needed a better answer than Ruby/Rails was offering. I cast about and asked many of the same questions that you've asked in your post.

I discovered Scala. It was love and first site and my passion about Scala the language and the Scala community has intensified over the last year.

I started working on a web framework for Scala: lift. lift's functionality is similar to Rails (a little more here and a little less there) and lift's code size is about have the code size for Rails. More interestingly, the lift feature set has grown significantly over the last 6 months, but the code size has remained fairly constant.

You may ask how that bit of paradox has happened. It's like this... as my mind thinks more and more "functional", I am able to reduce code size. As I go through and work on older parts of the code that look and feel like Java programs, I'm able to re-implement pieces more efficiently such that the core meaning of the code is the most prominent part of the given code block rather than the "setup" of control structures (for/if).

While Scala does not have Macros, its combination of an advanced type system, traits (interfaces with code attached), and implicits, one can define amazing blocks of composition. I find that I don't copy/paste Scala code.

A number of other folks have suggested looking at Scala. I'll one-up them... I'll happily spend a couple of days with you teaching you Scala and working on how to approach migrating your game to Scala. Just ping me.

Thanks,

David

I wrote a fairly large game in python.

To keep the code size small every couple of months I would try and make the code size smaller.

I would bring it down from 12,000 or so lines down to 7,000. Then I'd keep working, and repeat the process.

DRY is a good one to follow.

Also data driven programming. Putting complex things into data files, kind of like DSLs.

Separating out reusable libraries can help. Especially if you can add the functionality to a library someone else maintains.

Putting things into separate libraries also forces you to decouple things. Making the complexity smaller, because you can look at smaller pieces separately.


Look at your code and figure out how to make it smaller, and simpler.

Remove crappy features that take up space.

Stop using silly things like getters, and setters, and creating classes for all types of data.

Use data structures like lists, and dictionaries instead of classes. This can reduce code size amazingly.

How you structure events in games can mean that you get emergent behavior. This can save you *lots* of code. Think about how the complexity of interactions and figure out ways to get that to scale with minimal code.


Keeping code small is a process, and a skill. If you keep practicing to keep your code simple you learn techniques to do it.



-- ps. check out pygame.org the simple game library for python based on the Simple Direct media Library (SDL). That's what I work on, and have fun with. Making games should be about fun.

Steve, guys, I am looking for a piece of advice on a language to learn. I am torn between Haskell and OCaml, no surprise here. Doesn't necessarily need to be practical, it's more about deep culture and good taste. I don't care how steep the learning curve is, just about the ultimate intellectual payout. (I am a professional programmer, not a newbie).

I hope Stevey is not reading the comments. It can't be good for his blood pressure.

Oh, I'm reading them. Good stuff.

I'll comment on a few themes I've noticed.

Scala folks and Groovy folks: you're not big enough yet. For something as big as my game, I want a proven mainstream language. I picked Rhino as a complicated multidimensional compromise; the actual reasons are a full blog post. But the short answer is "you're not big enough." Sorry.

A slightly different answer for Scala, OCaml, Haskell and other static languages: my game is a dynamic system. I was astonished to find, circa 1999, that I had effectively implemented JavaScript's prototype system in Java (with no syntax). For dynamic systems you need dynamic languages. If I were writing a compiler, I'd use an H-M language (probably Scala). This game is a different beast; it's one of the most "living" systems in the world (notwithstanding its 1-year hibernation). You could do it in an H-M system, but it would be nearly as big as the Java system.

Folks wondering if dynamic typing can work in "big" systems: the answer is that they don't get all that big, because of the dynamic typing. Unit testing becomes crucial, but you should do that anyway, regardless of language. Some good discussion of the reasons for dynamic languages being so compressible can be found in http://www.amazon.com/o/asin/0262220695

Folks who wonder about how I could (re)write such a giant system without any "static" type annotations: http://tinyurl.com/26n2zq


Java folks who simply disagree with me: to paraphrase Nikita Khrushchev, "You will bury you."

Folks begging for Wyvern to come back: it'll happen before the end of this month, promise.

Folks advocating JRuby: you're really twising my arm. I miss Ruby, and will always be jealous of JRuby, even after ES4 comes around. *sigh*

+ 1 scala

I discovered it about 3 months ago and the more I know it, the more I love it.

If you come from a java world you tend to start coding a la java. Then you start to have some deeper understanding of the language, like thinking more functional, and you tend naturally to write shorter programs. And not only shorter are your programs, they are also more elegant and more readable.

With some other languages, making your programs shorter can lead to make them more cryptic. That's not my experience with scala.

I have a java background, have done a lot python, some C and ruby, and played happily with lisp. Scala combines the nurture for a joyful mind and the tool to make the job done.

I don't want to start a flame-war (famous first words)... but (famous conjunction)...

I did a fair amount if meta-programming in Ruby. I even gave some presentations on Domain Specific Languages and have implemented a few in my time.

Scala's traits and type system give me the same kind of power that I have with Ruby's meta-programing facilities, except that the composition of the classes happens at compile-time rather than run-time. But the ability to use a trait the same way that a Rails developer uses acts_as_xxx is there in Scala.

The ability to dynamically compose dispatch tables exists in Scala as well. Scala's Partial Functions (blocks of pattern matching) can be arbitrarily composed at compile time or run time. Scala's syntax is very flexible such that:

myobj ! ('my_message, param1, param2)

is a legal construct and would dispatch a message to myobj. You can use the Scala Actors library for asynchronous, Erlang style, dynamically dispatched components (yes, you can change the dispatch table on a call-by-call basis). If you want lighter weight synchronous components, you can build those as well.

Put another way, you can have dynamic, proto-type style components in Scala without sacrificing type-safety.

I can't speak to your criteria of a language being big enough, but I do posit that Scala has the appropriate technical resources behind it. I agree with your assessment of Groovy. I do think, however, that you may not have a complete picture of what's possible with Scala.

You may also be interest in The JVM Languages group.

Thanks (again),

David

Lua is big in the gaming world, and I see that an interface exists to Java:

Lua Java

I am always amazed at how more productive a good dynamic language makes a programmer...still, the jury is out on large systems where static type info is supposed to give you the edge. But these are exactly the dogs that need more dynamic code ;)

Very intersting post. The idea that a person can maintain a 500KLOC program single-handedly is pretty far-fetched. I am currently working on a code base that size, and our team consists of something like forty top-rank software engineers. Note: that's lines of Common Lisp, which is of course a lot less verbose than Java, so we are not comparing apples to apples exactly. And we use Lisp macros, which also helps save lines of code. Why so big? It's an airline reservation system and there are just a whole lot of things that it has to do.

The Lisp machine software (what Symbolics called Genera) could be looked at as one huge code base. Of course it was split up into pieces, which were split up into pieces, so it's hard to draw a line around some set of functions and methods and call it a "program". There must have been many millions of lines of code. We had a large team of awesome hackers developing and maintaining it.

I think having a Common Lisp running on the JVM, or the CLR/DLR in .Net, would make a lot of sense. What we have now is Armed Bear Common Lisp, based on the JVM. (See my survey paper at http://common-lisp.net/~dlw/LispSurvey.html.) I've never used it. I have seen some benchmarks that make it look rather slow, although that might not matter if you're spending all your computrons in the libraries, which is very possible. On the CLR/DLR front, there was an attempt called IronLisp, but they backed off because it was too hard and wrote IronScheme. Again, I know very little about it; you might want to track it down. I see the comment from "Igor" about sisc-scheme; I had not known about that. I have heard about Clojure but don't know much about it as I have been focusing on Common Lisp.

I don't think refactoring makes the size of the code greater. Done properly, it has the opposite effect: you do something once, rather than over and over.

Regarding not being able to fit your program into Eclipse: have you tried IntelliJ? It scales better. At BEA, where I used to work on WebLogic Server (a large Java program), everyone used IntelliJ. Attempts to use Eclipse seemed to founder on the size of the WebLogic Server code base. (Sorry, I don't know the number of lines of code.)

-- Dan Weinreb, co-founder of Symbolics

Why did you obfuscate your code from yourself?

The fundamental you are missing is readability. Martin would suggest breaking up a large method into smaller ones because is makes it more readable.

Consider this:

String uc = "blah" or
String userComment = "blah".

Which is easier to read? Which is longer? Your programming style sacrifices readability against all others, including size. So you obfuscated yourself. Size doesn't matter. What matters is how long it takes you to understand a hunk of code. Usually, more lines means longer to understand, except for self obfuscators such as yourself.

You are a self-deprecating and self-obfuscating programmer. You are in a bad funk. I would recommend health care as your next career. Challenging but not to dynamic.

Steve wrote that "folks who wonder about how I could (re)write such a giant system without any "static" type annotations: http://tinyurl.com/26n2zq".

As a co-creator of gradual transformations from dynamically typed to statically typed languages (see DLS 2006), I am perfectly aware of this idea. The problem is that Ecmascript's implementation of the idea is broken, unsound. So whatever advantages you would get from a sound system, such as ML's or even Java's, you won't get from ES 4. (The most you get is C's notion of types.)

Nobody has tested this notion on a big style. At PLT, we have ported some 10,000 lines to a sound system. Works like a charm and I firmly believe that soundness was important. When you get a type bug at run-time it's not in the typed part of your code. With Ecmascript, this is not true. You will need to search the whole thing.

;; ---

One more comment on size: my experience in teaching a programming course with free choice of language is that Ruby, Perl, Python beat Java and C# by a factor of 3, C and C++ projects usually fail (due to the lack of experience), and Scheme projects are always shorter and more complete than all others. This may or may not reflect the personal bias of the instructor, though.

-- Matthias Felleisen

"... I'm a bit skeptical about building a large production system in a weakly-typed language, for lots of reasons: refactoring tool support..."

"If you feel I'm wrong, please don't mail me with a long rant; publish your thoughts as a blog somewhere!"

Or 'somewhen' apparently.

I just think that the language flexibility has more impact over code bloat than static typing.
Ok, type things has some verbosity but C# 3.0 and F#, while still static, have very aggressive type inference mechanisms.
Comparing C#, especially C# 3.0, with java and putting them in the same group is just not fair. The new C# 3.0 is closer to ruby in verbosity, and is a Complete Language (you can write DX games, use pointers, windows applications, performant math libraries...).
About language simplicity vs code simplicity, I absolutely go for code simplicity. Having a complex language (as C#) has huge benefits over code simplicity, and you have to learn a language only once, while simplifying the language means you will deal with cumbersome techniques and libraries different on each project.

Thanks for writing another blog on a very important issue. Actually, I don't think it's about code base at all.

It is a language issue. There is a line drawn in the sand. You're trying, post after post to find good arguments as to why scripting languages in general - and js in particular - are better suited for most, if not all projects.

The size angle is very good, and was a reasonably good choice to get people thinking.

The problem seems to be that things that I take for granted, when you bring up this kind of discussion; That you have modularized your code as well you possibly can, and that you have the ability to actually compare the experience of coding very large projects in Java and JavaScript; is completely lost on most people. (Not the Scala people tough. :) I'll definitely try it out over Agnosticmas).

What I would like to see more of, is people commenting on language issues who have at least two years of experience with each language and having a couple of large projects under their belt.

I'm mostly referring to having experience with JavaScript, here btw :)

For the 400 pound Gorilla in the room is actually that it is kind of hard to describe why it is better to code *anything* in js (or actually any modern scripting language at all) instead of Java/C#.

The codebase issue is good, but as many people pointed out we need firmer evidence, smoking guns, silver bullets and a lot of patience :)

It is a big issue. I'm not in the honeymoon phase any longer (as far as I know), but I do my laps of pain in Eclipse fro customers every month, and as soon as I can get off to doing basically the same thing, in just Kate, or vi, in (incidentally) less lines of code, I'm as happy as a (grease)monkey.

We need a plan.

If your use of design patterns makes your code base bigger, it's because you're misusing the pattern, or you didn't need it in the first place. I refactored a bunch of cut and paste code a couple of years ago using the template method pattern (without previously knowing of this pattern), and the resultant code base was absolutely smaller

My favorite language is Forth, and it addresses the problem of code bloat quite aggressively. Forth is not only terse, and allows to compress code in every way you can imagine, but it's essence for this is that the compiler itself is extensible. Forth was strong in the early MUD game development, so it fits for that purpose.

BTW: I noted that one of the requirements you list for the Java replacement is "must scale for large programs". Hey, NOT! It must NOT scale for larger programs (large in the number of LoCs), because that will keep you disciplined, and experience the stopgap point where your program falls apart much sooner.

There was one comment which I can only second strongly: Keep your program around 10kLoCs. That should be some sort of upper limit - less is better, but more is awful. Think about the way your inner program logic works, and restart from scratch if you find too many verbose tasks.

My Forth system contains a GUI library, similar (I'd say superior) in functionality as e.g. Motif, but with just about 10kLoC code. It's build directly on Xlib and uses mostly gdi on Windows (and OpenGL for the 3D canvas), and that's the next "anti-pattern": Avoid unnecessary abstractions. If you interface some external component, do with the lowest level there, and do it right to what your task actually is. Interfaces impose some logic onto your program, and the higher level the interface is, the more likely is the logic the wrong choice.

The downside of Forth is that unless your system of choice comes with the libraries you need, you either have to write them yourself, or you have to interface with foreign linkable libraries (e.g. written in C) that don't fit well to the logic of the main language. There's no abundance of public abstraction stuff as with Java, and it's partly because Forth isn't that widely used, and the other part is because these abstractions are shunned, anyway.

The other disadvantage of Forth is its strange syntax ;-).

Very nice post. You write well and you seem to like writing, and I suppose you code well and like coding. But it's this creative energy that is ultimately the source of your problems. Write less!

But writing less requires a basic change in your behavior, which is not as easy as loading the next silver bullet into the chamber.

Perhaps there will be benefits from a language change and other refactorings, but "features" must be considered as the primary cause of source code. Rip some of them out!

As good as this post of yours was, the followup on the results of your rewrite will be even more education for us. Thanks.

I think this blog entry is quite possible an elaborate excuse to make a joke about a spitting zoo camel.

Here's the problem with Mr. Yegge's thesis. He identifies lines of code as a cause of complexity rather than what it really is -- an effect, a symptom.

The solution to complexity is not reducing the number of lines of code. Neither is taking an ice bath a solution to high fever. Nor does printing Finnegan's Wake in a smaller font make that work a more manageable read.

After all, if lines of code were a *cause* -- not an effect -- of complexity than Mr. Yegge should counsel everybody to rid their code base of all comments, make all variables public (eliminating accessors), and make heavy use of the ternary operator.

It's amazing how many people don't seem to understand what was said the main entry.
To those who keep saying: "You need static typing to manage a large code base." The point is the other way around: "Static typing makes you try to manage a large code base."
I haven't been in the industry anything like long enough to be able to take an authoritative stand on this issue but from what I've seen of Ruby, JS, and other dynamic languages this seems to be the way to go. From my own experience with a small-medium sized C# code base at work the points about code size and static typing ring true. I've found myself several times wishing I could call a method based just on its name without resorting to cumbersome reflection apis. I've got a few days off here around Christmas and I plan to play around with Ruby and other dynamically typed languages. Maybe I'll even start a blog and post my experiences.

Oh, one more thing. For the people making condescending comments about how he just needs more modular code and several sub-projects are simply making his point about code size (more modular just moves the dirt around) stronger.

Brad:

> make all variables public (eliminating accessors),

Good point actually. Why isn't there an @accessible modifier to a member that automatically creates the accessors? Would save a lot of code and not lose the functionality. Java is in dire need of a decent preprocessor.

> and make heavy use of the ternary operator.

Java semantics suffers from the distinction between expressions and statements; see the crazy way to only run the initializer of a final variable within a try.

Code size is a symptom, one of using a language with insufficient abstraction capabilities. This is just what the article says. (For an elaborate distincion between abstraction and mere compression see the recent stuff on weblog.raganwald.com)

Steve, the problem isn't the language, it's the programmer.

I cannot even begin to imagine how you got anywhere near 500,000 lines of code for any single-person project. For the past couple of years I've been working on a system in Java for a government agency here in the UK which does pretty sophisticated rules processing on XML documents. The codebase hit about 12,000 lines of code in the first release, including the hand-coded rules classes. Since then we have expanded the functionality, added a system which compiles rules from Schematron into Java classes and dynamically loads them and the codebase is up to around 15,000 lines of code.

The only way you can get to 500,000 lines of code is by simply not paying attention along the way. The way you dismiss patterns and other useful ways of thinking about programming gives me some idea of why you've hit this problem.

Get off your high horse about Java and take a good look at how you can develop your programming skills to keep you from getting into this mess on future projects.

But hey, I'm just an 'experienced programmer' and your post wasn't aimed at people like me. After all, god forbid that some of the experience I've gained along the way should have any bearing on writing software.

I almost always agree with you, which made today’s post doubly interesting. Thank you.

For the sake of argument, let’s assume that you have to write an unreasonable amount of Java code to get anything done. Let’s further assume that Rhino is exactly five times more expressive than Java is – that is, 20 lines of Rhino do exactly the same amount of work as 100 lines of Java.

Imagine that all your money is sitting in a big pile in your apartment. In pennies. Sucks, doesn’t it? Jars all over the house, full of pennies. Want to go buy fifty bucks worth of groceries? I hope you like lugging 28 pounds of pennies through the store. Everything you do with your money is slow and awful and stupid, and you will probably never even know how much money you actually have.

So you’re looking at this big mess of pennies, trying not to stand near it because you’re afraid the floor might give out, and you say, “God, what a mistake this was. I’ve got to convert all this to nickels!”

You’re right that nickels would make your life five times easier, but switching from pennies to nickels doesn’t actually attack the real problem, which is that all your money is sitting in a big pile in your apartment.

Switching gears: You say that you wrote 500,000 lines of code for your game, and that that’s way too many too maintain properly. So let’s say you switch to Rhino, and you knock it down to 125,000. Is the code really simpler now, or is it just shorter? It’s still expressing the same functionality and the same number of ideas, right?

Your game has way more than 500,000 lines of code. And those 500,000 lines of Java code are completely worthless without the code for the Java library and the code for the JVM and the code for the operating system that the JVM is running on and the code for all the god-knows-what that’s going on in your computer’s hardware. Oh, and your game is web-based, right? So let’s go ahead and count all the code that my browser uses to render it, as well as all the code for my operating system, and the code for both our TCP/IP stacks, and the code for all the networking hardware between here and there. I’m willing to bet that there are tens of millions of lines of code in play when your game runs, and that those same tens of millions of code (minus the relatively few that you wrote) are brought to bear on “Hello World”. It just really seems like the size of your codebase can’t be your real problem.

It actually seems like you might be the problem. I’m willing to bet that if you had access to the JVM code or to the browser code, you’d be hacking away at that stuff, too, because that’s just your nature. God knows it’s mine.

Maybe if you want to make that codebase maintainable, the real answer is to break off some of the back-end pieces and stuff them down into a black box somewhere you won’t be tempted to tweak them. Get those pennies into the bank.

If you can reduce the number of lines of code from 500,000 to 125,000, I'm not sure whether it should be called "simpler" but I'm pretty confident that it would be easier to understand and therefore to maintain, assuming that it was nice clean code (and not one of those silly examples written with single-letter variables and no whitespace).

In your analogy, you talked about the problem of lugging 28 pounds of coins around. Well, in nickels, that's a lot fewer pounds. I'm not sure why you mentioned the 28 pounds at all if the "real problem" is that your money is in a big pile in your apartment. In nickels, it's a much smaller pile.

I'm not sure I see your point in counting up the "millions of lines of code" in the operating system and so forth. The question is how many lines you have to maintain, don't you think?

@Daniel - Yes, intelligently reducing the number of lines will make the code easier to understand, in exactly the same sense that switching from pennies to nickels will make $50 less painful to lug around.

The Big Point I was trying to make is that if you are willing to compromise with yourself and not be a perfectionist about everything, you can reduce the number of lines of code that you maintain without reducing the number of lines of code total. You just tear off a big chunk of working low-level code and say, "I am not going in here again." It feels horribly wrong when you do it, but only because it's a choice. (In reality, we do it all the time, unknowingly forcing tens of millions of lines of other people's code into horrible contortions, mostly because we just don't know what's down there.)

Seems like there should be a limit not dissimilar to the Dunbar number, hitting which, the codebase should be broken up or rebuilt. My personal limit seems to be around 25K. Once a code base reaches this number, I start to notice a few things like engineers specializing in portions of the codebase, refactorings taking longer to apply and general tendency not to do anything drastic.

Ass that I am, I replied to the comments without noticing that Matthias Felleisen had visited my uncultured backwater of a tech blog: an event equivalent to Gregory Chaitin stepping in and correcting the math in an XKCD comic.

Matthias, I hope you'll forgive me for having grossly oversimplified my case, and for having wasted so much space for the sake of rhetoric.

I understand, of course, that ES4's type system cannot be sound. My decision to use ES4 (and back it, and promote it) is deeply pragmatic. The resistance to Scheme in the industry is truly astounding. The ES4 designers (Schemers to the last, if the rumors are correct) are doing the best they can with JavaScript. I believe, admittedly without any concrete practical experience to back me up, that ES4's type system may still prove quite useful. Of course it won't have the guarantees of a sound type system, but I think it can still help catch errors, clarify code and offer some purchase for optimization. Time will tell.

As far as blogging with more concrete detail about code 'size' as a proxy for bigger problems, so to speak, well - it's something I've attempted many times over the years, and I've never succeeded. The problem is that small examples fail to convince, and large examples are too big to follow. I've resigned myself to talking around the edges of the problem.

And thank you for your wonderful books and your awesome work on PLT Scheme.

By the way: http://www.amazon.com/o/asin/0262561158 - that was a great joke. I laughed from the first page to the last. :)

Well, I think your "minority" opinion is shared by thousands of Microsoft developers who are saddled with a 90MLOC Windows code base.

90% of what these devs do is maintaining it, day to day, year to year. A statistic: an average Windows developer writes 1500loc (yes, that's loc, in singles) per year, because most of their time is spent fixing bugs in what they aready have.

So the code size has a nature-imposed maximum :-) - at some point capabilities of dev org just tops out maintaining this max code base.

Anyway, I am not sure if fighting the code base size problem by replacing a less expressive language with more expressive one would work. The problem is that in a more expressive language there is higher potential to make a bug per line, and the bugs are worse.

I can write (prototype) a solution in JavaScript faster than I can do it in Java, and I can do it in Java faster than I can do it in C++ (let's say on Windows, Linux C++ development is stuck in a stone age, so it's unfair to compare Java and C++ there. I have a post on it: http://1-800-magic.blogspot.com/2007/12/c-development-on-linux.html).

But when it comes to shippable code, when you actually have to get the software to have reasonable startup time, performance, responsiveness, memory consumption, etc, etc, etc, you're suddenly hit by a bunch of problems that are extremely hard to debug, sometimes, impossible to debug on a higher-level language.

So then you start to introduce kludges, and your beautiful algorithm written so clearly gets its own memory management (the very thing you were going to avoid in Java), and parts of it migrate into native code for speed... etc, etc, etc. And what used to be 2000 lines in a very expressive language now looks more like 5000 in a mix of languages, and it's as unreadable and hard to maintain as it was before :-(...

I wonder which language Steve will use after ES4 to rewrite the game. My guess is that he will write a JMud DSL )))

Thanks for the post Steve.

The complexity problem you describe is an area where service orientation would shine. It can simplify the coupling between subsystems and provide huge gains in maintainability and reusability.

The issue of duplicated code and coding patterns can be solved via a domain specific language. DSLs can eliminate the bad code created by copy & paste style coding. Just surface repetitive elements of the architecture into a meta language oriented toward the problem space.

I believe that is the best way to combat code bloat & complexity.

I can't say as I'd call myself a professional programmer - though I have had occasion to write tools for work.

I've railed for years against software bloat.

Xen - the issue isn't complexity. The complexity of the application isn't going to change significantly no matter what language you chose to use.

The issue is maintenance and resource consumption.

I could have used Java for my tools - in one aspect,it would have been better than PERL or Python.. but I ultimately opted against it because it would take me 30 lines of code to get somethign done that would take only 10 in Python.

As a result, my tool was ugly to look at in operation (I should care about this ... why?), but was a darn sight faster than an almost identical tool written by our Java guru.*

When we opted to add features .. er.. additional tasks .. to the tool, adding a thousand lines of code to my tool was a lot easier than adding 4 thousand to the Java tool.

Bug hunting or making planned changes is simply a lot easier when the code is more concise. This leads directly to less time involved, and, therefore, to less cost of maintenance.


* why was there a second, identical tool? because the powers that be wanted others to have the tool, but didn't want one written in a "non-standard language like Python" running around the company. sigh

Complexity:

Fred Brooks distinguishes "inherent" and "accidental" complexity of a problem (in "The Mythical Man Month" - a must read for any serious software architect). The inherent complexity is the one you can't get rid off, even when you have a perfect solution. The accidental complexity however comes from the environment and how you approach the problem.

So if you are good and use whatever trick is necessary to solve the problem, the complexity should be more or less an invariant over different languages.

However, as David Pollak has already mentioned, domain specific languages are one key element to reduce the accidental complexity of a language, and meta-programming is important to create a domain specific language.

So choose your complexity-avoiding tool by how good it supports creating domain specific languages. And actually try to create domain specific languages where appropriate.

Hi steve,

i just read one of your blog:

Code's Worst Enemy, 2007 December 19, at
http://steve-yegge.blogspot.com/2007/12/codes-worst-enemy.html

Jesus, thru intricacies and excursions, from vacation preamble to the enticement of a mysterious game to code size theory and IDEs, with over 5k words, the bottom line is really just how java sucks and you were looking for another lang on JVM for your game?

Five thousand, two hundred, and fourty six words. I counted it.

If you said java lang bloats in the outset and i'll just fullheartedly agree and be done with it.

Though, i gotta say it's quite effective writing — sucking in your readers and give them a clout on head before they can get away.

Lol! ^_^

My prediction is that low level langs (or the qualitity of such lang, such as statically typed, or compiled), such as Java, C++, C#, will continuously to have smaller users until maybe reaching as little recognization as assembly today in 2017.

This is a inevitability due to hardware's exponential growth. This is particular already noticable in the past decade. Before 1995, people are still talking about C, Pascal, memory allocation, linked list, “data structures” in the sense of C's structs... these days, i bet that the majority of professional programers don't know what these are. They all work in PHP and Javascript or flash.

(majority = great than 50%. “Professional programer” is defined as those who write code as their primary income source. This includes HTML, visual basic, or any specialized script in different areas, say, 3D modeling...)

High level langs, such as Javascript (and lots more based on it such as Flash etc), Python, Ruby etc will become more and more popular.

Lisps will probably just hang on, maybe a bit more popular. I don't see lisp will ever become mainstream. The parens is a problem, and others due to age, but if i can just say one thing about lisp that i deem unsavable, is the cons business. (am aware many lisp experts here, also that the nature and brevity i stated my opinion about lisp problems are likely to make them think am idiot. To express my opinion about paren and cons will require a page or more... which this isn't the place. (however, the paren i've already expounded here: http://xahlee.org/UnixResource_dir/writ/notations.html ) )

Not sure about Haskell... on one hand it's high-level, but then it's compiled lang quality with the types thingy which necessarily bring it down.

of all langs i know, i think Mathematica beat them all by far.

----------------------
some comment on your essay...

you wrote long, and it is a very skilled writing. But i think the words/content ration is not very high, if there can be any criticism at all.

for example, you discuss vacation, your game, but these are just long leads to your main content, about code size and looking for a replacement for java. But even the code size theory, which you say are a problem, but i don't see much support or evidence. Code size is proportional to the complexity of the software in general. No matter what lang you use, its going to grow as your software grow. Games is one of the more complex software, especially if it is networked and graphical. Saying code size is a problem is like saying being too rich is a problem, or being too beautiful is a problem, or being big and powerful is a problem.

you say large code size cannot be solved by IDEs. But of course, when given a huge code size (and presume the code is lisp/mathematica/haskell written by top experts), really nothing can solve it.

So if i understand your essay's intention correctly, then your writing about code size, design patterns, IDEs... are just side effects of getting to your main point: how java is verbose and you are looking for a java replacement for your game.

... the above is just some random thoughts. Of course you didn't intend the essay to be a formal exposition... etc.

Xah
[email protected]
∑ http://xahlee.org/

From the dirt point of view languages with strong typing imply structured dirt like houses and bridges while languages without type system imply just piles of dirt.

I've got a unique perspective on code base size. Bascially I believe that a big code base at a microISV is a symptom of a microISV that's slow and inflexible. MicroISVs need to keep their code base size down because this forces them to think agile and be more flexible. I've blogged more about this: A huge code base WILL kill your microISV

Hi Steve,

your article is an interesting and I feel that at least some of the points are valid. Nevertheless, I find that I must disagree with the underlying premise. Granted, some languages may be more verbose than others. But in all instances you should never need to deal with all the code at once.

One of the most important tools in software engineering (and in many other domains) is "compartmentalisation". Gone are the days when we need to reimplement a linked list class for every new project. If we take the example of Java, a good programmer will leverage the base libraries provided in the JRE. If functionality is still missing, they will turn to open source or commercially available libraries. The only code which should need to be maintained is the code specific to the business problem at hand. Even for this code, much of it can be factored out into libraries or frameworks which can be maintained separately from the main project.

Computers are incredibly complex devices, yet nobody suggests that we should throw them away and replace them with something simpler. We can create these complex devices because we build on layers of complexity. An ASIC engineer uses existing libraries for implementing sections of a chip. A hardware engineer doesn't look at the internals of a chip but merely uses its published specifications to build a device which is more powerful than the sum of the individual parts. Object oriented languages allow similar compartmentalisation to be applied to the domain of software engineering.

Search not for a new language to solve issues with code size but rather break the problem up into manageable and distinct chunks.

Regards,
Dominic.

I'm really surprised that Eclipse indexing never finished with only 500,000 lines of code. It's my usual set-up in my workspace and never got such an issue.

As I work on this part of Eclipse code, I also often verifies that there's no problem to get indexing finished in a reasonable amount of time (approx. 1 min) with heavier setups:

1) All the source files of Eclipse which is now (eg. since 3.3) over than 3 millions lines of code
2) All the source files of Europa which is more than 17 millions lines of code...

First of all I would agree that code-base size is as big a problem as you say. However, I would posit that the size of the problem be more of a function of the structure/design of the code, than just the number of LoC. See.

Sigh.. javascript? scala? doom I say, doom..

Give Nemerle a try. It is what Groovy and Scala should have been.

Count me in that minority, too. But a half-million lines? You have my sympathy.

I've built GUIs and databases and virtual machines, and worked on one of the world's biggest codebases, and have never had to deal personally with that much. But years of MS-DOS coding (meaning tiny space and slow chips to run your software, and primitive tools to write it) and an allergy to of needless complexity have taught me that tight code rules, lose code drools.

I've never found choice of language to be an obstacle. I've coded professionally in FORTRAN, Pascal, C, C++, Java, Scheme, and more. It doesn't take that much sometimes to increase the amount of code you don't have to write. C macros can generate a lot of clean boilerplate if used well - look at what Boost.org does with them. You can build a nice object system with C macros. C++ templates are even more magical code generators, and Scheme and other Lisps are metamagical.

So sometimes I like to suffer a few lines of whatever I'm stuck in to write a Scheme interpreter and bust on out. From there I can architect my structures and logic in Scheme and customize my Scheme dialect to fit my architecture. And I can write my runtime in whatever I'm stuck in so at to leverage its runtime and provide interfaces to my colleagues left behind in Whateverland. I did that on a paid job once. I would have been fired if hadn't already quit.

Another way to try to get fired, and one with some advantages over an interpreter, might be to use an available Scheme environment to write Scheme code that takes Scheme descriptions of your structures and logic and translates them to code in Whateverland. But we didn't get fired for that, we got lots of rapidly-appreciating stock options for having successfully intergated a Whateverland virtual machine into one of the world's largest codebases. Thus enabling its continued metastatic growth.

In response to Snow:

Consider the case of a database system. Extremely complex, used by millions of users and required to be blindingly fast when accessing millions of records.

These systems are very large but maintainable as the design is good. Whether it's written in Lisp, Java or assembly language is not the deciding factor - the design's the thing.

I once worked on one of those beasts, and indeed it was huge and complex. And written in C. I never did grok enough of the code to judge the design, but it had held up to years of maintenance by thousands of programmers, so it must have gotten something right.

But it seemed the main reason it was still working was that there were so many of us programmers that none of us had to deal with more than a small piece of the beast, and every group had a very senior architect who made sure his group's piece fit into the big picture.

Also, the daily build was always followed by the daily regression tests, and breaking the build or the tests was a big no-no.

So again, choice of programming language is a small part of the picture.

When you write your own half-million-line code base, you can't dodge accountability. I have nobody to blame but myself, and it's given me a perspective that puts me in the minority.

500.000LOC = 65 years for coding for one person!

A quick look at the matrix from Steve McConnells book Rapid Development says that the schedule according to the data he collected from the industry for business projects 500.000 LOC would require 780 man-month, that is 65 year.
The data is collected from:
1) Software Engineering Economics (Boehem 1981)
2) An Empirical Validation of Software Cost Estimation Models (Kemerer 1987)
3) Applied Software Measurment (Jones 1991)
4) Measures for Excellence (Putnam and Myers 1992)
5) Assessment and Control of Software Risks (Jones 1994)

Either all the reports above are wrong or your code base stinks with duplication/autogenerated bloat code. I suggest you post fact about how long time it took to create that codebase of 500.000LOC and how many LOC that are duplicated using a duplicated finder tool.

Its weird, but you don't have a choice except Java (or C#?) in commercial programming world. I have played around with Ruby, Haskell, OCaml etc. Back in the days i used to be into C with some basic mixed assembly. There has been so much invested in Java for past 10 years by firms, that even if one wants to take a hard look at code and say enough is enough you cant do it. You simply cant.

I hate to say this to all Ruby people, but i don't like it at all. Scala ... now thats something.

Also, there is a difference between "what "kind" of code you write", "How you write" and "How you maintain". So not all code written in all companies is bloated (Leave out some of the financial firms where they have to do quick fixes ;)); And honestly, I don't really care / zealous about what anyone says. I liked Java back in the day because of its simplicity, and because i had to do a lot C with pointer arithmetic.


Even James Gosling said (and rightly so, despite the fact that he tried to play down his statements or worst recant it back) "Sometimes you have to take a hard look at a language and then again start with a clean slate" (i don't quite recall the words right now but yea (especially since its 12:33 in the night :)).

Regards
Vyas, Anirudh

Nice post, very enlightening. Just one question, why are you looking to get stuff that works in the JVM ? If there are languages out there that reduce the code bloat why not run them "au naturale" without the "taint" of the JVM. I say taint in the nicest possible way as I dont really hate java.

Hallo Folks. I am an avid user of blogs but just beginning to use tools to send comments to my favorite blogs.

I am seeking some hints on where to start building code to deliver the same text to multiple posts on a single blog or on multiple blogs. I don't want to send to every thread - just the ones where the context fits but sooner or later I wind up sending something to each and every post so I want to build robustly. Anyone found a way to send the same messages to many posts with code doing all the work?

+1 Scala

It seems to me that oo languages (in particular) make refactoring expensive. So much so that it is often put off for too long. It isn't just changing one function any more, it's changing a whole object tree of dependent functions (i'm assuming here the changes are required because the design has problems - if it doesn't why are you changing it?). Because of that cost most fixes don't take the holistic view and end up polluting the design purity it may have started with.

Good OO design is just harder than decent procedural design. It's easier to make more limiting mistakes.

But FWIW I find most of my refactoring (as opposed to 'rewriting') efforts involve deleting lots of code and moving bits of it to shared areas. Infact I wouldn't bother if I wasn't - what would be the point?

Hi
Code's Worst Enemy which makes a pretty sensible claim that his half million line of a game is too big. I think that it is based on the languages. I also indicates that wants to continue the code's worst enemy.
================
henry
There are a lot of sites out there showing book video. BookVideoTV, BookTelevision and of course CSPAN, but I like how BN.com and Reader's Entertainment TV have specific genre channels and original shows. There's just more to see and I can be specific in what genre I'm interested in. Anyone else watch online tv?
Reader's Entertainment

I don't know if I agree (or get your point).

As far as I'm concerned, zby had it right -- what about abstracting the majority of the code away in libraries? Isn't that what software development is all about anyway? You're already working on a codebase of millions of lines as soon as you start a new Java project. The only reason you don't consider these lines part of your application is that they are nicely encapsulated and abstracted away.

So in the end of the day abstraction is always the solution to your code size problems.

But, I'm training for decent athlete status myself.

I don't think you're going to read this, but if by chance you do, please drop me a line.

I started reading the article and liking what I was reading, it talked about something that I, a young student eager to learn and be a better programmer, always fought against: too much code. Not that I was taugh to see it as problem but nevertheless I felt it was one I could try to eliminate.

But I was disappointed to find that the article is nothing more than a bashing on Java language and Java programmers, and the fact that you're game is made in Java doesn't really justify it. If it was written in C++ would you just end bashing C++ language and C++ programmers?

I really felt like you ranted away from the point and made this just another "I don't like Java and Java programmers are dumb" post.

I would like to see a post on code size that could talk about the problem without a specific language in mind, a post that talked about the problem and did not hide it in the language X or Y features, or the lack of them.

Rhino may shrink the code base but the run-time will still suffer from Java's memory bloat. Here is an example from a 2003 blog post:

"First, consider Java's String class. Everytime you create a new String class, you are paying:

* 12 bytes for the String's object overhead
* 12 bytes for the String's member variables
* 12 bytes for the character array's object overhead
* 2 bytes for each character

So "hello world" consumes 58 bytes of memory, not counting the reference to the String object itself.

Now consider Java's HashMap class. For every entry in the map, you pay:

* 5 bytes for the reference to the Entry (taking into account the empty entries in the Map, given the default load factor of 75%)
* 12 bytes for the Entry's object overhead
* 16 bytes for the Entry's member variables

Now what do you think happens if you create a HashMap whose keys and values are both Strings? A 5 megabyte file will bloat into 50 megabytes of RAM!"

I thik Stevey is missing an important option... You are sometimes better off to make a program that *generates* souce code for the end program... You are then not tied to the weaknesses of the programming language because *you* are creating the programming language. So for example you could add #define and #if and even stuff c++ doesn't have like #while or #for

You can also create your own GUI for developing code, so for example a split screen, where the left side is a custom form that changes appearences depending on the function in the current line of code, and the right side is ordinary text view... so for example when you wish to code command to "open" a file, you get browse button, combo boxes and full documentation to optionally help you write the code.

Interesting that you mention, that you did a game on your own.

Because nowadays popular games are often created by huge teams. These enourmous costs demand that games have to be mainstream.

So some people which back the old days where 1 person could do a game.

I don't understand. You started with bloated language and said you made massive engineering mistakes in the past. You wrote the game 10 years ago, so I reckon that you must have made serious mistakes in your code for it to reach that big + that was 10 years ago. Things have improved now. I think you should stop blamming the tools. If you use the wrong tools for the wrong job, do not blame the tool.

Have you looked at REBOL? There is nothing else like it when it comes to reducing bloat. Take a look at http://re-bol.com and http://rebol.com .

Steve's right.

Personally I think C# is a large improvement over Java; it's not as concise as some languages, but LINQ, the var keyword, extension methods, lambdas and a non-broken generics implementation all help. Fundamentally, the C# culture doesn't think that concision is a dirty word.

Trying to master the C# type system, I took an interest in Scala, but I came back disappointed. Type erasure is a "broken window" that effaces all of the good things in the Scala type system. C# taught me what I could do with a decent generics system, and time and time again I'd discover that type erasure would get in my way. Other 'clever' ideas in Scala, like the way constructors work, just seem naive to me.

My feeling is that any large system is going to have certain sections that need to be built with efficiency in mind, and in those places, static typing has major benefits. Large systems are always surrounded by an aura of things that are scripty, and would benefit from dynamic typing. A combination of Java and something like Rhino seems like a good idea.

In the big picture, I don't see the problem as being as much about languages as about programming paradigms. Object-oriented programming has been useful, and will continue to be useful, but we're still finding that it's too expensive to develop software. If we want to build systems that are cheaper and do more, we need to move towards approaches that are declarative and ontology driven. Although I'll never accept Lisp's BLUBiness (the fact that it rejects 1960's developments in parser theory) I'm personally mining the 1970's and 1980's AI literature for ideas.

A little irrelevant but: The entire time I was reading your post, from design patterns on, I was hearing "Little Boxes" (The theme song to the excellent showtime show "Weeds").

Rhino sounds great - when will it be finished? I downloaded the latest version and the file I/O is in the "examples" folder, with a comment that says "This is what file I/O might look like if we ever get around to it."

I have a question about understanding large code bases. I seem to have no difficulty reading and understanding code that is say upto about 10,000 lines. Beyond that, I seem to lose track of things - I have tried various things to try and understand whats going on - keep note of the call stack as I read through code, look carefully at the documentation to get the general flow of things, try and understand the code by first breaking it down into small chunks, understand that part properly, and then see how it fits with the rest of the the code.

This works in some cases - like for example the Php scripts that run mediawiki (there are about 200 of them). But once I get to something thats larger, it is very hard to understand whats going on - I tried to understand MySQL code and Apache code - and it was hard to figure out what was going on.

Would you have some general advice on how to read and get a general idea of what some particular file is doing quickly? It seems that I am always working with large codebases - not being able to deal with the scale is slowing down my work.

Thanks!

Why do people who have emotionally abused me tell me that I am my own worst enemy?

That was a really good article.
Being a Computer Science student at a university that teaches java, I thought it a bit odd when one of my lecturers said "if you're just one guy, you want to use c++, and if you're thousands of guys, you want to use Java, and if you're in between, you want to use Python. So there's almost never a reason to use anything but Python."
But I guess what you're saying not only makes sense, but encourages me to make sure my code bases (when I, you know, actually get any) remain small.

Given I only really have *any* experience with Java, if one were to try and keep a codebase small, what techniques are you suggesting? Ignoring the fact that I wouldn't be using Java, that is :P
Is it a case of be sure to generalize everything so there's no duplication? Just plan the darn thing better, writing the code by hand?

That was one thing I thought odd about the difference between DrJava (notepad with colours, compilation and commandline runtime built in) and NetBeans. It seemed I could learn Java and write it myself, or I could learn to use NetBeans. Do things like NetBeans not detract from the ability of a programmer, as opposed to a computer user?

Thanks
PAz

> Unfortunately, a JVM language has to be a drop-in replacement for Java

JVM lisps are mostly just lisps that happen to run on the JVM. Clojure, on the other hand, is great for java interop. It doesn't shun standard java classes and I would recommend it for your next project (you appear to have chosen Rhino for this one).

Great Post. It's rare I read through the long ones, but I really enjoyed it.

Nice article. I know the pain... You might also want to check this one from Michael Feathers:

http://michaelfeathers.typepad.com/michael_feathers_blog/2011/05/the-carrying-cost-of-code-taking-lean-seriously.html

In LEAN terms: code is inventory, and should be minimized!

Well, it's been over two and a half years since you wrote this post. How did things work out?