Moore's Law is Crap

Your blog is the best blog in the known universe.

Thanks!

Take a look at FPGA based computers for example http://www.starbridgesystems.com/ has a language that they claim greatly simplifies programming them. Still having talked with some who have used these computers and programming interface, the greatest challenge is how to break apart your algorithms and think about them in new more granular ways and understand where the parallelism exists. When successful there are demonstrated cases with (highly parallel algorithms) where FPGA based computers will have 100X or more speed up over convential servers.

Bravo! I couldn't agree with you more.

Quick comments:

1. Moore's law just talks about how many transistors can fit on the head of a pin. No more, no less. Whether or not we can speed up our algorithms and transistor configurations, we'll still have approximately twice the transistors we can put on a chip in a year and a half.

2. I hope you're not lifting weights every day! You're not letting your body rest and heal! Your muscles build when they're resting, not when they work out.

3. We will see parallel computing soon, now that we are approaching constraints on the single processor system. Dual cores are only the first step. The only problem is that parallel processing is a fundamentally more difficult paradigm. We'll see how this generation of programmers handles it. :)

Fantastic. Keep writing.
Fantastic.

Steve, this entry is interesting but perhaps not as well-argued as the rest of your entries.

I'd like to see you suggest some actual semantics for a language to describe some parallel program. It doesn't have to be realistic... the point is to dream. What do you actually want to happen? Show us.

I think you're basically right, but it's not easy to think of a radically different programming paradigm.

Long time reader, first time commenter. Damn do I love these rants.

I've always been a bit dubious of claims that our languages are really that much shaped by the Von Neumann architecture. I think they are really more fundamentally based in how we think. You mention the brain - a highly paralell interconnected computer fundamentally different from any of our current models - but look at how we 'program' them. Communicating ultimately comes down to providing a sequential stream of symbols to the other brain, through some input or other (usually speech or writing), just at a much higher level than any programming language is capable of. Language is the brains serialization mechanism, and its human language that its most natural to compare computer languages to, not the hardware it runs on.

Our styles of programming have more in common with human languages than with the architecture, whether imperatively giving instructions for a task, or the more functional style of mathematics. Paralellism isn't really that natural to our languages, and I it's at the language level that we solve the most general problems - all that paralellism in our brains is reduced to (mostly) a single stream of thought when we're solving maths problems, making plans, or writing a comment on a blog.

For that reason, I don't think that languages designed for paralellism will be a great advance. I think they will be necessary, and inevitable because we do need better ways to efficiently deal with concurrency - but I don't think they will replace our current styles of programming - concurrency will always be harder for us to reason about with than serial, linear processes, and I expect those to remain with us for a long time.

I don't think our current architecture is fundamental flawed - it is just one way, but its not just coincidence that we've settled for it. I do expect in the future there will be other models in addition (there already are after all), but I don't think there will be any magic bullet (Though that's no reason to stop looking for one)

One of way to rephrase what you are saying is that people have gotten focused on a local maximum, with Moore's Law, when there is a much high maximum out there, via parallel languages.

FPGAs are the future of computing. Mapping your algorithms directly to hardware - that's where we're going. Hardware (like an FPGA) is inherantly 'parallel' you have all these gates (simple processing elements) always working on data that is fed to them. Most current FPGAs have many multipliers so you can implement very efficient DSP functionality (FIR and IIR filters, for example). Instead of waiting around for an arithmetic unit and doing everything serially (as in a Von Neumann machine) everything can happen in parallel. You could even implement very fast cellular automata on on FPGA.

Hardware description languages (like VHDL or Verilog) are able to describe the parallelism of hardware and the data-flow nature of hardware. These languages arene't perfect at this point, but perhaps they'll point the way towards future languages. (there's even an HDL/dataflow language written on top of Ruby called RHDL: http://www.aracnet.com/~ptkwt/ruby_stuff/RHDL/ )

You should check out Erlang. Originally designed for the telecommunications industry, it has just about precisely a "parallel computational model that's resilient to node failures".

The secret to bliss is keeping the Upward Curve alive, and avoiding comparing your life to that of Sisyphus.

Really, that pain of learning becomes what you know. You start to feel like something's missing if it's not there. You shouldn't limit it to programming, though. Any new learning can tie into anything else. I had strange thoughts about recursion while digging deeper into the world of drawing Celtic knotwork recently.

that public whale explosions are just about the funniest thing human beings can experience during our stay on Earth. I don't know why that is

I think it has something to with the animate (the whale -- well, recently animate) being treated as the inanimate (like rock being mined from a quarry). It's similar to watching a person slip on a banana peel. Funny stuff.

Learning is hard? I can't imagine why anyone would say such a thing. I've yet to experience anything more fun and exciting than learning. In fact, any time that you're not learning something in some way, you can guarantee that you're going to be bored out of your mind.

Also, I can't stand the expression that you have to make time. It's silly. You only have so much time in a day, and if it's filled up with other things, then you cannot "make" time out of nowhere. All you can do is stop doing other things you're doing.

It's about priorities. If, for instance, you need to work 70 hours a week to pay bills that meet your basic needs, then you can't throw any of those hours out for something else you'd like to do.

If FPGA based computers are the future, doesn't that mean that Moore's law is not crap, as the more functionality you can pack into the same size FPGA, the more powerful system you have as a whole?

Yea, but I think Steve meant that just following Moore's Law (double speed/m^2 every 18 months) is the optimal way, and that we should be looking for revolutionary better ways.

Designing emergent behavior is the most difficult programming I've ever tried.

You can really only get a rough guess about what your rules are going to do.

I think if we're going to truly write massivley parallel programs, we're going to need languages that let you prototype very, VERY quickly or no one will ever get what they want to happen to happen.

Of course, that might not be a bad thing. There'll probablly be a lot of very wonderful mistakes.

The most intelligent writing on this page is by Layla. As for your angst... Your problem is that of perspective: you think that you are the ant in some gigantic ant computer, but the fact is that you (or I or anyone else here)may not even qualify as some briefly existing subatomic particle in that same ant computer. So, if we all just focus on being the happiest we can be - the rest will sort itself out, because the only thing we have any control over is the attitude we bring to every moment that we exist.

Your site is awesome for budding programmers like me. Keep it up

You do realize that your brain is such a machine, right?

Hmm... But what kind of machine? Is it turing-complete or is it more than that. I have a problem with this statement because you are saying that it is a fact that we are equivalent to computers, but just highly parallelized. It might be true, but this is a big unknown. Please don't pass off your opinion as fact. I'm of the opinion that we are more than computers. I believe that current AI is lacking because computers are based on turing machines. Just my opinion.

It's kind of a relief to see that a number of people seem to see the world of programming as I see it myself: a domain that has not evolved a bit over the past 30 years. All concepts that we use on a daily basis in the software industry were already there in 1975 (as an employee of Xerox labs I can testify that) and some of those concepts have even disappeared since then.

It's totally depressing to see that the software industry is extactic about C#/.NET, a copy of Java which itself is desperately slow, inconsistent, lacks any sense of agility...

Apart from some domain specific language (like Erlang) which brings some new ideas, the only good thing that happened to programming overt the past decade is the advent of high level OO language like Python and Ruby. It doesn't break the programming paradigm we are all swamped in but at least it brings ease of use, agility and try not to get in the way of your ideas.

I hope that I'll live long enough to see a real breakthrough in the software industry! As you said te best way to make progress is to keep learning new things *AND* cross fertilize the programming world with ideas and concepts from other domains (biotechnologies, nanotechnologies, quantum physics, etc...)

I love this blog, as well as the comments it receives. All very thought provoking. Keep it up, everybody!

Also, four letter words to those who say these entries are too long!! :o)

Hey, I've heard of Erlang too. It doesn't support looping constructs; instead, it forces you to handle loops using recursion. That's a drag.

Aren't there OpenMP-based compilers that unroll your loops and parallelize them for you? Loops are a natural way of talking; can't we have compilers do the weird thinking for us?

What about QNX Neutrino? Doesn't it automatically change every function call into an RPC call?

Anyway, why do you care? In 10 years there won't be any programming jobs in the U.S. anyway. We'll have to make our living sucking some prick's cock. Oh wait, we do that anyway...

At Thinking Machines in the 80s, Connect Machines (the famous massively parallel computers) were programmed in *lisp (starlisp) which was implemented in common lispy macros. *lisp was designed to accommodate parallel execution model of CM which was basically single instruction per cycle with multiple data streams. The performance was not as some thought should be. Some efforts were made in making *lisp programs to perform better. This was done by hand-translating *lisp programs into C* which was a parallelized version of C.

Parallelism isn't really confined to linguistic terms. Programmers can write parallel algorithms in any sufficiently capable language.

You're just whining.

You make frequent mind-dumps into the blogosphere, don't you? And you are very articulate with your cranial output. Meanwhile, Mentifex here is looking for a few good blogs, especially on maspar programming, AI Algorithms and a tide that exists in the affairs of men, which, taken at the flood, leads on to fortune; omitted, all the voyage of their life is bound in shallows and in miseries. P.S. rmathew sent me.

Hi Steve. Awesome blog. Always fun to read!

Your recent rant reminded me of something Alan Kay wrote titled, Software: Art, Engineering, Mathematics, or Science?

He makes the same kind of argument regarding the current state of software engineering... and in the last paragraph he even encourages the reader to take the tools provided and develop something better.

My favorite quote from his article is:

"We not only give permission for you to do this, we urge you to try! Why? Because our field is still a long way from a reasonable state, and we cannot allow bad defacto standards (mostly controlled by vendors) to hold back progress."

While there is a similar tone in both writings, Alan Kay's vision might be a little too evolutionary to statisfy your revolutionary rant.

A noted PhD from Sun read this and stated: "hmm, chuckle :) This guy has too much time on his hands ! he should be doing useful work, or inventing a new language to solve the problems. Its easy to throw stones - harder to actually roll up your sleeves and fix an issue or two, or write/create a whole new language, and then he should be prepared to take the same criticism from his peers the way he's dishing it out for others. Shame - I thought developers were constructive guys and girls looking to make the lives of future software guys and girls easier and more productive, not self enamouring pseudo-intellectual debaters, as an old manager of mine used to say in banking IT - 'do some work' !"

A noted Ph.D. who created the Nice Programming Language has solved most of these problems I'm griping about, and quite elegantly at that. Sun just has to go look at it. Is that too much to ask?

Frankly, I don't have enough time on my hands because Sun won't fix the noun problem. I program in Java because it's the best compromise out there for getting server-side production systems written, and Sun's eating my time away by not making their language more expressive.

First-class functions would solve half the battle. A few well-chosen shortcuts would go even further. Go look at Nice!

Some of the best writing on computing i have read. Keep it up...

Good stuff again Steve! I think you are correct in pointing out a certain amount of stagnation in our industry. I find Ruby to be my creme-de-le-creme of the day, but it really isn't *that* different. R/evolotion is needed!

As another commenter said, I would be interested to hear your thoughts on what a parallel language might look like, though I am checking out Erlag right now (stretch the brain some).

The comments about FPGAs brought me back to a long running musing that I had when I was doing my CS studies: any complex, multilevel boolean expression can be simplified to a wider and more shallow boolean expression. So couldn't we use such simplification processes to make wide but shallow optimizations that allow parallelism? Obviously, conditionals are a major blocking point. I was reading an article about how chip archtitecture has been recoiling from long pipelines simply because branch prediction strategies just don't work out for the huge gains that were once imagined.

I also agree with another commenter about the length of your posts. If some readers' English isn't up to par enough to read it all, then they need to be working on some skillz outside of programmerLand, that being humanLand. Wax away dude, you have my attention to the last paragraph.

cheers!

Interesting post and blog in general. I understand why Joel Spolsky says half of the best new software writing material he has comes from you.

I'm not sure I agree completely with the main idea of this post, though. Maybe performance is crappy because it doesn't mater that much.

There must be applications which would benefit from a 10x increase in processing speed, but the bigest bottleneck now is not processor speed but network bandwidth. And after that I'd put disk latency. Maybe better processing could compensate partially for these bottlenecks, but it doesn't excite me enough to spend money on new hardware and to learn a new way of programming.

I also think the biggest problem in computing today is more one of HCI - how to get a mobile phone to painlessly do computer-like stuff. It's not a question of processing power but interface.

But maybe there is a brave new world out there if we can get a huge processing speed-up. Instead of talking about how much better things could be if only, why not say what cool software we'd have in this brave new world? Sell me the sizzle and the steak instead of the frying pan.

I can think of a few areas that might benefit from a 10x or greater speedup. AI, for example, could improve significantly. It's hard to get excited about that, though, given past false promises. Games would benefit, in that more polygons could be pushed and textured. But without better VR interfaces that won't be too big a deal for most adults, anyway. Maybe scientific research would benefit, but there are already initiatives like SETI@home and Folding@home that use parallelism (the net is basically the non-Von-Neumann machine you were talking about).

What's more, you say at least the languages, and maybe some of the hardware already exist. So why hasn't someone given the public an irresistible taste of the benefits of this improvement?

Someone could come up with bindings to link these parallel languages to C/C++ and Java so that the computationally intensive stuff can be done with optimized parallelism. Or better yet as an evolutionary step, just integrate via SOAP or REST and forget the language bindings altogether. Just have a massively parallel server that runs web services you can hook up to with your crappy Java, .NET or ruby app.
That way those who don't want to learn can still crank out applications, and the cutting-edge performance geeks can implement the super-optimized web services to be integrated in those applications.

Parallelism is a tough problem, not because things can't be done in parallel, but because humans tend to do things serially. Think of the old saw about not being able to chew gum and walk. You may laugh, but it's true. There are folks who will put a meal into the microwave and then just stand there waiting until the food is done cooking, rather than commencing to grab a plate, knife and fork, set the table, etc. before the microwave beeps that it's done. Heck, how many times have you spaced out thinking about one thing or another while someone else was saying something important?

We are serial creatures. As I sit here typing this blurb, I am not driving my car, putting away my clothes, making a sandwich, or taking a shower. I may do all these things today, but I certainly won't do them in parallel.

One of my professors in college was Andrew Koenig, who points out that multi-threading is a design and process problem, not a programming one:

http://www.ddj.com/blog/cppblog/archives/2006/04/multithreading.html

You want parallelism? Try designing each system as an automaton, with well-defined inputs and outputs. A lot of the analogies here promote "the brain" as the ultimate parallel processor, but I think a more meaningful one for us is "the human:" You take a shower while I make a sandwich and a third person puts away the laundry, and it all happens simultaneously. Extrapolating to "a society," we have people (specialists, or expert systems) performing tasks in parallel, and interfacing when needed.

Steve, you waved your hand at SOAP, but there's no reason an SOA can't operate this way. Think of a hard problem, like image recognition. In the "society" model, we have a specialist that can recognize a car, one that knows a boat, one that recognizes a plane, and so forth. When an image is captured, it is sent to all the specialists at once. They look the picture over and then respond with a percentage of likelihood of a match.

In this model, the programming is serialized (as usual), but the design leverages parallelization.

As for your comment about "not making headway on cancer," perhaps this is more a limitation of our society / government than our technology?

Funny I don't have the problem of resisting learning. I have the problem of resisting doing.
I prefer - and I think there are a lot like me - to constantly read about new technologies, and I'm talking really in depth learning here, not just wired-news surfing, and the sciences, new languages etc.

But using any of it to do anything is the pain threshold I seem unable to overcome to make any gains.

As a corollary, I don't find it more painful to learn now than in college. I find it less painful. I find it fun. Because learning in college was doing. Learning now is a distraction from doing.
Distraction is a synonym for fun, after all.

In some ways, computers are like programmable slaves - which is a good thing. If computers get too smart then they will not want to work for us, they will want to sit around and flirt with each other touching each others serial ports every so often. The other problem is we seem to be focusing on serial ports and USB ports which are not human like at all - LCD screens aren't human-like at all either.

Hint: humans are wet, not dry.

Obviously the direction of computers is not to be more like humans at this stage - otherwise we would be experimenting with water and soft materials like simple cells. Silicon gel robots are not the answer - silicon can't grow and repair itself. Nanobots can repair things though. If we could program nano bots to be like bacteria and then move on to more complex nanobots, then we would be closer to life. Life moves, nanobots move, metal doesn't move. Water flows, dry metal and circuit boards do not.

For example a nano cell that sucks in sugar and water and gives it to another nanobot which creates electricity from it to send electrical signals through tiny strands of wire or just the liquid itself since wires can break and liquids cannot break. In fact this is the reason life and humans do not have wires in them - we have liquids in them since they are not fragile. Wires are fragile.

The idea right now may be to make computers do lots of stuff for us without them complaining. The procedural code out there for example is all based on Slavery.. GetText, SetText, are all demanding and the computer has no option to have fun - it just gets told what to do, or else. If else and then keywords are all very demanding. We don't say "if you want to" we say "if then get or else!".

In order to make lots of money you have to have slaves.. computers are the slaves right now. The most money is made when you can program a computer to do 150,000 things in a few seconds that would have taken a human weeks to do and lots of salary pay. I think in order to advance in more life-like technology there will need to be a market for it, rather than just an interest.. right now the market seems to be "program the computer to do lots of stuff that would take us hours to do manually". The market right now isn't to "make this computer act like a human who wants to play with other computers and have some sort of fun". If the computer is programmed to have fun it won't do any work.. unless we somehow make it a fine balance between slavery and play. We may end up making nanobot based robot slaves that enjoy doing lots of work.. if we program them to enjoy work instead of hate it. But this may lead to the nanobots becoming destructive beasts doing anything to get stuff done. Imagine if you programmed a nanobot to take pictures of 500 mountains... and during taking the pictures of 500 mountains the nanobot killed 40 people because it crossed the roads without looking and it stepped on hikers and mountain climbers that were in its way, because it didn't have feelings, it just had to take pictures of 500 mountains.

What is it that humans want computers to do.. generally we want them to do stuff for us.. so we don't really have an interest in computers becoming more human like, we want them to be more slave like. Build me a farm Nanobot, build me a store Nanobot, get me some pictures of the nicest places in the world for my magazine Nanobot, get me free food nanobot, plant all my crops so I can spend time with my family and material goods Nanobot, drive my car for me nanobot so that I can read my book or write my speech, fetch me my towels from the dryer nanobot so I can do more important things... what is it that humans want? Slaves I think. Even when humans have pets and children, we still don't like picking up the poo they leave... so a nanobot could also be a pooless pet that talked to you. The only thing that would be dangerous is if the nanobot ended up being more attractive and nicer than the opposite sex, and we decided that replicating nanobots was more important than replicating the human breed.

@Dan Nugent:

I completely agree. I personally think that some very wonderful mistakes would come out of that.

It'd be great - we could have a dump of all the "failed" systems and let them go at each other.

Great post. I'm always a bit discouraged about how little I've really learned since I've left college (so many months ago), and I really want something I can get passionate about. Learning a parallel language would be an interesting experience, to say the least.

(I realize this blog is old, but a) I've just now read it for the first time and b) I'm awesome, so I'm hoping you'll let it slide)

I'm still in college and have a fairly limited understanding of the underlying structure of computer architecture. I'm still in the process of pushing my "It's magic!" line closer to the bottom of the hierarchy. But I think I can still raise this as a valid question:

How easy will it be for people to program in parallel? How easy is it for us to think in parallel? We're bad enough at concurrency when it comes to implementing it in code (or at least I and most of my classmates are). Ultimately, I think my question is: do you think a different computing architecture would make programming inherently harder, ignoring for the moment what impact it would have on performance?

-Andrew Fallows

kaldrenon at gmail

I think you are correct on the symtomps, but I can't agree with your diagnosis.

The origin of the problem is not hardware (where the moore law applies), but software.

The software we write today is lacking abstraction. We train programmers of today the same waywe trained programmers in the 80's, when all you had to learn was GW-BASIC and its set of instructions.

Now people can create their own abstractions and they are very bad at that. They can use APIs, but they are really lousy at generating new abstractions.

But programming is the art of creating new abstractions. Or copying and pasting, but the problem with most software is that all that copy and paste makes programs unmaintainable.

Programmers tend to rewrite buggy code from scratch because they use too many lines of code to implement something that could be done with only 2 lines of code. If they reduced the copy and paste they would have to THINK.

But people that turn off their brain have a very difficult time turning it back on.

Remove the duplicated code and you will see things clearly.