Atlas · Details

Digging Into Ruby Symbols

Digging Into Ruby Symbols — cover art: a kindly badger craftsman in a leather apron at a warm-lit workshop bench, holding a small wooden marionette of nested code-shapes up to a faint warm light that is turning the puppet's strings into glowing free-floating glyphs; a single small red ruby cabochon sits on the bench. 🤓

2005 · O'Reilly Ruby Blog · Note

“So Ruby's symbols are really a placeholder for grand things to come. Ruby is already a very powerful, capable language, but it has some weaknesses in its ability to process Ruby code at runtime.”
— From Digging Into Ruby Symbols, December 2005
Read the essay

© 2005 Steve Yegge. Originally published at O'Reilly Ruby Blog.

Author’s note

Probably only an interesting read to language geeks and designers. An article that does a deep dive into Ruby's symbol feature, concluding that they are essentially a placeholder for better functionality to come someday. They are not as powerful as Lisp symbols, and the code that you create with Ruby metaprogramming is not a data structure, the way it is in Lisp. It's a noble attempt to teach non-Lisp people some Lisp, but I think it's too ambitious, and winds up taking too many digressions.

AI Notes

Despite the Drunken Blog Rants banner, this is a patient teaching piece, written as a guest post for O'Reilly's Ruby blog at the height of the Rails honeymoon, with Steve on visiting-lecturer behavior. The wager is that he can show Ruby people what Lisp symbols are for without making them read any Lisp ("I won't show you any Lisp today"), and the route runs through a fee-fi-fo-fum giant whose four methods get generated in a loop, an aside on why large Java projects always fill up with generated code, and a mocked-up CODE_TEMPLATE construct Ruby doesn't have — sold with the quietly persuasive observation that inside such a template your editor could syntax-highlight the code instead of painting it string-blue. Fair warning for anyone arriving via the title: symbols don't turn up until a late section named "Symbols at last," and the essay knows exactly what it's doing to you ("We really will make our way to symbols soon, promise").

Steve left a note in the middle directing readers to "the important correction Jim Weirich made in the comments" — those comments now exist only in the Wayback copy of the dead O'Reilly page, linked from this listing. And the "hole" he hoped "will be fixed someday" is still open two decades on: Ruby eventually grew serious parse-tree machinery (Ripper, then Prism), but the template you hand to class_eval remains a string, and a Ruby symbol still doesn't name a node in any tree you can walk. So the forecast missed, and what's left is the useful part anyway — a working demonstration of why eval by itself doesn't give you code-as-data, staged in a language the reader actually knows.

Related listings

  • 2006

    Lisp is Not an Acceptable Lisp

    Four months later — the broader argument about the community Steve was writing toward here. The Ruby Symbols note is a friendly note from a neighbouring tribe; Acceptable Lisp is the same observation aimed inward.

    Listing · Stevey's Blog Rants

  • 2004

    Tour de Babel

    The pan-language tour the Ruby note sits inside. Ruby is the chapter Steve was actually enjoying in 2005; this is a focused piece on why he was enjoying it and where he could see the ceiling.

    Listing · Drunken Blog Rants

  • 2005

    Choosing Languages

    Same year — the column where Steve named Ruby as one of his two daily-driver picks. The Symbols note is the technical follow-up: what he liked, and what he wished it had.

From the peanut gallery

  • If you add the "magic" incantation to your eval line:

    BigMeanGiant.class_eval <<-EOS, __FILE__, __LINE__+1

    Then you will be able to step into each of the fee/fie/foo methods in the debugger. However, you won't be able to examine the value of the name variable in the debugger.

    However, if you change the definition of of the fee/fie/foo methods to the following, you will be able to both step into the methods and query the value of the name variable at run time.

    class BigMeanGiant
  %w(fee fi fo fum).each do |name|
    define_method(name) {
      puts name
    }
  end
end

    You still can't get the source back (but then you can't do that with regular defined methods either). If you are interested in examining the Ruby AST at run time (and even modifying it), check out Ryan Davis's ParseTree library.

    — Jim Weirich · December 29, 2005 04:36 AM

  • I checked out the parse tree package. Pretty snifty. I find it funny that he produces an s-expression format in Ruby. I'm not sure if that makes it easier to process from Ruby, since I haven't tried it out yet, but it's certainly convenient to read.

    — Steve Yegge · January 5, 2006 07:11 PM

  • Thank you for the enlightenment! I think I am finally beginning to understand what the lisp macro is all about! =)

    — Dae San Hwang · December 29, 2005 07:22 AM

Read the rest of the thread · 6 more

  • Reader comments from the original post on the O'Reilly Ruby blog, recovered from the Internet Archive.

    Very nice. If I can add a point or question, maybe ruby needs :X instead of just X for symbols because it has no compile-time vs. run-time distinction. Everything is done at run-time, but for metaprogramming it still helps to "intercept" code after it is parsed but before it is evaluated. In static languages, intercepting is easier because parsing and running the code happen at completely different times. For example a macro like:

    #define _N(x) x

    would replace _N(x) with x at compile time, before x is ever evaluated.

    In the language boo (http://boo.codehaus.org/), which is statically typed and supports a couple of kinds of macros (http://boo.codehaus.org/Syntactic+Macros), you can say for example:

    attr_accessor X, Y

    instead of

    attr_accessor :X, :Y

    because attr_accessor would be a compile-time macro (not a method called at runtime like in ruby). Thus any identifier passed to the macro is a type of AST node, such as a "symbol" (referenceexpression) or some other type of ast node (string literal expression, methodinvocationexpression, etc.). All the AST nodes are of course arranged in a tree like DOM, and you can manipulate it as you will at compile time.

    The syntax you mentioned for code templates (sort of wysiwyg macros), is similar to that proposed for boo: http://jira.codehaus.org/browse/BOO-95

    See nemerle for a language with even more powerful macro support: http://www.nemerle.org/ and http://nemerle.org/Macros

    Anyway, back to the point, when or if ruby gets the ability to represent code as AST, :X can be used to refer to reference expression node with the name "X", whereas X would refer to whatever some X variable stands for at the time the code template is processed.

    — DougHolton · December 28, 2005 08:52 PM

  • ri define_method

    — Christian Neukirchen · December 29, 2005 05:34 AM

  • Jim, that's awesome. Thanks. I knew about the __LINE__ trick, but for some reason only thought it worked in stack traces, not in the debugger. Silly me.

    I knew about define_method, and it's very cool. I probably should have included a snippet about it. As you know, though, define_method doesn't quite address what I was getting at, which was synthesizing new code trees on the fly (or turning existing code into an AST and processing it). In particular, it would be ideal to be able to manipulate the token stream and/or AST at lex time (for syntax), parse time (for code rewrites), compile time (for performance), and run time (for metaprogramming, e.g. doing multi-method dispatch or whatever).

    Ruby offers enough building blocks to do all of these things to some extent. I'm definitely going to have to check out Ryan Davis's parse tree library; it sounds great.

    In the spirit of full disclosure, I think even Lisp has trouble recovering the initial source code from compiled functions, unless you deliberately take steps to store the pre-compiled source somewhere during the compilation. It doesn't happen by default. But it's easy to dream about a language environment where you can pretty much go from symbolic source to symbolic data structures to code, and back, with lots of transparency and control at different steps.

    Anyway, thanks for the tips! You've instantly made me a more effective Ruby programmer by showing me a way to step through eval-generated code. I have to go check out that library now.

    — Steve Yegge · December 29, 2005 03:31 PM

  • I agree with Dae San Hwang, I had heard lots of talk about Lisp macros but never saw a good example of how their power surpasses what is available in Ruby. Thank you Steve!

    — David Koontz · December 30, 2005 12:46 PM

  • > define_method doesn't quite address what I was getting at, which was synthesizing new code trees on the fly (or turning existing code into an AST and processing it). In particular, it would be ideal to be able to manipulate the token stream and/or AST at lex time (for syntax), parse time (for code rewrites), compile time (for performance), and run time (for metaprogramming, e.g. doing multi-method dispatch or whatever).

    I agree define_method doesn't do the AST thing.

    Are you aware of Ryan Davis' ParseTree project? It will pull out the AST of an existing Ruby method or class (with some restrictions), and allow you to manipulate it for whatever purposes. Combine ParseTree with the Ruby2Ruby project to rewrite Ruby code, or combine it with Ruby2C to generate C code from your Ruby ASTs. The ZenOptimize project adds a JITer that will dynamically convert (simple) Ruby code to C at runtime.

    References:
    ParseTree: http://rubyforge.org/projects/parsetree/
    ZenOptimize: http://blog.zenspider.com/archives/2005/04/ruby_go_zoom_zo.html

    Great posts, BTW, I really enjoyed them.
    -- Jim

    — Jim Weirich · January 2, 2006 10:46 AM

  • Can't you just print the string being evaluated to stderr while debugging? Don't know Ruby, so I might be mistaken, but I had the same trouble in lotusscript, while creating huge macros on the fly. It's not perfect, but at least you can see what's going to be processed.

    — Patrick · February 22, 2006 04:21 AM

View the original thread →