RubyShell 0.3: Pipes, Parsing, and CDs
August 16th, 2007
It turns out, RubyShell 0.2 ran code twice when you asked it to run it once. Whoops…
Anyway, once again that’s only one of the several new features (okay, that one was a bug fix, too…) in RubyShell 0.3. If you want to skip the explanation, you can go ahead and download it now or look at the list of fixed/implemented issues.
RubyShell 0.3 brings a couple of brand spankin’ new things to the table. The first is the validation process.
Parsing/Validation
Validation is the process whereby RubyShell validates code as being Ruby. If it isn’t Ruby, then it’s run as a command. Originally, this was done using the defined? keyword, which, while kind of awesome, unfortunately actually ran the code embedded within it, which I had originally thought was not the case. Fear not, however! I have fixed this. RubyShell 0.3 uses ParseTree to make a real attempt to parse the code without any side-effects. In addition, it brings a restructuring of the validation so that validation is handled in its own class, with the evaluation class (the RubyEvaluator class) acting as the Parser class’s conduit to the validator. Multiline Ruby is now handled by looking for syntax errors of a particular kind, which makes it far more flexible than the previous versions, and means RubyShell should be able to handle most multiline Ruby code.
As part of this, the validation includes exceptions for two common shell practices, the trailing / and the trailing ~. For example:
cd ~
This is technically a valid first line of Ruby, with the next line promising to be the other side of the complement operator. However, since it is more common in a shell to be using ~ as `my home directory’ than it is in Ruby to use the complement operator at the end of a line, this is simply considered invalid Ruby and will be run as a command. The same applies for a trailing /, which is commonly used on the end of a directory name, as in shadowfiend/ruby/.
CDs
The CdCommand got a few updates during this release. It now correctly handles tidle expansion (by using the handy File.expand_path method… Kickass method, that!), and running a `cd’ with no parameter is now synonymous to `cd ~’. It’s kind of awesome.
Pipes
Finally, and, for me, most awesomely, this release brings the improved piping syntax to RubyShell. This syntax lets you use $pipe as an implicit receiver for the first message of a line of Ruby code in a piped expression (actually, in any expression). For example, say you wanted to run an `ls’ and then print each line with a trailing `RubyShell rocks!’. In RubyShell pre-0.3, you would do1:
ls | $pipe.each_line { |ln| puts "#{ln}RubyShell rocks!" }
Not horrendously ugly, but also not very pretty. RubyShell 0.3 still allows this syntax, of course; however, there’s a new one. Now, any calls to methods that start with with_ or and_ are assumed to mean `$pipe.’. The above example in done this way:
ls | with_each_line { |ln| puts "#{ln}RubyShell rocks!" }
There, doesn’t that look better? How about we want to have everything in the current directory, but we want to censor any mentions of `java’:
ls | and_gsub 'java', ''
Shiny, eh?
Other Fixes/Improvements
The presence of the new validator means a lot of validation-related issues were fixed for this release. Frederick Ros also provided a patch to fix RubyShell’s crashing shamelessly when you hit `enter’ with nothing on the line, ‘cause he’s just a fabulous person. Plus, now if a line of code wasn’t evaluated as Ruby and failed to run as a command, an error is printed. Some refinement need still be done—for example, printing an error is unnecessary if the command itself prints anything to stderr or stdout. Not sure how to determine whether that happened, though. It may be that the error will instead be printed if a valid command cannot be found. Again, however, this will be somewhat difficult to determine, since Kernel#system returns false whether that’s the case or the command just didn’t exit happily. Might be able to use the exit code for that.
Le Future!
Or, in Indo-Chinese2, `the future!’ While I wanted to do output redirection for this release, I decided to shelve it for now so I could think out and research the syntax a little better. The feature issue tracking it is over here, and I already posted some initial thoughts on how I might do the syntax. Also on the list is better error handling—i.e., it would be nice if RubyShell didn’t die a horrible death when you made a syntax error in the code you typed.
Further out, there’s the idea of improving shell completion, though that’s really dropping further on the list the more I think about it, since having filename completion (which is already present) is really by far the most important thing for a shell. Also on the `indeterminate future’ list is an initialization file (~/.ruby_shellrc or something) and the ability to run actual files of RubyShell code. These two should be relatively simple, and the former depends on the latter. I also need to implement backtick notation (``), but making that basically interpolation of any RubyShell command, including regular Ruby. That’s all I can think of for now, but more ideas are welcome! And I’m sure I’ll come up with a few more of my own, as usual.
Enjoy!
1 Actually, whether you would really do this is anybody’s guess, since RubyShell pre-0.3 handled blocks in piped expressions a little oddly, and would sometimes explode in a shower of fiery sparks.
2 Really, that should have been `Sino-Indian’, if only to use `sino’.
7 Responses to “RubyShell 0.3: Pipes, Parsing, and CDs”
Sorry, comments are closed for this article.
August 22nd, 2007 at 03:57 PM
Hey! Could you post login line for your svn, or distribute ruby_shell with .svn that doesn’t need password? It’s browsable through project page + you show changes history, but there’s no way to update :( (other than packaged versions of course)
August 22nd, 2007 at 04:53 PM
I know, that’s why I’m probably going to move it over to Rubyforge. Site5 unfortunately doesn’t provide anonymous svn access, so I’d have to basically make an SSH login that stands in for anonymous SVN access and distribute the private key, which is not so great a plan. In the meantime, though, you can use the mildly secret WebSVN to nab tarballs of SVN trunk.
August 23rd, 2007 at 12:27 PM Tried sending this to nomothetis, but I’ve noticed it’s lately only shadowfiend committing changes. Multiline string patch (mail me if it gets b0rked in the comment)
--- orig/lib/ruby_shell/parser.rb 2007-08-16 06:47:49.000000000 +0100 +++ ruby_shell_0.3/lib/ruby_shell/parser.rb 2007-08-23 00:57:36.000000000 +0100 @@ -153,7 +153,29 @@ # Determines whether the given code is part of a multiline command or not. def needs_continuation?(code) - @evaluator.valid_multiline_ruby?(code) + @evaluator.valid_multiline_ruby?(code) || has_unfinished_string?(code) + end + + # Determines whether the given code has unfinished string that needs + # continuation + def has_unfinished_string?(code) + unfq=false; unfdq=false + escaped=false + code.scan(/./m) do |c| + if !escaped + if c.eql?('"') && (!unfq) + unfdq=!unfdq + elsif c.eql?('\'') && (!unfdq) + unfq=!unfq + elsif c.eql?('\\') + escaped=true + end + else + escaped=false + end + end + + unfq || unfdq end end endAugust 23rd, 2007 at 06:30 PM
Ok, first, about the patch -- it looks pretty good. I would prefer to have the relevant code in the valid_ruby? part of the RubyValidator, since I've moved the multiline stuff in there. It might be easier to just look for the `unterminated string meets end of file' error, since that way the code is only handled once. This is largely the reason why I try to use error messages rather than Ruby code to determine multiline-ness -- because we get detailed enough error messages from ParseTree’s SyntaxErrors that they can be used for that purpose. I’ll think about whether I’m going to do that approach or yours. Thank you very much, regardless.
This ties more or less directly into your point concerning preferring commands over Ruby code, and falling back on Ruby. While it would definitely be substantially simpler, the priority of Ruby code was chosen deliberately. You’ll notice, amongst other things, that RubyShell commands (defined as classes) take priority over system commands, too. The idea is to follow the Ruby way of allowing you to override things if you want to. There are plenty of standard shells, and we kind of want to make this a non-standard shell. This is our opportunity to try to bring the Ruby mentality to the world of shells, and to try and play with some new, hopefully better, ideas in the world of shells while we’re at it.
Meantime, I’m considering revising the multiline code so that it will only continue a line when it gets an unexpected $ error and has a letter or number as the last character on the line. This should exclude most edge cases without substantially limiting the Ruby that can be used. The check for multiline strings would, naturally, also be included.
Btw, I just submitted the Rubyforge registration, so if and when that gets approved, I’ll get to work moving the SVN repository and other stuff over there.
I’m glad you’re interested enough to submit patches!
August 23rd, 2007 at 10:15 PM
(Praying that formating will be ok…)
I hope RubyForge will be up soon, so I won’t make this place a mess :) Function goes in Executor:
http://pastebin.com/m46bd6a1e
This goes into Executor.build_command_from, before str.split str = substitute_ruby_expressions str
It’s supposed to enable expressions and variables in shell commands. I don’t know how to write ruby specs (or ruby at all as you can see :) – you’ll probably refactor that function), so testcases are:
echo a#{1+2}b => “a3b”
echo a#{1+}b => “a#{1+}b”
echo a#{1+2}b} => “a#{1+2}b}”
echo a#{1+2b#{3+4}c => “a#{1+2b7c”
echo a#{1+b}c => stderr:Ruby Error in subexpression ‘1+c’...; stdout: “a#{1+b}c”
Two commented lines are for escaping \#{...}, but ”\” breaks multiline commands so far, so it’s not used.
WebSVN works nicely.
August 24th, 2007 at 12:17 PM
Now that’s what I’m talking about. I’ll look at this in more detail on Sunday, as I have to catch a flight this evening and will be back then. This looks really good though. Hopefully by Monday at the latest we’ll have a RubyForge project.
September 3rd, 2007 at 11:59 AM
FYI, I’ve committed this change to the RubyForge repository. I modified your code pretty heavily, and refactored a lot of the RubyShell code so everything drops in cleanly, but it should be working as needed.