Post cover image

March 10, 2016

Continuous Learning: Learning 5 Different Scripting Languages

“I’ve known ________ for so many years so why would I use a different language?”

By Peter Campbell

10 min read

"I've known ________ for so many years so why would I use a different language?"

This is is implicit mindset that I found myself thinking last year. In my case it was Perl. But it could be anything. It is so tempting to swallow this logic because we are cozy with it.

But this mindset is fallacious. Technology in general is fast moving, while software in particular is torrential. One cannot afford to move slowly when software is advancing so quickly. Being a software engineer means learning, re-evaluating new techniques and new technologies continuously to remain relevant and not become a "legacy" developer.

Last time I wrote I made a call for scripting in "Why Laziness, Impatience and Hubris Drives Great Developers to Script". In it I explored scripting with Perl using a simple script to grep performance data from AWS ELB log files. In it I revealed that I needed to learn some new languages — perhaps Perl is not the best general-purpose scripting language in 2016. Perhaps there are others that are faster to create, faster to run and more fun?

And so I started an experiment to re-write the ELB performance grep script in five different languages to understand them better and allow a comparison. I chose,

  1. Ruby because all the cool kids use it, and it great for both glue and web development. It is also closely related to Perl.
  2. Python because well, lots of people talk about it.
  3. Lua since it is lightweight and is said to have great runtime performance.
  4. Perl 6 because it is new, a different language to Perl 5 and has some innovative features that could make Perl popular again.
  5. Node.js because Javascript is the #1 scripting language today and it has been optimised for performance more than any other scripting language.

The code is available in GitHub for you to fork. Some sample data is included so you can run comparative tests. I must emphasise that I am learning all these languages so I am therefore not expert in any of them. I'm sure there is some idiom I've missed when translating the Perl script to your language. Feel free to suggest improvements particularly for run-time performance in the comments.

Ruby for the Cool Kids

Since Ruby has native object-oriented language features I though it would be fun to write an object-oriented version of the script to understand this. You can see this in grep_el.rb where I have two classes: CommandLineOpts to process the command line arguments and Elblogfile to process a single log file looking for slow response times. This makes the code more structured, though similar structure could be enforced with subroutines/functions in other procedural scripting languages.

I found Ruby fairly similar to Perl in terms of its basic types, operators and features. In fact it was an easy transition from Perl to Ruby for me. It provided good command line argument handling with the optparse module but also added OO and proper exception handling which was neat. The code is perhaps noob Ruby code but even so it is more more verbose at 113 lines long — 45% more than the Perl version.

But lets see how speedy[1] it is when we run it.

$ time ruby grep_elb.rb -t 3 testdata/*.log
…
3.50 real 2.87 user 0.18 sys

I wasn't expecting this. The ruby script runs more than 50% quicker than the Perl script. How can that be? I'm still trying to understand why this could be so much faster but this has encouraged me to do some more Ruby.

Python

You didn't expect me to learn Python did you? I couldn't bring myself to learn Python with all of those enforced indents. So I'm leaving the Python edition of the grep script to others.

Lightweight Lua

Lua has been around for a long time — it was born in 1993 about the same time as Perl X. Its provenance is completed different than all the other languages here. It was created at the PUC-Rio university in Brazil and its design principles are aimed at being small, portable, fast and embedded. So I was expecting it to be super fast.

It's grammar is smaller, the distribution is much smaller and because of this many features you come to expect in the Perl and Ruby distros are missing from Lua. But this is the benefit of the language: it cannot be both lightweight and feature-rich.

The grep_elb.lua script caused me the most pain translating from Perl. There is no command line parsing built-in and no CPAN (large-scale module library). Yes there is a lua package manager, luarocks but it doesn't support OSX. So I had to go hunting for a module and have explicitly included in with the script in the Gitlab repo. I choose lapp because I liked that it inferred the syntax from the usage statement which is a nice touch.

Lua is a functional programming language and I've used functions liberally. However, it is frustrating that there is no built-in split function for strings. Again I had to go hunting for one. Now the code isn't that large or complex but it makes lua development so much harder for command line applications when basic features need to be discovered or written.

Lua syntax is frustratingly restrictive — especially to a Perl programmer. Take the if statement for example. There is only one allowed form. unless is not included. There is no if as a postfix, e.g. some-action if some-condition.

if some-condition then some-action end

The lack of equivalent of nil and false is also strangely frustrating for a rapid development language where you need boilerplate to initialise booleans. This code for example is particularly ugly and unnecessary.

if argv.e1 == nil then argv.e1 = false end
if argv.e2 == nil then argv.e2 = false end
if argv.e3 == nil then argv.e3 = false end
if not argv.e1 and not argv.e2 and not argv.e3 then argv.e1, argv.e2, argv.e3 = true, true, true end

I also struggled to convert a table (associate array or indexed array) to list context like I could in Perl and Ruby. I'm sure this is just a language feature I wasn't aware of. But hey what about the run-time performance, was it not blistering?

$ time lua grep_elb.lua -t 3 testdata/*.log
…
7.59 real 6.63 user 0.23 sys

Shock! My Lua script is slower than my Perl script. This was disappointing. But what about luajit the JIT version? It is much quicker but still falls short of the Ruby script.

$ time luajit grep_elb.lua -t 3 testdata/*.log
…
4.10 real 3.57 user 0.16 sys

It's Christmas it must be Perl 6

Perhaps the biggest shock for the open source community last year was the release of Perl 6.0.0 in December. It has famously been promised by Larry Wall at Christmas…many many years before. Perl 6 is a new language from Perl 5, not an immediate successor or replacement. It is intended as a language for "the next 100 years" but as a language with a 15 year gestation period, it has lots to loose.

I listened to Larry Wall announce Perl 6 at FOSDEM 2015 and have listened to introductions of the language from the Perl community (take a read at Ovid's Perl for Mere Mortals talk for example). It is evident that the Perl language designers are perhaps the smartest around — and Perl 6 is the smartest language around too. It adds features and meta features that others languages haven't thought of yet while bringing some order to the chaos of Perl 5. Take these features for example,

  1. Rational number support. So if you run "0.1 + 0.2–0.3" you will get the right answer in Perl 6 unlike many other languages.
  2. Gradual typing with constraints are very powerful and readable. So you can now optionally define the type of your variables with validation constraints. You can see this in the USAGE subroutine of grep_el.p6.
  3. Subsets: define your own types with validation/constraints.
  4. Proper object-orientation with meta-object programming. All the usual object-oriented concepts are in Perl 6.
  5. Functional programming. So for example there are immutable variables by default. Read write is optional.
  6. Lazy evaluation of lists whether of sequence 1 .. 10000 or file content, etc.
  7. Parallel programming: promises, supplies, channels. Read Jonathan Worthington's presentation on it.
  8. More powerful regexs.
  9. Generics.

But how is grep_elb.p6 really different than the Perl 5 edition? Well I haven't used an object-orientated or functional programming style for Perl 6, I have remained with the procedural style so the short answer is fairly similar.

Perl 6 has built-in support for command line argument processing as arguments to the MAIN subroutine which is a probably the neatest way of any of the languages to handle it.

Lazy evaluation is everywhere including the for loop that reads each line from the log file. But unfortunately it is a bit slow. So I have instead used the alternative while loop form,

while (defined my $line = $fh.get)

not

for $fh.lines -> $line

Everything is now an object and everything is a reference. So instead of splitting the log file line by spaces we can now just call the words method on this string.

$line.words

Another lovely feature is the all() function (or junction as it is called). It evaluates a list again a scalar for equality and returns true if all are true. There is similar junctions for none(), each() and one(). So now I can write this to check that all of the $time1, $time2 and $time3 (which are now in an array) are equal to '-',

unless ‘-’ eq all (@time)

instead of the Perl 5,

unless ($time1 == “-” and $time2 == “-” and $time3 == “-”)

It pains me but we must also discuss performance. Runtime performance of grep_elb.p6 is off the chart terrible. So terrible it makes grep_el.p6 unusable. 6 minutes to do what Ruby did in 3.5 seconds.

$ time perl6 grep_elb.p6 — threshold=3 testdata/*.log
…
361.40 real 355.40 user 3.19 sys

The Perl 6 community has released rakudo-star (the current default Perl 6 implementation) and openly admits the release was aimed at functional completeness. Performance has not been a priority. It is now a priority in 2016 and beyond. And remember Perl 6 is not Rakudo Star. Rakudo Star is one implementation of the Perl 6 specification running on one VM today (MoarVM). In future we may have more choice.

I did some experiments and it is the file reading code that is the major culprit. Take the following stripped back perl scripts to read and echo file content. What it shows is that Perl 6 is about 300% slower than equivalent Perl 5 code. I have not used any of the asynchronous features for this — why should I have to when ruby and python achieve much better performance without this complexity.

This is basic Perl 5 to open a file and echo the contents. It assumes the filename is in the special variable $_.

die “specify filename” unless $ARGV[0];
while (<>)
{
    print “$_”;
}

For reference against the first log file which has 81424 lines it takes 2.19 secs to complete. The Perl 6 version if this that uses lazy loading within the for loop takes 9.40 seconds. And the Perl 6 while loop version comes in at a slightly faster 8.44 seconds.

use v6;
for $*ARGFILES.lines -> $line
{
   say $line;
}

Now I realise Perl 6 has a completely different architecture to Perl 5. It is running in a virtual machine (MoarVM for now, JVM soon) and compilation and optimisation into IL. But I do hope the Perl guys can bring it at least to parity with Perl 5 this year.

Node.js

Node is different. It takes the Javascript language and the Chrome browser's V8 engine and has created an event-driven, non-blocking by default generic application framework. You can create web applications and command-line applications without fear of threading but taking advantage of non-blocking IO.

It also has npm, the Node Package Manager which allows you to package your node applications for distribution or publishing. It also allows you to describe dependencies and grab published packages. And it is big, currently containing 250,000 packages.

Creating a node script means creating a package by default. This is a little bit more work than writing your script and immediately running it in REPL or the shell. But the advantage of the packaging approach is the ease of distribution once packaged. Node will also allow you to run it just like a built-in command, e.g. I can now deploy the node grep_elb package and run it like a command not a script:

$ sudo npm install -g
$ grepelb -t 3 testdata/*.log

This syntactic sugar is made possible by npm. The packaging metadata is fairly straightforward and includes the script, dependent packages, licensing, description, etc.

So what about the code? The Javascript language is a fairly straightforward procedural language with a Java and C influenced style. The major change that Node brings is the event-based, asynchronous IO by default model. This caused me a problem for a while since reading each file is asynchronous with callbacks for each line processed. I have used the line-by-line module to make the code simpler but the challenge I faced was figuring out how to associate all of the console output for a specific log file given it is expected to appear sequentially, one file after another.

The solution I ended up with was to chain the parseELBlogfile() function to initiate sequentially rather than in parallel. Clearly this avoids some of the performance won from parallel asynchronous IO processing — but if anyone reading can figure out how to synchronise the need for sequential console output with asynchronous file processing without doing this please let me know.

[Embedded content: 184c06fc6862ef4ea209f3f210ea6d92]

This can be seen with two different functions to accomplish the same processing. parseElblogfile() will parse each line asynchronously with each file initiated one after the other. You can enable this mode by setting the global variable sequentially = false. Alternatively the default mode is sequential initiation with asynchronous processing. Only when each log file finishes is the next log file initiated. This is the parseElblogfileSequential() function that takes in an array of log filenames and recursively processes them (with async file IO).

Node has a great selection of modules. For parsing command-line arguments I used the Commander module which allows for a straightforward command-line option specification and parsing of the results into a hash.

In terms of the script language itself Javascript remains fairly limited in its support for more expressive keywords that Perl or Ruby might provide. For example to assign out values from the log file line that are split into words I could find no way to list-assign. This means I had to resort to the immediately readable if rather ugly form of splitting to an array then assigning each array index to give it a readable name. So,

var items = line.split(‘ ‘);
var date = items[0], elb = items[1], clientip = items[2], referrer = items[3], request_processing_time = items[4], backend_processing_time = items[5], response_processing_time = items[6], elbstatus = items[7], backendstatus = items[8], rbytes = items[9], sbytes = items[10], verb = items[11], url = items[12];

What about performance? Performance is fantastic. Even for the sequential file processing of grep_elb.js the node version takes just over 3 seconds to process the test log files. This is faster than the ruby version which is itself 100% quicker than the Perl 5 version.

$ time grepelb -t 3 testdata/*.log
…
3.25 real 2.52 user 0.50 sys

For the incredibly fast version you will need to take a look at the Golang version that davey.mcglade has written in Is Compiling the New Scripting?

Javascripting FTW

The big winner here is Node.js, I've been really impressed. Its performance is great for a scripting language. It is easy to use, learn, package and publish. I will be writing more Node scripts in future.

Can you do better? I'd love you to re-factor and improve my scripts or write the grep script in other new languages to showcase the learning experience and run-time performance.

[1] MacBook, Early 2015 (MacBook 8,1). 1.3 Ghz, 8Gb RAM 1600 Mhz DDR3, 256Gb SSD (FileVault enabled), Intel HD Graphics 5300 1536Mb. El Capitan, 10.11.3 with Perl 5.22.1, Ruby 2.3.0, Lua 5.2.4, Luajit 2.0.4, Rakudo Star 2016.01.1 on MoarVM 2016.01 and Node 4.2.6.