What is Io? A quick review

Io is a programming language, obviously, created in the early 2000's by Steve Dekorte. It was featured in the original Seven Languages in Seven Weeks ^[1] book, which is how I learned of its existence. It's a general purpose, interpreted language, with semantics inspired by Smalltalk, Self and Lisp, and with look and feel reminiscent of TCL and Scheme.

While TCL tries to answer the question what can be done with just strings (and lists) and Scheme shows what happens when everything is a lambda (and an s‑exp), Io explores the consequences of assuming that everything is an object (and a message). They all share traits of conceptual purity and elegant minimalism of their designs and implementations. It would be a mistake, however, to think about them simply as works of art - this may come as a surprise to the uninitiated, but despite the simplicity (some would say, because of it) all these languages are mind-bendingly expressive and powerful. It's true that you need to change the way you approach problems to use their full power - but once you do, you will quickly realize that that power is over 9000!

Even though I speak about simplicity, Io is a fairly complete and usable high-level language. Its strongest superpower is probably its dynamism: nearly everything, everywhere, every time is inspectable and changeable from within the language. As for other features, from Smalltalk (and Ruby) it takes its purely object-oriented character, while Self and Lua inspired some of its object system, which is prototypal (you probably know this style of OO from JS) and with multiple inheritance. It has very simple syntax, which is customizable with incredibly powerful meta-programming tools.

Non-blocking, asynchronous Input/Output and concurrency support (with co-routines) are important parts of the language (see example on the left) - it even provides automatic deadlock detection. It has relatively simple C API for both embedding and extension and built-in CFFI for wrapping C libraries.

There are of course some disadvantages and problems: for example, Io is not very performant at this point^[2] and should not be used for number-crunching (unless it can be vectorized^[3]) or other performance-critical code. On the other hand, it works well even for writing bigger, complex programs which are IO-bound - all kinds of servers and (micro)services come to mind.

Another thing is that it isn't very popular currently (to put it mildly), which means the selection of ready to use libraries is narrower than with more mainstream languages. There are use-cases where it doesn't matter, though, like when embedding it as a scripting language for larger programs. It could also be used as BASH replacement for more complex scripting.

The problem, the hack and its effects

I picked Io because it seemed a good fit for one of my projects. When starting it, I considered various languages, but, taking all the requirements into consideration, only a couple were viable: LPC, Pike, Lua and Io. The process of eliminating all the other languages deserves its own post, but for now, let's focus on the winner, Io.

One way or another, my project evolved and at some point I decided that I need a ported implementation of PyParsing^[5], in Io. After defining some helper methods it was an easy job, with nearly one-to-one correspondence between lines of code in both languages. This is important, because in that case it's easy to convert between the languages with a set of simple regexes ran over the lines in a file. Io incredible expressivness allowed it to emulate most of Python features used in the library - with a glaring exception of tuple unpacking, also known as destructuring bind or (a weak version of) pattern matching.

As a quick reminder, tuple unpacking is a feature of an assignment operator in Python (and many others), which is used to extract elements from a sequence and giving each of them a name. It looks like this in Python:

1
2
3
4

  def some_fun():
      some_tuple = (1, 2, 3)
      (one, two, three) = some_tuple
      assert one == 1 and two == 2 and three == 3

It's very convenient, especially if you keep most of the data in tuples of a known length, which is how most of PyParsing is written. As mentioned, it's also missing from Io, which made the translation process more tedious than it could be.

It was a surprising omission on Io part, considering that the language already gives you ways of defining custom operators, including assignment operators. To simplify the porting of Python code (and for fun, obviously!), I decided to try defining a <- operator, that would implement the semantics of destructuring bind. In more specific terms, I wanted to extend Io so that the following code is valid and gives expected results:

1
2
3
4
5

  someMethod := method(object,
      object := Object clone
      object [one, two, three] <- [1, 2, 3]
      assert(object one = 1, object two = 2, object three = 3)
  )

It should be possible to do this, thanks to the mentioned features of Io: its extensible parser, which allows you to define new operators, and the lazy, on-demand, evaluation of method arguments. Actually, the code for doing so is hilariously simple, just a couple of lines of code^[6]:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

  Object destructure := method(
      # target [a, b] <- list(9,8) becomes: target a := 9; target b := 8
      msg := call argAt(0)
      lst := call evalArgAt(1)
      target := call target
      msg arguments foreach(i, x,
          target setSlot(x name, lst at(i))
      )
      target
  )
  # inform the parser about our new operator
  OperatorTable addAssignOperator("<-", "destructure")

It should have worked! - but it didn't ☹. Why? Also, what's perhaps more important to you right now (if you don't know Io), how was it supposed to work in the first place, anyway? And then, finally, is it even possible to make it work? (hint: yes!)

Relevant Io semantics explained

One trick to reading the above code is to realize that whitespace between words is not a separator, but attribute access operator. In other languages, attribute access is very often written with a dot (.) - so the literal translation to JavaScript (for example), would look like this:

1
2
3
4
5
6
7
8
9

  Object.destructure = function () {
      let msg = call.argAt(0)
      let lst = call.evalArgAt(1)
      let target = call.target
      msg.arguments.foreach( (i,x) =>
        target.setSlot(x.name, lst.at(i))
      )
      return target
  }

Now, to explain the rest of the example, we just need to know what call object is, what attributes it has and what it does. It's really simple (in a monad-like kind of way...): the call is just a runtime representation of a message send! (Just a Monoid in the category of endofunctors, right?)

Joking aside, what is a message send? Known as a method call in other languages, the message send is simply another name for a syntactic construct describing an invocation of a method with given arguments on some object.

I prepared a little diagram^[7] illustrating the concept:

Some additional description:

• message arguments - any expression is allowed, not just simple variables or literals. Expressions passed as arguments are only evaluated on demand. It's a very important feature, which means that you can pass unquoted (but still syntactically correct) code as an argument and it won't be evaluated unless the body of a method explicitly extracts and evaluates them. You can access a list of unevaluated argument expressions via call message arguments and you can access individual arguments with shortcut methods call argAt(n) and call evalArgAt(n).
• target - an object whose method is going to get called. Target may be a variable name in the simplest case, but in general it is an arbitrary expression, which is evaluated to obtain a reference to an object. If Io interpreter encounters a message send without a target, it is by default set to the sender (aka. context) of the call.
• context - a dynamic environment in which the message send is going to be evaluated. Is has no representation during compile-time, it only exists during runtime, and it's simply a mapping from variable names to object references, like what you get out of locals() function in Python. It is resolved on runtime. It's accessible via call sender attribute.

Io has no other syntax than a message send, which means that message sends and expressions are the same thing. Io doesn't have statements (in imperative sense) at all, which in turn means that the message send is the whole syntax of Io. As is normal for expressions, message sends return a value when evaluated.

A call, then, represents a (parsed) message send coupled with an environment (a context) in which the message send is evaluated this time. It is accessible (via interpreter magic) in block and method bodies, not unlike the arguments object in JavaScript. Such call is an object of type (ie. with a prototype set to) Call, which has message, sender (context) and target attributes. Further, message is an object of type Message, which has name and arguments attributes.

That's it - it's almost complete description of Io syntax. As you can see, Io is conceptually very simple and consistent. It manages to stay very expressive despite of this.

Defining operators

One of the missing pieces in the above description is the syntactic sugar which Io offers to enable operators. In Io, operators are simply messages which do not need to have their arguments parenthesized. The parser maintains a special object, called OperatorTable, which contains names of all the operators along with their corresponding precedence. It then automatically inserts parentheses around values to the right of an operator, taking precedence into account. For example:

1
2
3
4

  1 + 2 * 2
  # is converted (while parsing) to:
  1 +(2 *(2))
  # ==> 5

There are languages, such as Smalltalk, where this isn't the case: they use strict left-to-right evaluation order, only alterable by explicit parentheses. This parentheses inference is used for arithmetic operators, string concatenation operators and so on, but it's also used for faking statements, like return, break or yield.

Assign operators parse transform

Operators of a special kind, called assign operators, are parsed in yet another way. In this case, operator name (eg. :=) is first mapped to a method name (eg. setSlot) via OperatorTable object, and then the call is transformed like this:

1
2
3

  someObject someName := someExpression
  # is converted (while parsing) to:
  someObject setSlot("someName", someExpression)

It should now be easy to see why my attempt at writing destructure (as shown above) operator failed. The parse transform assumes the first argument to the assign operator to always be a simple name. It then puts quotes around that name and passes resulting string as the first argument to the method implementing the operator. If that assumption is broken (by eg. operating on a more complex expression), the conversion to a quoted string fails and the whole thing errors out.

This is enforced during parsing, before any Io code has a chance to run, so it's impossible to change it from within the language. If you take a second to think about it, that restriction (to simple names only) doesn't make any sense and is not present anywhere else in the language. To fix it, however, I had to delve deeper, into the C code of Io interpreter.

Deep dive into Io interpreter

Io is implemented in C, with implementation consisting of a custom, tri-color mark and sweep^[8] Garbage Collector, low-level co-routines^[9] implementation and an interpreter^[10].

It took me more time than I'm comfortable admitting to find the relevant code. While the documentation for Io exists, it's not very extensive - there's little information about the internals of the interpreter and its overall design. However, the bigger problem was wrestling with CMake and my own lack of knowledge about typical C tooling. It's slightly embarassing, since I worked with C and later C++ for years (although it was decades ago at this point). Once I brushed up on my skills in this area, locating the place to fix and developing patch wasn't that hard, fortunately.

The IoMessage_opShuffle.c module

As mentioned above, operators in Io are implemented as a parse transform. Most of it is implemented in IoMessage_opShuffle.c file. Main definitions there are Level and Levels structs:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

  enum LevelType {ATTACH, ARG, NEW, UNUSED};

  typedef struct {
      IoMessage *message;
      enum LevelType type;
      int precedence;
  } Level;

  typedef struct {
      Level pool[IO_OP_MAX_LEVEL];
      int currentLevel;

      List *stack;
      IoMap *operatorTable;
      IoMap *assignOperatorTable;
  } Levels;

Io C source^[11] is written in an object-oriented style, with C struct types being treated as classes and structs instances as objects. Following this style, function names are prefixed with class name on which instances they are supposed to operate; they also always take a pointer to a structure as their first argument (most often called self, like in Python).

Ignoring the boilerplate code for constructing the Levels objects out of raw Messages, the method which does the actual shuffling of operators is called Levels_attach, with the following signature:

1

  void Levels_attach(Levels *self, IoMessage *msg, List *expressions)

IoMessage objects contain a IoMessage* next field, which makes them a low-level implementation of a linked list (not to be confused with the List type). Despite the singular form of msg, it represents both a single message and a list of messages (just like char * represents both a string and a pointer to first character). The function takes the message, transforms it (warning: in-place!) and appends the following (next) message to expressions list. The IoMessage_opShuffle function (defined in lines 549-573), which calls Levels_attach, does so in a loop and repeats until there are no messages to process:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

  List *expressions = List_new();

  List_push_(expressions, self);

  while (List_size(expressions) >= 1) {
      IoMessage *n = List_pop(expressions);

      do {
          Levels_attach(levels, n, expressions);
          List_appendSeq_(expressions, DATA(n)->args);
      } while ((n = DATA(n)->next));

      Levels_nextMessage(levels);
  }

The Levels_attach function has to handle at least two cases: that of normal and assign operators. We're currently not interested in normal operators - what we need is to locate the code which handles messages of the form:

1

  optionalTarget msg(name1, name2) assignOp(arg1, arg2) nextMsg

Fortunately, it was easy to find it - there's even a comment showing our exact case next to the code, in lines 396-400. The problem is that this case is apparently considered an error and is handled as follows:

1
2
3
4
5
6
7
8

  if (IoMessage_argCount(attaching) > 0) { // a(1,2,3) := b ;
    IoState_error_(
      state, msg,
      "compile error: The symbol to the left of %s cannot have arguments.",
      messageName
    );
    return;
  }

Right, but Steve, why?! I'd like to know why that restriction was put in place; my intuition tells me that this code was written relatively early in language development, wasn't thought through completely and later nobody wanted to touch it^[12]. Well, it at least explains why my initial attempt failed.

The patch - proper handling of our case

Well, at this point I at least knew exactly, where to put my code for handling this! After checking out the source and setting up a build environment, I started implementing the code. It wasn't as easy as I'd like: first, internal APIs are mostly undocumented and second, most functions which implement "methods", are defined using IO_METHOD macro, which my "Go to definition" plugin didn't like☹. Other than this, the mutable nature of IoMessage objects and the need to deep copy (not just clone) them^[13] were a bit of a PITA.

Still, after a bit of tinkering, putting a lot of printfs here and there and a fair share of segfaults, I managed to produce a working implementation. Actually, I'm still surprised that it works... but it does! Let me show you (assuming the destructure operator to be defined):

1
2
3
4
5
6
7
8
9

  o := Object clone
  o [wa, wb, x] <- list(3, 123)
  o println

  # prints:
  #  Object_0x19bcb60:
  #  wa               = 3
  #  wb               = 123
  #  x                = nil

As you can see, it works and returns desired results! The patch to Levels_attach is not too long (about 20 LOC) and not very complicated, which was a pleasant surprise. Let's explain what happens in it, line by line.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

  Level *currentLevel = Levels_currentLevel(self);
  IoMessage *attaching = currentLevel->message;
  IoSymbol *setSlotName;

  /* ... */

  if (IoMessage_argCount(attaching) > 0) { // a(1,2,3) := b ;
    // Expression: target msgName(v1, v1, v3) assignOp   v4   ; rest
    //                    ^^^^^^^^^^^^^^^^^^^ ^^^^^^^^  ^^^^  ^^^^^^^
    //                      slotNameMessage     msg      val    rest
    // becomes:    target assignOpName(msgName(v1, v2, v3), v4) ; rest

    IoSymbol *tmp = IoSeq_newSymbolWithCString_(state, "");
    setSlotName = Levels_nameForAssignOperator(
      self, state, messageSymbol, tmp, msg
    );

    IoMessage *slotNameMessageCopy = IoMessage_deepCopyOf_(attaching);

    IoMessage *slotNameMessage = attaching;
    DATA(slotNameMessage)->name = setSlotName;
    DATA(slotNameMessage)->args = List_new();

    IoMessage_rawSetNext_(slotNameMessageCopy, NULL);
    IoMessage_addArg_(slotNameMessage, slotNameMessageCopy);

    IoMessage *value = IoMessage_deepCopyOf_(DATA(msg)->next);
    IoMessage_rawSetNext_(value, NULL);
    IoMessage_addArg_(slotNameMessage, value);

    IoMessage *rest = IoMessage_deepCopyOf_(DATA(DATA(msg)->next)->next);
    IoMessage_rawSetNext_(slotNameMessage, rest);

    return;
  }

In lines 13-16, setSlotName is a Symbol struct pointer, containing a string extracted from OperatorTable, which we know as the second argument in the call to OperatorTable addAssignOperator. In other words, it's a name of the method which implements given operator. Once we have the name, in lines 18-22, we create a copy of current slotNameMessage message, and start modifying the original. We set its name to the one obtained above, and we set its argument list to empty list.

In lines 24-25, we mutate the slotNameMessageCopy by cutting off its tail (as mentioned, every message carries a pointer to the following messages) and put it as the first argument to the original slotNameMessage. This is the most important change to the logic of Levels_attach: without it, the destructure operator wouldn't work.

Further, in lines 27-29, we get the first message to the right of the original operator (the one before conversion to method name, <- in our case) - in other words, the value that we want to destructure - and put it as a second argument to the operator method. Again, we need to cut off its tail, otherwise we'd put the following messages inside the argument list as well.

Finally, in lines 31-32, we attach the first of the rest of the messages as a tail of slotNameMessage. This completes the transformation.

That's it!

To be honest, the whole thing took me nearly a year to finish (including writing this post). I could work on it only once in a while, and I hit a few roadblocks which took many sessions to work around. I mentioned it briefly before, but getting Io code to compile was one such roadblock - I've never used CMake before, for one, and then some add-ons refused to compile on my system. It took me a while to sort all that out, and I couldn't start hacking without this.

With that done, I realized that I have no idea about what is where in the codebase. I had a rough idea of the architecture, as it is mentioned in the docs, but it was on a level high enough as to be mostly useless. I spent many an evening just reading the C sources, trying to familiarize myself with its style and design.

Once I felt vaguely comfortable with my knowledge, I started poking here and there with printfs. It's not that easy to print Io objects from C side - there's a bit of a ceremony involved. For example, to display the slotNameMessage I'd do:

1
2

  printf("slotNameMessage: %s\n",
         CSTRING(IoObject_asString_(slotNameMessage, msg)));

Memory management wasn't that big of an issue - Io objects are garbage collected, and I didn't need to allocate memory on the heap from C at all. Understanding how the GC works, and how different structs are to be interpreted as classes in a class hierarchy was a real challange, though.

Once I understood most of the IoMessage_opShuffle.c and implemented my fix, I realized that many parts of the interpreter could really use a serious refactoring. My first reflex was to put const qualifier almost everywhere - which backfired, because mutable state is everywhere and it's hard to say which argument to the function will be modified and which will be left alone. Some functions use multiple return statements, each in a surprising place; all these returns return nothing, basically acting as goto to the end of a function.

The internal APIs for manipulating objects are underdocumented (yes, I know I said it already) and incomplete. Working at the C level is made awkward because of that, as it's never clear if you should call a method or a helper function. Methods are also defined incosistently, either with IO_METHOD macro or without. It took me awhile to understand the DATA macro and why is it redefined for every Io object. There's a lot of commented out code and some areas of the code are simply a mess.

Despite all this, it was an interesting, if a bit long, journey. I learned a lot, which was my main goal anyway, and also managed to make the desired feature work, which was a nice by-product.

The End

If you reached this point - thanks for reading. I hope it wasn't too boring a write-up. I'd be incredibly happy if this post inspired you to take a closer look at Io, to try to use it, or perhaps even to try developing it.

Despite some messy parts, Io codebase is relatively short and simple, and Io as a language has a couple of features that make it a viable alternative to other languages in certain circumstances. With Io, you get Lisp-level meta-programming support without being tied to s-exp syntax. The ability to add your own syntax to the language, coupled with its incredible reflection support makes molding the language to closer fit your problem domain a breeze. Unlike Scheme, Io is based on a familiar, object-oriented metaphor, which makes it easier to read and learn by most programmers.

The slowness of Io - and I'm not even sure how big it is, I didn't measure - only means that there probably are many low-hanging optimizations to be done in its source. After reading the code I get the feeling that the authors wanted the language more than they wanted the implementation, which means they chose to add features instead of polishing existing ones. It's actually a good strategy early in language development and it's usually left to the 1.0 release preparation work to clean the code and make it efficient. It's just that Io died before the effort to make it 1.0-grade software even started.

I think it's still not too late, that Io still has potential and that it could, with time, be made into a serious competitor for Lua in some cases.

• More precisely, "Seven Languages in Seven Weeks: A Pragmatic Guide to Learning Programming Languages", a book which could serve as a Polyglot Manifesto if only polyglot was more of a thing... ↵
• It could get much better with JIT compilation, but it's not implemented currently. ↵
• Io has built-in support for these. ↵
• A descendant language of -->

String interpolation is a convenient language feature which makes creating human-readable strings that contain variables easier. The classical approach to this in languages without string interpolation is something like printf - a function which accepts a format string and a variable number of arguments. The format string contains special sequences of characters which get replaced with the values of passed arguments, like this:

1
2

printf("An error occured at %s:%d.", "file-name.c", 42);
// Displays: An error occured at file-name.c:42.

This has various problems, for example the possibility to supply more (or fewer) arguments than expected. String interpolation allows you to embed variables directly into the format string:

1
2
3

LINE=42
FILE="file-name.c"
echo "An error occured at $FILE:$LINE."

As with many features, some languages provide string interpolation and some don't. In the dark and troubled past of JavaScript, for example, some people switched to CoffeeScript for its string interpolation support (among other things.) More recently, string interpolation came to Python in the form of f-strings (PEP-498) and to plain JavaScript in the form of template literals (supported by Babel and other transpilers). While the feature is most often associated with dynamic, dynamically typed languages, some statically typed ones, for example C# and Swift, also support it, so it's not exclusive to the former.

Help! No string interpolation in Racket!

Racket - like JS at one point - lacks the string interpolation support. If Racket was JS, the only options would be to either write a transpiler, or switch to another language. Both choices normally require a lot of work, so developers tend to learn to live without the feature instead and move on.

But Racket is not JS, it's a Lisp. All Lisps are programmable programming languages^[1], but Racket takes it up to eleven with the multidute of tools for language extension (and creation)^[2]. How much work would it be to make Racket interpolate strings?

How Racket processes code

To process any code, Racket performs these steps:

• reading - accomplished by two functions, read and read-syntax, reading means taking a string and returning an AST^[3] (which happens to be a normal list with normal symbols, strings, numbers, etc. Such a list can be wrapped with a syntax object, which adds some meta-data to the raw list. You can always get a raw list out of syntax object with syntax->datum).
• expansion - all the special forms and macros in the read code are expanded until only the most basic forms (modules, function applications, value literals) are left.
• compilation - Racket performs byte-code compilation on the expanded code. This is where most of the optimizations are applied.
• execution - compiled byte-code is handed to the VM for execution. At this stage, on some architectures, further JIT compilation happens, turning (possibly portions of) bytecode into native code.

The first two steps are completely customizable: code expansion with macros, and code reading via hooks into the Racket parser (you can also write a new parser from scratch (or generate it) if you want, but frequently it's not necessary). Symbolically, what happens during reading and expansion can be presented^[4] like this:

Extending the parser

The Racket parser is a simple, recursive-descent one and it works by utilizing a readtable - a mapping from characters to functions responsible for parsing what follows the given character. The read and read-syntax functions utilize current-readtable, which is a parameter you can override. Adding a new syntax to the reader requires just three steps:

1. create a function for parsing the new syntax into valid Racket forms
2. get the current-readtable and add your function to it
3. make that expanded readtable from the prev step the new current-readtable

Interpolated string syntax

We'll get to writing the parsing function in a minute, but first we need to know what is it we're going to parse and what the results of the parsing should be. The exact syntax I have in mind would look like this (with #@ distinguishing interpolated strings from normal ones):

1

#@"An error occured at @{file}:@{line}."

Interpolated string compilation

String interpolation, for example in CoffeeScript, is often compiled to an expression which concatenates the chunks of the string, like this:

1
2
3

"An error occured at #{file}:#{line}."
# would be compiled to:
"An error occured at " + file + ":" + line + "."

In Racket we have no + operator for concatenating the strings and even if we did have it, the prefix notation and resulting nesting wouldn't be pretty. The easier and more natural way is to use a string-append procedure, which - conveniently - accepts any number of arguments. The syntax introduced in the previous section would compile down to something like this:

1

(string-append "An error occured at " file ":" line ".")

The parsing function

The parsing function needs to take the string form as an argument and output the code as above. Here's the simplest implementation I could think of:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

(define (parse str)
  ;; Assume that @{} cannot be nested and that the braces are always matched.
  ;; Obviously, in a real parsing function, these assumptions would need to be validated.
  (define lst (regexp-split #rx"@{" str))
  ;; After splitting we have a list like this: '("An error occured at " "file}: " "line}.")
  ;; We'll go over it, building a list of expressions to be passed to string-append as we go.
  (define chunks
    (for/fold ([result '()])
              ([chunk (rest lst)])        ; we don't need the first element here
      (let*
          ;; convert a string into a port
          ([is (open-input-string chunk)]
           ;; call original read to get the expression from inside brackets
           [form (read is)]
           ;; read what remains in the port back into a string
           [after-form (port->string is)]
           ;; drop the closing brace
           [after-brace (substring after-form (add1 (s-index-of after-form "}")))])
        ;; ~a is a generic "toString" function in Racket
        (append result `((~a ,form) ,after-brace)))))
  ;; chunks now looks like this: ((~a file) ":" (~a line) ".")
  `(string-append ,(first lst) ,@chunks))

It's dead simple - just a couple of lines - and it glosses over important details, like the possibility of nesting the @{} expressions and syntax errors handling, but it works for the simple cases. We can easily test it, like this:

1
2
3
4

(parse "An error occured at @{file}:@{line}.")
;; Returns:
'(string-append "An error occured at " (~a file) ":" (~a line) ".")
;; which is exactly what we wanted!

The boilerplate code

The hard part ends here - the rest is just some boilerplate we need to make Racket aware of our extension to its reader. In Racket, this is often done by defining a new language - working with #lang lang-name syntax, but in our case we don't need (or want) to create a whole language (it's simpler than it sounds, but it's still a bit more work) - we just change the reader and can reuse everything else from some other language.

For situations like these, Racket supports a special syntax to its #lang line, which looks like this:

1
2
3

#lang reader <reader-implementation-module> <language-the-reader-will-be-used-with>
;; for example:
#lang reader "reader.rkt" racket

Generating a reader module

Let's first define reader.rkt. Writing a whole new reader is of course an option but it's a lot of unnecessary work, as the generic read and read-syntax^[5] need to handle some more complex things, like module declarations, which we don't care about. Instead, we can use a special language for defining readers, which will generate appropriate read and read-syntax implementations for us provided we supply a single function to it:

1
2
3
4
5

#lang s-exp syntax/module-reader
#:language read
#:wrapper1 wrap-read

(require "ext.rkt")

The #:language option takes a function which accepts a string (of what remains after the reader module name on the #lang line) and returns a language name to use. A simple read happens to work here.

The #:wrapper1 option takes a function, which takes another function as input and returns the parsed code. The passed in function, when invoked, performs normal read.

Extending and installing a readtable

With #:wrapper1 option, in conjunction with parameterize, it's very easy to install our readtable^[6]:

1
2
3
4
5
6
7
8

(define (wrap-read do-read)
  (define rt (make-readtable (current-readtable)
                             ;; register our function to be called when #@ is encountered
                             #\@ 'dispatch-macro
                             ;; name of the function to be called
                             read-str))
  (parameterize ([current-readtable rt])
    (do-read)))

Wrapping parse so that it works inside readtable

The only part missing is a read-str function, which is a bit more complicated. The function passed to make-readtable may be called in two different ways, depending on whether it's used from inside read or read-syntax. The function distinguishes the cases based on a number of arguments it gets, and returns either syntax object or a simple list depending on the case. Here's its implementation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

(define read-str
  (case-lambda
    [(ch in src line col pos)
     ;; The caller wants a syntax object. The current position of the `in` port
     ;; is just after the #@ characters, before the opening quote.
     (let*
         ;; Here we read the next expression (which we assume is a string)
         ([literal-string (read in)])
       ;; then we parse the contents of the string and return the result,
       ;; wrapped in a new syntax object. The `in` port is now just after the
       ;; closing quote and the parser continues from there.
       (datum->syntax #f (parse literal-string)))]

    [(ch in)
     ;; The caller wants a simple list. Let's parse the input into syntax and
     ;; transform to a list (this saves as writing the same code twice).
     (syntax->datum (read-syntax #f in))]))))

It works!

The reader extension can be used like this:

1
2
3
4
5
6

#lang reader "reader.rkt" racket

(define line 42)
(define file "file-name.rkt")
(displayln #@"An error occured at @{file}:@{line}.")
;; Displays: An error occured at file-name.rkt:42.

It also works for more complex expressions out of the box:

1
2
3
4
5

#lang reader "reader.rkt" racket
(require racket/date)

(displayln #@"Current date&time: @{(date->string (current-date) #t)}")
;; Displays: Current date&time: Friday, April 28th, 2017 7:57:11pm

This is of course a pretty basic as far as string interpolation features go, but it took thirty (34 to be precise) lines of code (excluding comments) to build it. Adding another twenty lines would be enough to implement error checking, and using one of the parser generators could even shorten the code.

That's it!

Aside from how easy it is to extend Racket syntax one thing deserves a mention. That is, such syntax extensions are easily composable (as long as they don't want to install extensions to the same entry in the readtable) - you can load them one by one, each extending the readtable.

The language extension and creation tools of Racket don't end there. Robust module system with support for exporting syntax extensions, powerful syntax-parse, traditional syntax-rules and syntax-case, many tools for generating parsers and lexers - all these features are geared towards meta-lingustic programming. Racket is a programming language laboratory and allows you to bend it to better suit your needs safely (ie. ensuring that only your code will be affected). Very few languages or transpilers offer that much freedom with that kind of safety guarantees.

It's an ideal environment for people who know what they need from their language - with Racket you're never stuck, waiting for the next release to support some feature. On the other hand, despite there being a great many helpers and libraries, you need to know at least some basics of programming language creation to do this yourself. The good news is that the extensions written by others are easily installable with a single command of a Racket package manager.

Addendum: doing the same without messing with the reader

A commenter on Hacker News provided an interesting solution using normal macros only. Turns out there's a special macro, called #%datum, which gets wrapped around every literal in the code (when the code is read). By default it's a noop, but we can override it with arbitrary logic. In this solution there's no need for marking strings for interpolation: all strings are interpolate-able by default. Here's the implementation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

(require
  (rename-in racket (#%datum core-datum))
  (for-syntax syntax/parse))

(define-syntax (#%datum stx)
  (syntax-parse stx
    [(_ . x:str) (interpolate #'x)]
    [(_ . x)     #'(core-datum . x)]))

(define-for-syntax (interpolate stx)
  (define re            #rx"@{[^}]+}")
  (define (trim match)  (substring match 2 (- (string-length match) 1)))
  (define (to-stx val)  (datum->syntax stx val))
  (define (datum val)   (to-stx (cons #'core-datum val)))
  (define (source text) (to-stx (read (open-input-string text))))
  (let* ([text     (syntax->datum stx)]
         [matches  (regexp-match* re text)]
         [template (datum (regexp-replace re text "~a"))]
         [values   (map (compose source trim) matches)])
    (if (null? matches)
        template
        (to-stx `(,#'format ,template ,@values)))))

I don't have the time to go over this line by line and explain it right now, but it's possible I'll write another post featuring this solution in the future.

• The quote comes from Paul Graham's famous book and is attributed to John Foderaro. ↵
• A good starting point for learning more would be Racket is ... article by Matthias Felleisen. At least I assume it's his - it's not signed, but clearly in Matthias home directory ;) ↵
• Abstract Syntax Tree, see Wikipedia ↵
• Taken from a paper Type Systems as Macros by Stephen Chang, Alex Knauth, Ben Greenman ↵
• Any module, which exports these two functions, can be used as a reader. ↵
• In the actual source code the (make-readtable ...) part is extracted into separate procedure, which makes it easier to compose this syntax extension with others. ↵

• See RⁿRS for an official definition of Scheme. The newest Report is in this document ↵
• See this document for an in-depth explanation of what that means. ↵
• See wiki ↵
• Racket now supports three main types of macros, the latest documented here ↵
• Now called TypedRacket ↵
• See design by contract ↵
• Called PLaneT back then, Racket since changed its package management tool and now the repo is here. ↵
• Racket includes DSLs for making slides and writing Literate Programs, among other things.↵
• It only stopped a couple of years later with Quicklisp ↵
• See wiki ↵
• In Lisps known as DESTRUCTURING-BIND ↵
• In 2011, or at least that's when the repo starts ↵
• Transducers are one of the more famous (and recent) examples. ↵
• Called Leiningen. ↵

Apparently, I can't directly upgrade to whichever version I choose, I need to go one by one, doing fedora-upgrade and rebooting in a loop as needed.

I realize that my case is somewhat extreme, but watching the same set of packages getting downloaded five times (~640MB each time) feels wrong somehow.

The good news are that the upgrade took its time, but in the end finished successfully. Quite a feat for that large defference in versions: default package manager and system upgrade tools changed in the meantime, but still managed to produce a working system with minimal fuss.

Damn, I'm getting back to Linux as fast as I find a decent laptop; working on Mac OS - although, with effort, made vaguely ok - is tiring as I need to cope with all the "oh, it doesn't work that way here" moments...

RSS feed

As you can probably guess with a quick glance, this blog is self-hosted by me and is a result of "static page generation", which means there's a bunch of files and a few scripts which output all the needed HTML/CSS/JS to run the site, you just upload it to the server and you're done. This approach is very popular lately and many bloggers do the same.

One - slightly - unusual bit is that I wrote all of the generators and scripts myself, from scratch (using mainly Python). This means, among other things, that I have no pile of already written plugins or modules I could resort to in case I found my generator lacks an important feature...

For a long time I was pretty much the only user of both the generator and the blog, so I didn't mind. Recently, though, quite a few people who said they'd like to read my next post (my prospective readers! Finally, after four years, I have a chance to get my very own readers! ...err, sorry, got a bit too happy there for a moment...) suggested adding RSS feed to the blog.

I had known nothing about RSS and it took a couple of hours, but I finally added RSS feed generation to my scripts. Now every time I publish a post, a new /statics/feed.xml gets generated and, hopefuly, every subscriber will be notified!^[1] Even better, you will be even notified about post updates (probably)!

Thank you! (Or: Rkt vs. Clj is comming!)

Ok, so I know simply adding RSS feed is hardly important enough to write a post about it, so here is the other part: thank you very much! All of you who upvoted my comment on Hacker News or even sent me an email - only to encourage me to write the promised post, thank you. It's thanks to you that I got motivated enough to finally start writing that post about Clojure and Racket comparison.

Please stay tuned and be patient: it's a lot of work, especially because I want to stay as fair as possible and give as many detailed examples as possible (without copyright-related problems).

So, once again, thank you for expressing your interest in my future post and please stay patient while waiting for me to finish; I'll try my best, on my part, to write it all well and on time!

• Let me know if it doesn't work for you or your reader! I don't know much about RSS and could have messed up). ↵

I worked on a Web project in Clojure and saw how frontend/JS devs had a hard time adjusting to the syntax. Here is a breakdown of which tools I find essential for editing lisps as well as a list of possible tool suites (IDEs with plugins, standalone editors, etc.).

It's normal, I think, to get overwhelmed by the sheer amount of parens you need to manage when learning a Lisp. When taken on its own, the syntax really is simple, but introducing another strange thing on top of already unfamiliar concepts doesn't help learning.

Some people will tell you that you will "get used to it" soon enough and that it's just a matter of practice. While there is some truth to these claims, I'm a pragmatist and so I prefer another approach: simply set up a decent environment for working with Lisp before you begin. There are many tools which make reading and writing Lisp code much easier: you need to either configure your editor to enable them or change the editor. There are some Clojure-specific editors out there, you can simply use one of them (my thoughts on them below).

Main things/features useful when working with Lisps:

• syntax highlighting for your Lisp (obviously!)
• automatic indentation (and re-indent) for your Lisp
• ability to highlight matching paren
• ability to jump to matching paren
• configurable syntax coloring and/or rainbow-style coloring of parens
• ability to wrap a text selection with parens
• automatically inserting closing paren
• ability to delete parens in pairs automatically
• par-edit or par-infer style structural editing
• auto-completion for module/function names^[1]
• quick access to the docs for a name under cursor
• "Go to definition..." is good to have, but you can usually make do with grep or "Find in files..." editor command

See the screenshots and a video for visual demonstration of what I'm talking about (click on the image to get a bigger version):

Also see here for a very good general introduction to editing Lisp along with explanation of what Parinfer is.

Clojure specific editors

After writing the above I realized that it would be good to give a couple of examples of beginner-friendly editors, which implement the features mentioned above. To my surprise there are some editors which target Clojure, some of them even maintained. Here's a short summary of what I found out:

Nightcode - I only played with it a little but I'm impressed at what it can do. I tried using parinfer some time back and it is much friendlier and easier to use than Paredit^[2]. In short: you never need to worry about parens when using Nightcode. Place a cursor where you need it and start typing: the parens will magically appear. It worked out of the box for me on Mac, however it refused to run on Fedora.

Clooj - inactive since 2012, has nearly none of the useful features, looks ugly and is slow. Forget it.

LightTable - supports many things mentioned above as plugins, but they tend to be disabled by default and enabling them is not as easy as it should be. The plugin-based architecture makes it interesting for polyglot projects, but it needs some configuration to get started and Nightcode needs none.

Cursive - a Clojure plugin for IntelliJ IDEA from JetBrains. I don't have any of their IDEs installed and so I didn't try it. Its feature list looks decent, though, so if you like IntelliJ this may be the best option for you.

Emacs and CIDER - that's what I use. I'd recommend you to only try this route if you already know some Emacs, otherwise it's going to be frustrating for a good while until you internalize all the strange names and such. CIDER itself is great, though, and integgrates with Leiningen, offers inline code evaluation, auto-completion, connecting to a running REPL and so on.

In short: give Nightcode a try if you can, otherwise use a Lisp/Clojure plugins for your current editor, like Sublime, IntelliJ or Eclipse. Come over to Emacs side once you get bored with those.

• A full "IntelliSense" - context aware auto-completion - is always nice to have, but you can live without it. For a while. ↵
• I stick with Paredit because I already got used to it and am efficient enough with it; I'd go for parienfer had I started learning about Lisps now. ↵

I gave a talk on 15.03 about polyglot programming, based on the example project I wrote some time before. The project was a web scraper, with downloading part written in Elixir and processing part written in Python.

SLIDES

The talk was recorded, so I'll link to the video as soon as it becomes available.

I gave a talk at work yesterday about OpenRESTY, which I think is the easiest way to start scripting Nginx. Here are the slides:

SLIDES

One thing I'd like to add is that I don't trust OpenRESTY as an app platform yet. The project is being actively developed and the author writes tons of good code, but he is but a single man. This means there is severe lack of tools and libraries in the ecosystem: they simply don't exist yet.

On the other hand, OpenRESTY is a painless way to script Nginx with Lua and that's something you can do with just the modules provided by OpenRESTY. Being able to communicate asynchronously with anything (literally: anything that wants to talk over sockets), including databases, external services and so on, is very nice thing indeed.

There's one caveat though, which is that you can't use any blocking (non-async) constructs, which includes using normal Lua sockets. This is not the problem in your code, but if you happen to need a Lua library which comes as a C extension, and if the functions in the library block, you will probably need to fix the C code. At this point it's purely theoretical, because I didn't encounter any such library yet.

TL;DR or What is this post about?

In short: it's about me exploring - or at least getting into contact with - a couple of interesting things:

• X Window System APIs via Xlib bindings
• low-level Linux APIs for elevating privilages, checking passwords
• GCC options and some C
• and of course the Nim language

Introduction

At my work we strongly discourage leaving logged-in accounts and/or unlocked screens of computers. I happen to agree that locking your computer is a good habit to have, so I've had no problems with this rule... Up to the point when I switched from KDE to StumpWM (I wrote about it some time ago, in this, this and this posts) and my trusty Ctrl + Alt + L stopped working.

The only idea that came to my mind was to use the venerable xscreensaver, but: a) I didn't really need any of the 200+ animations (I just wanted a blank screen) and b) I didn't like how the unlock dialog looked like^[1].

I needed something more lightweight (xscreensaver is ~30k loc of C^[2]), simpler and either better looking or without any UI altogether.

slock to the rescue

There's a site called suckless.org where you can find some impressively clean and simple tools, implemented mostly in C. You can read more about their philosophy here, which I recommend as an eye opening experience. Anyway, among the tools being developed there, there is also slock - a very simple and basic screen locker for X. It's 310 lines of code long and it's all pretty straightforward C.

The program suited my needs very well: minimal, fast, and good looking. Well, the last part it got through cheating, as slock simply has no interface at all - but this means there's no ugly unlock dialog, so it's all good.

Why Nim?

As I used slock I read through its code a couple of times. It seemed simple and learnable, despite the fact that I knew nothing about X and didn't use C seriously in 15 years. Fast forward to last week and I finally found some free time and decided to learn some Xlib stuff. Re-implementing slock in Nim looked like a good way of learning it: this may sound a bit extreme, but it allowed me to gain exp points in two stats simultaneously^[3] - in Xlib usage and Nim knowledge!

Nim is a very interesting language. I'd call it "unorthodox" - not yet radical, like Lisp or Smalltalk, but also not exactly aligned with Java and the like. For one tiny example: return statement in functions is optional and if omitted, a function returns its last expression's value. That's rather normal way to go about it; hovewer, in Nim you can also use a special variable name result and assign to it to have the assigned thing returned. It looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

  proc fun1() : int =
    return 1

  proc fun2() : int =
    1

  proc fun3() : int =
    result = 1

  assert fun1() == fun2() == fun3()

At a first glance this may look strange, but consider this in Python:

1
2
3
4
5

  def fun():
      ret = []
      for x in something:
          ret.append(...something...)
      return ret

It's practically an idiom, a very common construct. Nim takes this idiom, adds some sugar, adapts it to the world of static typing and includes in the language itself. We can translate the above Python to Nim with very minor changes^[4]:

1
2
3
4

  proc fun4() : seq[int] =
    result = @[]
    for x in something:
      result.add(...something...)

As I said, it's not a groundbreaking feature, but it is nice and good to have, and it shows that Nim doesn't hesitate much when choosing language features to include. In effect Nim includes many such conveniences, which may be seen both as a strength and as a weakness. While it makes common tasks very easy, it also makes Nim a larger language than some others. It's not bad in itself, rather, depending on how well the features fit together it's harder or easier to remember and use them all. Nim manages well in this area, and also it most definitely is not like C++ with its backwards compatibility problem, so I think even Pythonistas with taste for minimalism will be able to work with Nim.

Being Nimble - the project setup

Many modern languages include some kind of task runner and package manager either as part of stadard distribution or as downloadable packages. Nim has Nimble, which takes care of installing, creating and publishing Nim packages. Assumming that you have Nim already installed^[5], you can install Nimble with:

1
2
3
4
5

  $ git clone https://github.com/nim-lang/nimble.git
  $ cd nimble
  $ git clone -b v0.13.0 --depth 1 https://github.com/nim-lang/nim vendor/nim
  $ nim c -r src/nimble install
  $ export PATH="$PATH:~/.nimble/bin/"

Note the addition to the PATH variable: ~/.nimble/bin/ is where Nimble installed itself and where it will place other binaries it installs. Make sure to have this directory in your PATH before working with nimble.

Creating a project

Creating a project is easy, similar to npm init:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

  $ mkdir slock
  $ cd slock
  $ nimble init
  In order to initialise a new Nimble package, I will need to ask you
  some questions. Default values are shown in square brackets, press
  enter to use them.
  Enter package name [slock]:
  Enter intial version of package [0.1.0]:
  Enter your name [Piotr Klibert]:
  Enter package description: Simplest possible screen locker for X
  Enter package license [MIT]:
  Enter lowest supported Nim version [0.13.0]:

  $ ls
  total 4.0K
  -rw-rw-r--. 1 cji cji 188 13/02/2016 01:02 slock.nimble
  $ cat slock.nimble
  # Package

  version       = "0.1.0"
  author        = "Piotr Klibert"
  description   = "Simplest possible screen locker for X"
  license       = "MIT"

  # Dependencies

  requires "nim >= 0.13.0"

  $ mkdir src
  $ touch src/slock.nim

The generated slock.nimble is a configuration file for Nimble; it's written in a NimScript, which looks like a very recent development in Nim and it replaces the previous INI-style config files. This means that many examples and tutorials on the Internet won't work with this format. The most important difference is the format of dependencies list for your app: it now has to be a seq. For example, to add x11 library to the project:

1

  requires @["nim >= 0.13.0", "x11 >= 0.1"]

The dependencies should be downloaded by Nimble automatically, but you can also dowload them manually, like in the example below. You'll notice that I also install a c2nim package - I will say more about it later.

1
2
3
4
5

  $ nimble install c2nim x11
  Searching in "official" package list...
  Downloading https://github.com/nim-lang/c2nim into /tmp/nimble_18934/githubcom_nimlangc2nim using git...
  Cloning into '/tmp/nimble_18934/githubcom_nimlangc2nim'...
  ...etc, etc, etc...

Workflow and tooling

Nim is a compiled language, but working with it proved to be comparable to working with a dynamic language, mainly thanks to type inference and blazingly fast compiler. You dcan ommit type declarations where they are obvious from the context and Nim will deduce the correct type for you. It's not a full program type inference like in OCaml, but rather a local type inference as used in Scala or modern C++ or Java. Even with this limitation it's immensely useful and reduces code verbosity by a lot.

Compiler speed is important, because it encourages frequent testing. If compilation is going to take a long time you tend to "batch" changes in your code together and only compile and run once in a while instead of after every single change. This, in turn, makes it harder to find and fix regressions if they appear. Dynamic languages work around this issue by rejecting compilation step completely, at the cost of run-time safety and performance. Nim - which is similar to Go in this respect - makes compilation effortless and nearly invisible. The command for compile-and-run looks like this:

1

  $ nim c -r module.nim

The problem with this command is that it doesn't know about Nimble and dependencies, so in reality I used a bit different commands:

1
2
3
4

  $ nimble build && sudo ./slock

  # or, if you need to pass some additional parameters to Nim compiler:
  $ nimble c --dynlibOverride:crypt --opt:size --passL:"-lcrypt" -d:release src/slock.nim  -o:./slock && sudo ./slock

While Nim's compiler and Nimble are very good tools, they're not enough to work comfortably on more complex codebases. Nim acknowledges this and provides a couple of additional tools for working with code, like nimsuggest. However, nimsuggest is rather new and it SEGFAULTed on me a couple of times^[6]. I used it via - of course - Emacs with nim-mode, and again, I encountered a couple of irritating bugs, frequent "Mismatched parens" errors when trying to jump a word or expression forward. However, when nimsuggest works, it does a good job, particularly its "Go to definition" feature works and is very helpful.

Nim's definitive source of documentation is The Index, which does a surprisingly well as a reference. Ctrl + f works just as well as search boxes other documentation sites provide. Nim docs are also nice in that they link to the source: you get a link to the function implementation beside its documentation. I like this trend and I'm happy that more documentation is like this - a quick peek at the source can sometimes save hours of unneeded work.

In this project, I had to work with Xlib and turns out its documented in man pages, and the pages were already installed on my system. I don't remember installing them, so maybe they come with X by default on Fedora. Anyway, for Xlib tutorial I used Xlib Programming Manual and for reference I simply used the man pages: just type, for example, man XFreePixmap and you get XCreatePixmap (3) man page. The same is true for Linux/POSIX functions - try, for example, man getpwuid.

That's it, for now

In the next part I'm going to show some Nim code, translated - both manually and automatically - from C. I'll focus on Nim's language features that C doesn't have and I'll show how they can be used to write shorter, safer and more readable code. In the later, last part I'm going to write about Nim's interop with C and various ways of connecting the two, using Xlib as an example.

• And it seems that JWZ doesn't like people modifying the looks of this dialog, so I didn't even try. ↵
• Very well written C. ↵
• Every power gamer will understand how nice this is! ↵
• But note the lack of unnecessary return statement in Nim. ↵
• I recommend installing from GitHub - it's always good to have the sources for your compiler and stdlib around, and you won't get it if you install just the binaries. And master branch is stable: checkout devel branch for the in-development code. ↵
• This might be because of my own incompetence, of course. ↵

Recently I was writing a little set of scripts for downloading and analyzing manga meta-data, for use with my Machine Learning pet project. True to a polyglot style, the scripts were written in two languages: Python and Elixir. Elixir is a language that works on Erlang VM (see my previous blog post on it, Weekend with Elixir) and Python doesn't need any introductions I think. I used Elixir for network related stuff, and Python with pandas for analysis.

The biggest problem with such a setup (and with polyglot approach) is passing the data around between the languages. At first I just used a couple of temporary JSON files, but then I remembered a project I once saw, called ErlPort, which is a Python implementation of the Slang's external term protocol. In short, it enables seamless integration between Python and Erlang and - by extension - Elixir. ErlPort not only lets you serialize data, but lets you also call functions across language boundaries. It also supports Ruby besides Python.

In general, using ErlPort was a success in my case, but it's limited as it can only connect three languages. I'd normally say it's good enough and leave it, but a couple of days later, in "the best of Python lib of 2015" thread on HN, I discovered another project called python-bond, which provides similar interoperability features for Python and three more languages: PHP, Perl, JavaScript. The two libraries - ErlPort and python-bond - make it embarrassingly easy to integrate a couple of different languages in a Python project. Along with Cython, which lets you easily call C-level functions, it makes Python a very good base language for Polyglot projects.

The hardest part with polyglot style, and in particular architecture, is deciding which language should be a base one, the one which runs and integrates the rest of the languages. In my case, I ended up with most of the control flow inside Elixir, because of the very neat supervision tree feature it provides (thanks to Erlang/OTP). It remains to be seen if Elixir is capable of filling the role of a main implementation language in a polyglot project. I'll make sure to write a post about it in the future.

For a very long time I used normal search to get to a next occurence of something. However, it wasn't very comfortable. For this to work, I need to:

I found a better way. iedit is a mode for Emacs for editing all the occurences of something at once. I alternate between iedit and multiple-cursors mode when I need to do something simple to a word in many places in the code. However, iedit also provides an iedit-next-occurence, which by default is bound to TAB.

Using iedit I only need to:

• move point to anywhere inside a word
• press C-; for iedit
• press TAB to jump to next and S-TAB (shift + tab) to jump to previous occurence

One more feature of iedit I find useful sometimes is a toggle-unmatched-lines-visible command. It duplicates occur mode functionality a bit, but it hides unmatched lines in your current buffer. This makes it easy to quickly switch between a global occurences list and a local occurence context.

After reading Lenses in Pictures I felt rather dumb. I understood what is the problem, but couldn't understand why all the ceremony around the solution. I looked at OCaml Lens library and became enilghtened: it was because Haskell! Simple.

Anyway, the code looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79

z = require "lodash"            # `_` is a syntactic construct in LS
{reverse, fold1, map} = require "prelude-ls"

pass-thru = (f, v) --> (f v); v
abstract-method = -> throw new Error("Abstract method")


#
# LensProto - a prototype of all lens objects.
#
LensProto =
    # essential methods which need to be implemented on all lens instances
    get: abstract-method (obj) ->
    set: abstract-method (obj, val) ->
    update: (obj, update_func) ->
        (@get obj)
        |> update_func
        |> @set obj, _

    # convenience functions
    add: (obj, val) ->
        @update obj, (+ val)


#
# Lens constructors
#
make-lens = (name) ->
    LensProto with
        get: (.[name])
        set: (obj, val) ->
            (switch typeof! obj
                | \Object => ^^obj <<< obj
                | \Array  => obj.slice 0  )
            |> pass-thru (.[name] = val)

make-lenses = (...lenses) ->
    map make-lens, lenses


#
# Lenses composition
#
comp-lens = (L1, L2) ->
    LensProto with
        get: L2.get >> L1.get
        set: (obj, val) ->
            L2.update obj, (obj2) ->
                L1.set obj2, val

# Lensable is a base class (or a mix-in), which can be used with any object and
# which provides two methods for obtaining lenses for given names. The lens
# returned is bound to the object, which allows us to write:
#
#   obj.l("...").get()
#   obj.l("...").set new_val
#
# instead of
#
#   make-lens("...").get obj
#   make-lens("...").set obj, new_value
#
Lensable =
    # at - convenience function for creating and binding a lens from a string
    # path, with components separated by slash; for example: "a/f/z"
    at: (str) -> @l(...str.split("/"))

    l: (...names) ->
        # create lenses for the names and compose them all into a single lens
        lens = reverse names
            |> map make-lens
            |> fold1 comp-lens

        # bind the lens to *this* object
        lens with
            get:  ~> lens.get this
            set: (val) ~> lens.set this, val

to-lensable = (obj) -> Lensable with obj

Some tests for the code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60

#
# Tests
#

o = Lensable with
    prop:
        bobr: "omigott!"
        dammit: 0


[prop, dammit] = make-lenses "prop", "dammit"
prop-dammit = comp-lens dammit,  prop


console.log z.is-equal (prop.get o),
    { bobr: 'omigott!', dammit: 0 }

console.log (prop-dammit.get o) == 0

console.log z.is-equal (prop-dammit.set o, 10),
    { prop: { bobr: 'omigott!', dammit: 10 } }

prop-dammit
    |> (.set o, "trite")
    |> (.l("prop", "bobr").set -10)
    |> z.is-equal { prop: { bobr: -10, dammit: 'trite' } }, _
    |> console.log


out = o
    .at("prop/bobr").set "12312"
    .at("prop/argh").set "scoobydoobydoooya"
    .at("prop/lst").set [\c \g]
    .at("prop/dammit").add -10
    .l("prop", "lst", 0).set \a
    .l("prop", "lst", 2).set \a

console.log z.is-equal out, {
    prop: {
        bobr: '12312', dammit: -10,
        argh: 'scoobydoobydoooya',
        lst: ["a", "g", "a"]}}


out = o
    .at("prop/bobr").set "12312"
    .at("prop/argh").set "scoobydoobydoooya"
    .at("prop/dammit").add -10

console.log z.is-equal out,
    { prop: { bobr: '12312', dammit: -10, argh: 'scoobydoobydoooya' } }


transform =
    (.at("prop/bobr").set "12312") >>
    (.at("prop/argh").set "scoobydoooya") >>
    (.at("prop/dammit").add -10)

console.log z.is-equal  (transform o),
    { prop: { bobr: '12312', dammit: -10, argh: 'scoobydoooya' } }

This showcases many of the ways you can use lenses in LiveScript. LS has many ways of creating functions and it has syntax for pipe'ing and composing functions, so it's a natural fit for everything FP-looking. LS does not do "curried by default" like Haskell or OCaml, but it gives you a partial application syntax and allows defining curried functions. It's much like Scala in this regard.

Anyway, this is what lenses are supposed to do - they should support functional updates over data sctructures. They should offer get, set and update methods, and also there should be a compose operator for them. And that's all - you can read the linked OCaml implementation to see that it's really that simple. Most of that implementation are convenience methods for creating lenses for various OCaml structures; the core code is really short.

The first day of the conference is nearing an end right now, I don't have much time, but I decided to at least enumerate the talks I attended. I'll probably try expending on them later in the evening.

1. Reducing the Slippery Surface of Failure with Dependent Types - a talk about dependent types in Scala, and their applications. Examples given were: command line parsing, string internationalization.
2. Concurrency + Distribution = Scalability + Availability, a Journey architecting Erlang Systems - very interesting talk about the higher-level things you need to think about when desigining any kind of distributed system.
3. The Pendulum - batch processing vs. interactive computing in the eyes of Bruce Tate. Some interesting anecdotes about the history of computing, Java and scalability.
4. Function-Passing, A New Model for Typed, Asynchronous and Distributed Programming - another Scala-themed talk, this time about a "new" model of distributed programming. If I understand the concept correctly it's not exactly new, for example PicoLisp uses something rather similar. But, this being Scala, this approach guarantees static, compile-time checking for functions which are not suitable for passing around.