String interpolation is a convenient language feature which makes creating human-readable strings that contain variables easier. The classical approach to this in languages without string interpolation is something like printf - a function which accepts a format string and a variable number of arguments. The format string contains special sequences of characters which get replaced with the values of passed arguments, like this:

1
2

printf("An error occured at %s:%d.", "file-name.c", 42);
// Displays: An error occured at file-name.c:42.

This has various problems, for example the possibility to supply more (or fewer) arguments than expected. String interpolation allows you to embed variables directly into the format string:

1
2
3

LINE=42
FILE="file-name.c"
echo "An error occured at $FILE:$LINE."

As with many features, some languages provide string interpolation and some don't. In the dark and troubled past of JavaScript, for example, some people switched to CoffeeScript for its string interpolation support (among other things.) More recently, string interpolation came to Python in the form of f-strings (PEP-498) and to plain JavaScript in the form of template literals (supported by Babel and other transpilers). While the feature is most often associated with dynamic, dynamically typed languages, some statically typed ones, for example C# and Swift, also support it, so it's not exclusive to the former.

Help! No string interpolation in Racket!

Racket - like JS at one point - lacks the string interpolation support. If Racket was JS, the only options would be to either write a transpiler, or switch to another language. Both choices normally require a lot of work, so developers tend to learn to live without the feature instead and move on.

But Racket is not JS, it's a Lisp. All Lisps are programmable programming languages^[1], but Racket takes it up to eleven with the multidute of tools for language extension (and creation)^[2]. How much work would it be to make Racket interpolate strings?

How Racket processes code

To process any code, Racket performs these steps:

• reading - accomplished by two functions, read and read-syntax, reading means taking a string and returning an AST^[3] (which happens to be a normal list with normal symbols, strings, numbers, etc. Such a list can be wrapped with a syntax object, which adds some meta-data to the raw list. You can always get a raw list out of syntax object with syntax->datum).
• expansion - all the special forms and macros in the read code are expanded until only the most basic forms (modules, function applications, value literals) are left.
• compilation - Racket performs byte-code compilation on the expanded code. This is where most of the optimizations are applied.
• execution - compiled byte-code is handed to the VM for execution. At this stage, on some architectures, further JIT compilation happens, turning (possibly portions of) bytecode into native code.

The first two steps are completely customizable: code expansion with macros, and code reading via hooks into the Racket parser (you can also write a new parser from scratch (or generate it) if you want, but frequently it's not necessary). Symbolically, what happens during reading and expansion can be presented^[4] like this:

Extending the parser

The Racket parser is a simple, recursive-descent one and it works by utilizing a readtable - a mapping from characters to functions responsible for parsing what follows the given character. The read and read-syntax functions utilize current-readtable, which is a parameter you can override. Adding a new syntax to the reader requires just three steps:

1. create a function for parsing the new syntax into valid Racket forms
2. get the current-readtable and add your function to it
3. make that expanded readtable from the prev step the new current-readtable

Interpolated string syntax

We'll get to writing the parsing function in a minute, but first we need to know what is it we're going to parse and what the results of the parsing should be. The exact syntax I have in mind would look like this (with #@ distinguishing interpolated strings from normal ones):

1

#@"An error occured at @{file}:@{line}."

Interpolated string compilation

String interpolation, for example in CoffeeScript, is often compiled to an expression which concatenates the chunks of the string, like this:

1
2
3

"An error occured at #{file}:#{line}."
# would be compiled to:
"An error occured at " + file + ":" + line + "."

In Racket we have no + operator for concatenating the strings and even if we did have it, the prefix notation and resulting nesting wouldn't be pretty. The easier and more natural way is to use a string-append procedure, which - conveniently - accepts any number of arguments. The syntax introduced in the previous section would compile down to something like this:

1

(string-append "An error occured at " file ":" line ".")

The parsing function

The parsing function needs to take the string form as an argument and output the code as above. Here's the simplest implementation I could think of:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

(define (parse str)
  ;; Assume that @{} cannot be nested and that the braces are always matched.
  ;; Obviously, in a real parsing function, these assumptions would need to be validated.
  (define lst (regexp-split #rx"@{" str))
  ;; After splitting we have a list like this: '("An error occured at " "file}: " "line}.")
  ;; We'll go over it, building a list of expressions to be passed to string-append as we go.
  (define chunks
    (for/fold ([result '()])
              ([chunk (rest lst)])        ; we don't need the first element here
      (let*
          ;; convert a string into a port
          ([is (open-input-string chunk)]
           ;; call original read to get the expression from inside brackets
           [form (read is)]
           ;; read what remains in the port back into a string
           [after-form (port->string is)]
           ;; drop the closing brace
           [after-brace (substring after-form (add1 (s-index-of after-form "}")))])
        ;; ~a is a generic "toString" function in Racket
        (append result `((~a ,form) ,after-brace)))))
  ;; chunks now looks like this: ((~a file) ":" (~a line) ".")
  `(string-append ,(first lst) ,@chunks))

It's dead simple - just a couple of lines - and it glosses over important details, like the possibility of nesting the @{} expressions and syntax errors handling, but it works for the simple cases. We can easily test it, like this:

1
2
3
4

(parse "An error occured at @{file}:@{line}.")
;; Returns:
'(string-append "An error occured at " (~a file) ":" (~a line) ".")
;; which is exactly what we wanted!

The boilerplate code

The hard part ends here - the rest is just some boilerplate we need to make Racket aware of our extension to its reader. In Racket, this is often done by defining a new language - working with #lang lang-name syntax, but in our case we don't need (or want) to create a whole language (it's simpler than it sounds, but it's still a bit more work) - we just change the reader and can reuse everything else from some other language.

For situations like these, Racket supports a special syntax to its #lang line, which looks like this:

1
2
3

#lang reader <reader-implementation-module> <language-the-reader-will-be-used-with>
;; for example:
#lang reader "reader.rkt" racket

Generating a reader module

Let's first define reader.rkt. Writing a whole new reader is of course an option but it's a lot of unnecessary work, as the generic read and read-syntax^[5] need to handle some more complex things, like module declarations, which we don't care about. Instead, we can use a special language for defining readers, which will generate appropriate read and read-syntax implementations for us provided we supply a single function to it:

1
2
3
4
5

#lang s-exp syntax/module-reader
#:language read
#:wrapper1 wrap-read

(require "ext.rkt")

The #:language option takes a function which accepts a string (of what remains after the reader module name on the #lang line) and returns a language name to use. A simple read happens to work here.

The #:wrapper1 option takes a function, which takes another function as input and returns the parsed code. The passed in function, when invoked, performs normal read.

Extending and installing a readtable

With #:wrapper1 option, in conjunction with parameterize, it's very easy to install our readtable^[6]:

1
2
3
4
5
6
7
8

(define (wrap-read do-read)
  (define rt (make-readtable (current-readtable)
                             ;; register our function to be called when #@ is encountered
                             #\@ 'dispatch-macro
                             ;; name of the function to be called
                             read-str))
  (parameterize ([current-readtable rt])
    (do-read)))

Wrapping parse so that it works inside readtable

The only part missing is a read-str function, which is a bit more complicated. The function passed to make-readtable may be called in two different ways, depending on whether it's used from inside read or read-syntax. The function distinguishes the cases based on a number of arguments it gets, and returns either syntax object or a simple list depending on the case. Here's its implementation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

(define read-str
  (case-lambda
    [(ch in src line col pos)
     ;; The caller wants a syntax object. The current position of the `in` port
     ;; is just after the #@ characters, before the opening quote.
     (let*
         ;; Here we read the next expression (which we assume is a string)
         ([literal-string (read in)])
       ;; then we parse the contents of the string and return the result,
       ;; wrapped in a new syntax object. The `in` port is now just after the
       ;; closing quote and the parser continues from there.
       (datum->syntax #f (parse literal-string)))]

    [(ch in)
     ;; The caller wants a simple list. Let's parse the input into syntax and
     ;; transform to a list (this saves as writing the same code twice).
     (syntax->datum (read-syntax #f in))]))))

It works!

The reader extension can be used like this:

1
2
3
4
5
6

#lang reader "reader.rkt" racket

(define line 42)
(define file "file-name.rkt")
(displayln #@"An error occured at @{file}:@{line}.")
;; Displays: An error occured at file-name.rkt:42.

It also works for more complex expressions out of the box:

1
2
3
4
5

#lang reader "reader.rkt" racket
(require racket/date)

(displayln #@"Current date&time: @{(date->string (current-date) #t)}")
;; Displays: Current date&time: Friday, April 28th, 2017 7:57:11pm

This is of course a pretty basic as far as string interpolation features go, but it took thirty (34 to be precise) lines of code (excluding comments) to build it. Adding another twenty lines would be enough to implement error checking, and using one of the parser generators could even shorten the code.

That's it!

Aside from how easy it is to extend Racket syntax one thing deserves a mention. That is, such syntax extensions are easily composable (as long as they don't want to install extensions to the same entry in the readtable) - you can load them one by one, each extending the readtable.

The language extension and creation tools of Racket don't end there. Robust module system with support for exporting syntax extensions, powerful syntax-parse, traditional syntax-rules and syntax-case, many tools for generating parsers and lexers - all these features are geared towards meta-lingustic programming. Racket is a programming language laboratory and allows you to bend it to better suit your needs safely (ie. ensuring that only your code will be affected). Very few languages or transpilers offer that much freedom with that kind of safety guarantees.

It's an ideal environment for people who know what they need from their language - with Racket you're never stuck, waiting for the next release to support some feature. On the other hand, despite there being a great many helpers and libraries, you need to know at least some basics of programming language creation to do this yourself. The good news is that the extensions written by others are easily installable with a single command of a Racket package manager.

Addendum: doing the same without messing with the reader

A commenter on Hacker News provided an interesting solution using normal macros only. Turns out there's a special macro, called #%datum, which gets wrapped around every literal in the code (when the code is read). By default it's a noop, but we can override it with arbitrary logic. In this solution there's no need for marking strings for interpolation: all strings are interpolate-able by default. Here's the implementation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

(require
  (rename-in racket (#%datum core-datum))
  (for-syntax syntax/parse))

(define-syntax (#%datum stx)
  (syntax-parse stx
    [(_ . x:str) (interpolate #'x)]
    [(_ . x)     #'(core-datum . x)]))

(define-for-syntax (interpolate stx)
  (define re            #rx"@{[^}]+}")
  (define (trim match)  (substring match 2 (- (string-length match) 1)))
  (define (to-stx val)  (datum->syntax stx val))
  (define (datum val)   (to-stx (cons #'core-datum val)))
  (define (source text) (to-stx (read (open-input-string text))))
  (let* ([text     (syntax->datum stx)]
         [matches  (regexp-match* re text)]
         [template (datum (regexp-replace re text "~a"))]
         [values   (map (compose source trim) matches)])
    (if (null? matches)
        template
        (to-stx `(,#'format ,template ,@values)))))

I don't have the time to go over this line by line and explain it right now, but it's possible I'll write another post featuring this solution in the future.

• The quote comes from Paul Graham's famous book and is attributed to John Foderaro. ↵
• A good starting point for learning more would be Racket is ... article by Matthias Felleisen. At least I assume it's his - it's not signed, but clearly in Matthias home directory ;) ↵
• Abstract Syntax Tree, see Wikipedia ↵
• Taken from a paper Type Systems as Macros by Stephen Chang, Alex Knauth, Ben Greenman ↵
• Any module, which exports these two functions, can be used as a reader. ↵
• In the actual source code the (make-readtable ...) part is extracted into separate procedure, which makes it easier to compose this syntax extension with others. ↵

• See RⁿRS for an official definition of Scheme. The newest Report is in this document ↵
• See this document for an in-depth explanation of what that means. ↵
• See wiki ↵
• Racket now supports three main types of macros, the latest documented here ↵
• Now called TypedRacket ↵
• See design by contract ↵
• Called PLaneT back then, Racket since changed its package management tool and now the repo is here. ↵
• Racket includes DSLs for making slides and writing Literate Programs, among other things.↵
• It only stopped a couple of years later with Quicklisp ↵
• See wiki ↵
• In Lisps known as DESTRUCTURING-BIND ↵
• In 2011, or at least that's when the repo starts ↵
• Transducers are one of the more famous (and recent) examples. ↵
• Called Leiningen. ↵

Apparently, I can't directly upgrade to whichever version I choose, I need to go one by one, doing fedora-upgrade and rebooting in a loop as needed.

I realize that my case is somewhat extreme, but watching the same set of packages getting downloaded five times (~640MB each time) feels wrong somehow.

The good news are that the upgrade took its time, but in the end finished successfully. Quite a feat for that large defference in versions: default package manager and system upgrade tools changed in the meantime, but still managed to produce a working system with minimal fuss.

Damn, I'm getting back to Linux as fast as I find a decent laptop; working on Mac OS - although, with effort, made vaguely ok - is tiring as I need to cope with all the "oh, it doesn't work that way here" moments...

RSS feed

As you can probably guess with a quick glance, this blog is self-hosted by me and is a result of "static page generation", which means there's a bunch of files and a few scripts which output all the needed HTML/CSS/JS to run the site, you just upload it to the server and you're done. This approach is very popular lately and many bloggers do the same.

One - slightly - unusual bit is that I wrote all of the generators and scripts myself, from scratch (using mainly Python). This means, among other things, that I have no pile of already written plugins or modules I could resort to in case I found my generator lacks an important feature...

For a long time I was pretty much the only user of both the generator and the blog, so I didn't mind. Recently, though, quite a few people who said they'd like to read my next post (my prospective readers! Finally, after four years, I have a chance to get my very own readers! ...err, sorry, got a bit too happy there for a moment...) suggested adding RSS feed to the blog.

I had known nothing about RSS and it took a couple of hours, but I finally added RSS feed generation to my scripts. Now every time I publish a post, a new /statics/feed.xml gets generated and, hopefuly, every subscriber will be notified!^[1] Even better, you will be even notified about post updates (probably)!

Thank you! (Or: Rkt vs. Clj is comming!)

Ok, so I know simply adding RSS feed is hardly important enough to write a post about it, so here is the other part: thank you very much! All of you who upvoted my comment on Hacker News or even sent me an email - only to encourage me to write the promised post, thank you. It's thanks to you that I got motivated enough to finally start writing that post about Clojure and Racket comparison.

Please stay tuned and be patient: it's a lot of work, especially because I want to stay as fair as possible and give as many detailed examples as possible (without copyright-related problems).

So, once again, thank you for expressing your interest in my future post and please stay patient while waiting for me to finish; I'll try my best, on my part, to write it all well and on time!

• Let me know if it doesn't work for you or your reader! I don't know much about RSS and could have messed up). ↵

I worked on a Web project in Clojure and saw how frontend/JS devs had a hard time adjusting to the syntax. Here is a breakdown of which tools I find essential for editing lisps as well as a list of possible tool suites (IDEs with plugins, standalone editors, etc.).

It's normal, I think, to get overwhelmed by the sheer amount of parens you need to manage when learning a Lisp. When taken on its own, the syntax really is simple, but introducing another strange thing on top of already unfamiliar concepts doesn't help learning.

Some people will tell you that you will "get used to it" soon enough and that it's just a matter of practice. While there is some truth to these claims, I'm a pragmatist and so I prefer another approach: simply set up a decent environment for working with Lisp before you begin. There are many tools which make reading and writing Lisp code much easier: you need to either configure your editor to enable them or change the editor. There are some Clojure-specific editors out there, you can simply use one of them (my thoughts on them below).

Main things/features useful when working with Lisps:

• syntax highlighting for your Lisp (obviously!)
• automatic indentation (and re-indent) for your Lisp
• ability to highlight matching paren
• ability to jump to matching paren
• configurable syntax coloring and/or rainbow-style coloring of parens
• ability to wrap a text selection with parens
• automatically inserting closing paren
• ability to delete parens in pairs automatically
• par-edit or par-infer style structural editing
• auto-completion for module/function names^[1]
• quick access to the docs for a name under cursor
• "Go to definition..." is good to have, but you can usually make do with grep or "Find in files..." editor command

See the screenshots and a video for visual demonstration of what I'm talking about (click on the image to get a bigger version):

Also see here for a very good general introduction to editing Lisp along with explanation of what Parinfer is.

Clojure specific editors

After writing the above I realized that it would be good to give a couple of examples of beginner-friendly editors, which implement the features mentioned above. To my surprise there are some editors which target Clojure, some of them even maintained. Here's a short summary of what I found out:

Nightcode - I only played with it a little but I'm impressed at what it can do. I tried using parinfer some time back and it is much friendlier and easier to use than Paredit^[2]. In short: you never need to worry about parens when using Nightcode. Place a cursor where you need it and start typing: the parens will magically appear. It worked out of the box for me on Mac, however it refused to run on Fedora.

Clooj - inactive since 2012, has nearly none of the useful features, looks ugly and is slow. Forget it.

LightTable - supports many things mentioned above as plugins, but they tend to be disabled by default and enabling them is not as easy as it should be. The plugin-based architecture makes it interesting for polyglot projects, but it needs some configuration to get started and Nightcode needs none.

Cursive - a Clojure plugin for IntelliJ IDEA from JetBrains. I don't have any of their IDEs installed and so I didn't try it. Its feature list looks decent, though, so if you like IntelliJ this may be the best option for you.

Emacs and CIDER - that's what I use. I'd recommend you to only try this route if you already know some Emacs, otherwise it's going to be frustrating for a good while until you internalize all the strange names and such. CIDER itself is great, though, and integgrates with Leiningen, offers inline code evaluation, auto-completion, connecting to a running REPL and so on.

In short: give Nightcode a try if you can, otherwise use a Lisp/Clojure plugins for your current editor, like Sublime, IntelliJ or Eclipse. Come over to Emacs side once you get bored with those.

• A full "IntelliSense" - context aware auto-completion - is always nice to have, but you can live without it. For a while. ↵
• I stick with Paredit because I already got used to it and am efficient enough with it; I'd go for parienfer had I started learning about Lisps now. ↵

I gave a talk on 15.03 about polyglot programming, based on the example project I wrote some time before. The project was a web scraper, with downloading part written in Elixir and processing part written in Python.

SLIDES

The talk was recorded, so I'll link to the video as soon as it becomes available.

I gave a talk at work yesterday about OpenRESTY, which I think is the easiest way to start scripting Nginx. Here are the slides:

SLIDES

One thing I'd like to add is that I don't trust OpenRESTY as an app platform yet. The project is being actively developed and the author writes tons of good code, but he is but a single man. This means there is severe lack of tools and libraries in the ecosystem: they simply don't exist yet.

On the other hand, OpenRESTY is a painless way to script Nginx with Lua and that's something you can do with just the modules provided by OpenRESTY. Being able to communicate asynchronously with anything (literally: anything that wants to talk over sockets), including databases, external services and so on, is very nice thing indeed.

There's one caveat though, which is that you can't use any blocking (non-async) constructs, which includes using normal Lua sockets. This is not the problem in your code, but if you happen to need a Lua library which comes as a C extension, and if the functions in the library block, you will probably need to fix the C code. At this point it's purely theoretical, because I didn't encounter any such library yet.

TL;DR or What is this post about?

In short: it's about me exploring - or at least getting into contact with - a couple of interesting things:

• X Window System APIs via Xlib bindings
• low-level Linux APIs for elevating privilages, checking passwords
• GCC options and some C
• and of course the Nim language

Introduction

At my work we strongly discourage leaving logged-in accounts and/or unlocked screens of computers. I happen to agree that locking your computer is a good habit to have, so I've had no problems with this rule... Up to the point when I switched from KDE to StumpWM (I wrote about it some time ago, in this, this and this posts) and my trusty Ctrl + Alt + L stopped working.

The only idea that came to my mind was to use the venerable xscreensaver, but: a) I didn't really need any of the 200+ animations (I just wanted a blank screen) and b) I didn't like how the unlock dialog looked like^[1].

I needed something more lightweight (xscreensaver is ~30k loc of C^[2]), simpler and either better looking or without any UI altogether.

slock to the rescue

There's a site called suckless.org where you can find some impressively clean and simple tools, implemented mostly in C. You can read more about their philosophy here, which I recommend as an eye opening experience. Anyway, among the tools being developed there, there is also slock - a very simple and basic screen locker for X. It's 310 lines of code long and it's all pretty straightforward C.

The program suited my needs very well: minimal, fast, and good looking. Well, the last part it got through cheating, as slock simply has no interface at all - but this means there's no ugly unlock dialog, so it's all good.

Why Nim?

As I used slock I read through its code a couple of times. It seemed simple and learnable, despite the fact that I knew nothing about X and didn't use C seriously in 15 years. Fast forward to last week and I finally found some free time and decided to learn some Xlib stuff. Re-implementing slock in Nim looked like a good way of learning it: this may sound a bit extreme, but it allowed me to gain exp points in two stats simultaneously^[3] - in Xlib usage and Nim knowledge!

Nim is a very interesting language. I'd call it "unorthodox" - not yet radical, like Lisp or Smalltalk, but also not exactly aligned with Java and the like. For one tiny example: return statement in functions is optional and if omitted, a function returns its last expression's value. That's rather normal way to go about it; hovewer, in Nim you can also use a special variable name result and assign to it to have the assigned thing returned. It looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

  proc fun1() : int =
    return 1

  proc fun2() : int =
    1

  proc fun3() : int =
    result = 1

  assert fun1() == fun2() == fun3()

At a first glance this may look strange, but consider this in Python:

1
2
3
4
5

  def fun():
      ret = []
      for x in something:
          ret.append(...something...)
      return ret

It's practically an idiom, a very common construct. Nim takes this idiom, adds some sugar, adapts it to the world of static typing and includes in the language itself. We can translate the above Python to Nim with very minor changes^[4]:

1
2
3
4

  proc fun4() : seq[int] =
    result = @[]
    for x in something:
      result.add(...something...)

As I said, it's not a groundbreaking feature, but it is nice and good to have, and it shows that Nim doesn't hesitate much when choosing language features to include. In effect Nim includes many such conveniences, which may be seen both as a strength and as a weakness. While it makes common tasks very easy, it also makes Nim a larger language than some others. It's not bad in itself, rather, depending on how well the features fit together it's harder or easier to remember and use them all. Nim manages well in this area, and also it most definitely is not like C++ with its backwards compatibility problem, so I think even Pythonistas with taste for minimalism will be able to work with Nim.

Being Nimble - the project setup

Many modern languages include some kind of task runner and package manager either as part of stadard distribution or as downloadable packages. Nim has Nimble, which takes care of installing, creating and publishing Nim packages. Assumming that you have Nim already installed^[5], you can install Nimble with:

1
2
3
4
5

  $ git clone https://github.com/nim-lang/nimble.git
  $ cd nimble
  $ git clone -b v0.13.0 --depth 1 https://github.com/nim-lang/nim vendor/nim
  $ nim c -r src/nimble install
  $ export PATH="$PATH:~/.nimble/bin/"

Note the addition to the PATH variable: ~/.nimble/bin/ is where Nimble installed itself and where it will place other binaries it installs. Make sure to have this directory in your PATH before working with nimble.

Creating a project

Creating a project is easy, similar to npm init:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

  $ mkdir slock
  $ cd slock
  $ nimble init
  In order to initialise a new Nimble package, I will need to ask you
  some questions. Default values are shown in square brackets, press
  enter to use them.
  Enter package name [slock]:
  Enter intial version of package [0.1.0]:
  Enter your name [Piotr Klibert]:
  Enter package description: Simplest possible screen locker for X
  Enter package license [MIT]:
  Enter lowest supported Nim version [0.13.0]:

  $ ls
  total 4.0K
  -rw-rw-r--. 1 cji cji 188 13/02/2016 01:02 slock.nimble
  $ cat slock.nimble
  # Package

  version       = "0.1.0"
  author        = "Piotr Klibert"
  description   = "Simplest possible screen locker for X"
  license       = "MIT"

  # Dependencies

  requires "nim >= 0.13.0"

  $ mkdir src
  $ touch src/slock.nim

The generated slock.nimble is a configuration file for Nimble; it's written in a NimScript, which looks like a very recent development in Nim and it replaces the previous INI-style config files. This means that many examples and tutorials on the Internet won't work with this format. The most important difference is the format of dependencies list for your app: it now has to be a seq. For example, to add x11 library to the project:

1

  requires @["nim >= 0.13.0", "x11 >= 0.1"]

The dependencies should be downloaded by Nimble automatically, but you can also dowload them manually, like in the example below. You'll notice that I also install a c2nim package - I will say more about it later.

1
2
3
4
5

  $ nimble install c2nim x11
  Searching in "official" package list...
  Downloading https://github.com/nim-lang/c2nim into /tmp/nimble_18934/githubcom_nimlangc2nim using git...
  Cloning into '/tmp/nimble_18934/githubcom_nimlangc2nim'...
  ...etc, etc, etc...

Workflow and tooling

Nim is a compiled language, but working with it proved to be comparable to working with a dynamic language, mainly thanks to type inference and blazingly fast compiler. You dcan ommit type declarations where they are obvious from the context and Nim will deduce the correct type for you. It's not a full program type inference like in OCaml, but rather a local type inference as used in Scala or modern C++ or Java. Even with this limitation it's immensely useful and reduces code verbosity by a lot.

Compiler speed is important, because it encourages frequent testing. If compilation is going to take a long time you tend to "batch" changes in your code together and only compile and run once in a while instead of after every single change. This, in turn, makes it harder to find and fix regressions if they appear. Dynamic languages work around this issue by rejecting compilation step completely, at the cost of run-time safety and performance. Nim - which is similar to Go in this respect - makes compilation effortless and nearly invisible. The command for compile-and-run looks like this:

1

  $ nim c -r module.nim

The problem with this command is that it doesn't know about Nimble and dependencies, so in reality I used a bit different commands:

1
2
3
4

  $ nimble build && sudo ./slock

  # or, if you need to pass some additional parameters to Nim compiler:
  $ nimble c --dynlibOverride:crypt --opt:size --passL:"-lcrypt" -d:release src/slock.nim  -o:./slock && sudo ./slock

While Nim's compiler and Nimble are very good tools, they're not enough to work comfortably on more complex codebases. Nim acknowledges this and provides a couple of additional tools for working with code, like nimsuggest. However, nimsuggest is rather new and it SEGFAULTed on me a couple of times^[6]. I used it via - of course - Emacs with nim-mode, and again, I encountered a couple of irritating bugs, frequent "Mismatched parens" errors when trying to jump a word or expression forward. However, when nimsuggest works, it does a good job, particularly its "Go to definition" feature works and is very helpful.

Nim's definitive source of documentation is The Index, which does a surprisingly well as a reference. Ctrl + f works just as well as search boxes other documentation sites provide. Nim docs are also nice in that they link to the source: you get a link to the function implementation beside its documentation. I like this trend and I'm happy that more documentation is like this - a quick peek at the source can sometimes save hours of unneeded work.

In this project, I had to work with Xlib and turns out its documented in man pages, and the pages were already installed on my system. I don't remember installing them, so maybe they come with X by default on Fedora. Anyway, for Xlib tutorial I used Xlib Programming Manual and for reference I simply used the man pages: just type, for example, man XFreePixmap and you get XCreatePixmap (3) man page. The same is true for Linux/POSIX functions - try, for example, man getpwuid.

That's it, for now

In the next part I'm going to show some Nim code, translated - both manually and automatically - from C. I'll focus on Nim's language features that C doesn't have and I'll show how they can be used to write shorter, safer and more readable code. In the later, last part I'm going to write about Nim's interop with C and various ways of connecting the two, using Xlib as an example.

• And it seems that JWZ doesn't like people modifying the looks of this dialog, so I didn't even try. ↵
• Very well written C. ↵
• Every power gamer will understand how nice this is! ↵
• But note the lack of unnecessary return statement in Nim. ↵
• I recommend installing from GitHub - it's always good to have the sources for your compiler and stdlib around, and you won't get it if you install just the binaries. And master branch is stable: checkout devel branch for the in-development code. ↵
• This might be because of my own incompetence, of course. ↵

Recently I was writing a little set of scripts for downloading and analyzing manga meta-data, for use with my Machine Learning pet project. True to a polyglot style, the scripts were written in two languages: Python and Elixir. Elixir is a language that works on Erlang VM (see my previous blog post on it, Weekend with Elixir) and Python doesn't need any introductions I think. I used Elixir for network related stuff, and Python with pandas for analysis.

The biggest problem with such a setup (and with polyglot approach) is passing the data around between the languages. At first I just used a couple of temporary JSON files, but then I remembered a project I once saw, called ErlPort, which is a Python implementation of the Slang's external term protocol. In short, it enables seamless integration between Python and Erlang and - by extension - Elixir. ErlPort not only lets you serialize data, but lets you also call functions across language boundaries. It also supports Ruby besides Python.

In general, using ErlPort was a success in my case, but it's limited as it can only connect three languages. I'd normally say it's good enough and leave it, but a couple of days later, in "the best of Python lib of 2015" thread on HN, I discovered another project called python-bond, which provides similar interoperability features for Python and three more languages: PHP, Perl, JavaScript. The two libraries - ErlPort and python-bond - make it embarrassingly easy to integrate a couple of different languages in a Python project. Along with Cython, which lets you easily call C-level functions, it makes Python a very good base language for Polyglot projects.

The hardest part with polyglot style, and in particular architecture, is deciding which language should be a base one, the one which runs and integrates the rest of the languages. In my case, I ended up with most of the control flow inside Elixir, because of the very neat supervision tree feature it provides (thanks to Erlang/OTP). It remains to be seen if Elixir is capable of filling the role of a main implementation language in a polyglot project. I'll make sure to write a post about it in the future.

For a very long time I used normal search to get to a next occurence of something. However, it wasn't very comfortable. For this to work, I need to:

I found a better way. iedit is a mode for Emacs for editing all the occurences of something at once. I alternate between iedit and multiple-cursors mode when I need to do something simple to a word in many places in the code. However, iedit also provides an iedit-next-occurence, which by default is bound to TAB.

Using iedit I only need to:

• move point to anywhere inside a word
• press C-; for iedit
• press TAB to jump to next and S-TAB (shift + tab) to jump to previous occurence

One more feature of iedit I find useful sometimes is a toggle-unmatched-lines-visible command. It duplicates occur mode functionality a bit, but it hides unmatched lines in your current buffer. This makes it easy to quickly switch between a global occurences list and a local occurence context.

After reading Lenses in Pictures I felt rather dumb. I understood what is the problem, but couldn't understand why all the ceremony around the solution. I looked at OCaml Lens library and became enilghtened: it was because Haskell! Simple.

Anyway, the code looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79

z = require "lodash"            # `_` is a syntactic construct in LS
{reverse, fold1, map} = require "prelude-ls"

pass-thru = (f, v) --> (f v); v
abstract-method = -> throw new Error("Abstract method")


#
# LensProto - a prototype of all lens objects.
#
LensProto =
    # essential methods which need to be implemented on all lens instances
    get: abstract-method (obj) ->
    set: abstract-method (obj, val) ->
    update: (obj, update_func) ->
        (@get obj)
        |> update_func
        |> @set obj, _

    # convenience functions
    add: (obj, val) ->
        @update obj, (+ val)


#
# Lens constructors
#
make-lens = (name) ->
    LensProto with
        get: (.[name])
        set: (obj, val) ->
            (switch typeof! obj
                | \Object => ^^obj <<< obj
                | \Array  => obj.slice 0  )
            |> pass-thru (.[name] = val)

make-lenses = (...lenses) ->
    map make-lens, lenses


#
# Lenses composition
#
comp-lens = (L1, L2) ->
    LensProto with
        get: L2.get >> L1.get
        set: (obj, val) ->
            L2.update obj, (obj2) ->
                L1.set obj2, val

# Lensable is a base class (or a mix-in), which can be used with any object and
# which provides two methods for obtaining lenses for given names. The lens
# returned is bound to the object, which allows us to write:
#
#   obj.l("...").get()
#   obj.l("...").set new_val
#
# instead of
#
#   make-lens("...").get obj
#   make-lens("...").set obj, new_value
#
Lensable =
    # at - convenience function for creating and binding a lens from a string
    # path, with components separated by slash; for example: "a/f/z"
    at: (str) -> @l(...str.split("/"))

    l: (...names) ->
        # create lenses for the names and compose them all into a single lens
        lens = reverse names
            |> map make-lens
            |> fold1 comp-lens

        # bind the lens to *this* object
        lens with
            get:  ~> lens.get this
            set: (val) ~> lens.set this, val

to-lensable = (obj) -> Lensable with obj

Some tests for the code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60

#
# Tests
#

o = Lensable with
    prop:
        bobr: "omigott!"
        dammit: 0


[prop, dammit] = make-lenses "prop", "dammit"
prop-dammit = comp-lens dammit,  prop


console.log z.is-equal (prop.get o),
    { bobr: 'omigott!', dammit: 0 }

console.log (prop-dammit.get o) == 0

console.log z.is-equal (prop-dammit.set o, 10),
    { prop: { bobr: 'omigott!', dammit: 10 } }

prop-dammit
    |> (.set o, "trite")
    |> (.l("prop", "bobr").set -10)
    |> z.is-equal { prop: { bobr: -10, dammit: 'trite' } }, _
    |> console.log


out = o
    .at("prop/bobr").set "12312"
    .at("prop/argh").set "scoobydoobydoooya"
    .at("prop/lst").set [\c \g]
    .at("prop/dammit").add -10
    .l("prop", "lst", 0).set \a
    .l("prop", "lst", 2).set \a

console.log z.is-equal out, {
    prop: {
        bobr: '12312', dammit: -10,
        argh: 'scoobydoobydoooya',
        lst: ["a", "g", "a"]}}


out = o
    .at("prop/bobr").set "12312"
    .at("prop/argh").set "scoobydoobydoooya"
    .at("prop/dammit").add -10

console.log z.is-equal out,
    { prop: { bobr: '12312', dammit: -10, argh: 'scoobydoobydoooya' } }


transform =
    (.at("prop/bobr").set "12312") >>
    (.at("prop/argh").set "scoobydoooya") >>
    (.at("prop/dammit").add -10)

console.log z.is-equal  (transform o),
    { prop: { bobr: '12312', dammit: -10, argh: 'scoobydoooya' } }

This showcases many of the ways you can use lenses in LiveScript. LS has many ways of creating functions and it has syntax for pipe'ing and composing functions, so it's a natural fit for everything FP-looking. LS does not do "curried by default" like Haskell or OCaml, but it gives you a partial application syntax and allows defining curried functions. It's much like Scala in this regard.

Anyway, this is what lenses are supposed to do - they should support functional updates over data sctructures. They should offer get, set and update methods, and also there should be a compose operator for them. And that's all - you can read the linked OCaml implementation to see that it's really that simple. Most of that implementation are convenience methods for creating lenses for various OCaml structures; the core code is really short.

The first day of the conference is nearing an end right now, I don't have much time, but I decided to at least enumerate the talks I attended. I'll probably try expending on them later in the evening.

1. Reducing the Slippery Surface of Failure with Dependent Types - a talk about dependent types in Scala, and their applications. Examples given were: command line parsing, string internationalization.
2. Concurrency + Distribution = Scalability + Availability, a Journey architecting Erlang Systems - very interesting talk about the higher-level things you need to think about when desigining any kind of distributed system.
3. The Pendulum - batch processing vs. interactive computing in the eyes of Bruce Tate. Some interesting anecdotes about the history of computing, Java and scalability.
4. Function-Passing, A New Model for Typed, Asynchronous and Distributed Programming - another Scala-themed talk, this time about a "new" model of distributed programming. If I understand the concept correctly it's not exactly new, for example PicoLisp uses something rather similar. But, this being Scala, this approach guarantees static, compile-time checking for functions which are not suitable for passing around.

Working with TXR without syntax highlighting was getting frustrating, so I finally decided to write a mode for Emacs. It's very simple and offers syntax highlighting only - ie. it scratches my own itch. It may be useful for others anyway.

I will probably expand it as I work with TXR. TXR proves to be a really nice language - its matching rules have some quirks, but once you learn them you can parse nearly anything without much hassle.