micahlerner.com

Unix Shell Programming: The Next 50 Years (The Future of the Shell, Part I)

Published July 14, 2021

Found something wrong? Submit a pull request!

Unix Shell Programming: The Next 50 Years

This week’s paper won the distinguished presentation award at HotOS 2021, and discusses the potential for future innovation in the tool that many use every day - the shell!A previous submission of this paper on Hacker News elicited a number of strong reactions. One reaction was the assertion that there are in fact modern shells - Elvish features most prominently. While I think several of the comments on the original posting are in the right direction, my takeaway from this paper was that modern shells could take advantage of exciting advances in other areas of systems research (in particular, data flow and transparent parallelization of computation).

It not only proposes a way forward to address (what the authors view) as the shell’s sharp edges, but also references a number of other interesting papers that I will publish paper reviews of over the next few weeks - the gist of several of the papers are mentioned further on in this article:

The good, the bad, and the ugly

In Unix Shell Programming: The Next 50 Years, the authors argue that while the shell is a powerful tool, it can be improved for modern users and workflows. To make this argument, the paper first considers “the good, the bad, and the ugly” of shells in order to outline what should (or should not) change in shells going forward.

The paper identifies four good components of modern shells:

Next - four bad features are detailed, with the note that, “It’s hard to imagine ‘addressing’ these characteristics without turning the shell into something it isn’t; it’s hard to get the good of the shell without these bad qualities”As an example, the paper links to previous research that word expansion (“the conversion of user input into…a command and its arguments”) make up a significant portion of user commands. :

Lastly, four ugly components are detailed:

Enabling the shell to move forward

The paper next argues that two sets of recent academic research are enabling the shell to move forward: formalizing the shell and annotation languages.

Recent work on formalizing the shell is detailed in Executable Formal Semantics for the POSIX Shell, which has two major components: Smoosh and libdash - the artifacts for both are open source.

Smoosh is an executable shell specification written in LemWhich can then be translated to different formats, including proof languages like Coq . A shell specification written in code (versus the extensive written specification) meant that the aforementioned paper was able to test various shells for undefined behavior, in the process finding several bugs in implementation (not to mention, bugs in the test suite for the POSIX shell specification!)Another interesting feature of Smoosh is that it provides two interfaces to interact with the OS - one actually invokes syscalls, whereas the other mode simulates syscalls (and is used for symbolic execution). This vaguely reminds me of the testing system used in FoundationDB, covered in a previous paper review. . libdash transforms shell scripts from (or to) abstract syntax trees, and is used by Smoosh.

Annotation languages can allow users to specify how a command runs, in addition to possible inputs and outputs. Strictly specifying a command allows for it to be included as a step (with inputs and outputs) in a data flow graph, enabling more advanced functionality - for example, deciding to divide the inputs of a step across many machines, perform computation in parallel, then coalescing the output. If this type of advanced functionality sounds interesting to you, stay tuned! I’ll be reading about the two papers that fall into this category (PaSH & POSH) over the next few weeks.

After discussing these two research areas, the paper discusses a new project from the authors, called Jash (Just Another SHell). It can act as a shim between the user and the actual execution of a shell command. Eventually, Jash seems like it could implement functionality similar to an execution engine or query planner, evaluating commands at runtime and deciding how to perform the requested work (providing feedback to the user if the script will produce unintended side effects).

The future

The paper outlines five functionalities for the future of the shell:

Conclusion

The shell is an integral part of systems, and this paper makes a case for revisiting the shell’s sharp edges, while revamping its functionality for modern use cases. I’m excited to keep diving deep on this topic - this is the first post in a series I’m doing! If you enjoyed it (or otherwise have suggestions), find me on Twitter. Until next time.

Follow me on Twitter or subscribe below to get future paper reviews. Published weekly.

Found something wrong? Submit a pull request!