Why I am again a Korn Shell Fan

A lot of Linux fanboys these days love everything Gnu, and like to talk shit about non-Gnu stuff. Perhaps it’s just a modern-day version of the empty Commodore vs. Apple vs. TRS-80 arguments we had back in the day: they each had strengths and weaknesses, so a lot depended on what you wanted to do. And a lot had to do with what you were familiar with.

Nevertheless, there are reasons I prefer Korn shell. If you have different reasons, you may prefer other shells. And that’s just fine too—but if you’re just sticking with something because it’s all the rage, that’s not a good reason.

So let’s start with the small stuff and work our way up.

Floating point math

You don’t need it often, but when you do, you don’t have to screw around using bc or dc to do it.

Korn shell’s built-in sleep accepts floating-point values too.

print command

echo has differing variants on different platforms. Ksh’s print works the same everywhere.

Compound variables

You can create a variable that has multiple fields in it. A simple struct of sorts. See typeset -C, which, by the way, you can do with…

Built-in documentation

Try it out: typeset --?? (or typeset --man) to learn about the myriad of options to typeset. No longer do you need to pore through a crazy long man page to find what you want, it’s right at your finger tips. Almost all the built-ins have them. And the man page isn’t all: -? gives a short synopsis, and --help shows the synopsis and option details.

And if you want to print that:

typeset --nroff 2>&1 | groff -man -T ps | ps2pdf - typeset.pdf

It also does --html, --about, --short, --long. See getopts --man for more, and getopts --??help for a full list of display options.

Self-documenting getopts

Those man pages just mentioned? They come from the getopts string using Ksh’s long format. (The commonplace short format is accepted too.) The long format is described in the getopts built-in man page, but you may also need to cookbook from an example; I’ve included some info after this essay. The format is a little weird, but easier than writing raw man pages, which you can create with the --nroff option. And it’s used to provide the same sort of short synopses, long synopses, and man pages on demand. Lastly, the getopts long options string is both directive and documentation, ensuring docs and code are in sync.

Good documentation

Bash has decent documentation, comparable to the quality of Korn shell. zsh documentation, however, can be sparse, especially regarding some of the command options.

FPATH

You can set a function path. If a function is requested and not known, Korn shell searches and loads it if found. You can have a library of functions, without having to copy-and-paste your code (or the bugs that go with that code).

Zero-based Arrays

The first element of an array is 0, like God intended it. Same as it is for strings.

Unlike zsh, which decided arrays should be 1-based (unless you set flag to make them 0-based), even though character positions in strings are always numbered from 0. Holy inconsistency, Batman!

True local variables

In modern shells, there are two ways to declare functions:

name () { ...

and

function name { ...

Usually, these behave the same. And because of backward combatability, the former must behave like the old Bourne Shell, which was insanely simple by today’s standards, to the point of being broken by modern expectations. But it’s a long-lived problem in software: we can’t fix anything because of all the things the fixes would break.

One of those simplicities is the lack of local variables. Recursion wasn’t possible in the old world. (Well, maybe with judicious use of eval. Is your hair gray yet?)

And looking back, Bourne shell only had the first form of function invocation. Korn shell introduced the second form with different behaviors to correct the defects.

So in Korn shell, if you create a local variable in a C-style function declaration (with the parenthesis), the shell ignores you and uses a global variable.

When you use function, you do get local variables. Real, true, lexical-scoped function scope locals: the functions you call can’t see them, nor fudge them up. And if they forget to declare their own locals, they pollute the global namespace instead of silently screwing up someone else’s locals.

In Bash, locals are created in the current context. But they are dynamically scoped, thus available to every function you call. Or any function they call in turn. And if one of those functions forgets to declare a local, it may very well unintentionally alter a variable belonging to another function. Pollution of the global namespace isn’t good, but it’s less hazardous than this nonsense.

Pipeline variable retention

When running a pipeline, Korn shell retains any values set at the end of the pipeline. So this construct works:

grep "^#" < source | sed -e ... | some other stuff | read variable

But in Bash, it doesn’t. The variable is read into a subshell which is disbanded when the pipeline completes. Now, the argument from Bash proponents is to just do this:

variable=$(grep "^#" < source | sed -e ... | some other stuff)

And that’s true enough when all you want is one line out of a file. But what if you want to stick a while loop on the end?

float total=0
grep "cleared" < checkbook | awk '{print $NF}' | 
while read value
do
        let "total += value"
done
print "The total is $total."

In Bash, this wouldn’t work because it’s using floating point math. But even if it was integer math, it wouldn’t work because it would total up the value, then the pipeline would close and the subshell would exit, taking total with it.

I realize there’s other ways to do this particular example. It wouldn’t be hard to run that through sed and make input you could feed to bc or dc to get a total. Or you could feed in a sentinel value to cause the loop to print its value, and capture that. But this is clearer and simpler. No hurdles.

And even if this is a hokey example, the concept stands. The pattern of read input, process it through a pipeline, and then amalgamate/ aggregate it at the end is not rare.

Note that since Bash 4.2, this is behavior is available via shopt -s lastpipe. However, this behavior only takes effect if job control is disabled, so it won’t work on the command line or in any functions you have loaded.

Interactive Use

For interactive use, the two shells seem very similar to me. Both have emacs and vi edit modes, quoting is identical (or nearly so) between the two, and many of the built-in variables are the same.

Bash has the bind command for keymapping; ksh has a KEYBD trap. I’m not sure which is better.

To the extent that Bash has emulated lots of Korn-shell pioneered behaviors, they’ve done well. Except the pipeline handling.

I have spotted at least one feature—case manipulation in parameter substitution—that’s been added to ksh based on its presence in Bash. So the copy-catting is going both ways.

And there are other unique Bash features, like the hook function for missing commands; ksh has no equivalent. (If I was implementing it, though, I’d make it a trap. It would make more sense than a magically named function.)

Imperfections

I do worry about the state of Korn shell. David Korn moved on, and though AT&T open sourced the code, will Korn shell be kept up?

The code is old, and it looks it; like something written in the 80s. How much edit fatigue does it have? I’ve delved enough to know there code is ugly.

Back before the COVID pandemic, one of the maintainers ransacked the code like a bull in a China shop. He tried to do some good work—ripping out a legacy, AT&T-proprietary memory manager and moving to standard malloc—but he also aggressively removed anything he didn’t care about, like Windows support. And while I don’t care about that, I’ll bet there’s someone else who does. I tried submitting one patch to fix something broken, but he ended up ripping out the feature entirely because he thought it resembled Bash to much. Another thing I wanted fixed he refused.

Only one of my patches did go through—it had to do with column-counting in output—but when I later saw some older versions there had been code that did something similar to mine. At some point it was ripped out. I don’t know with certainty who or why it was excised, but I have my suspicions.

The problem person’s “my may or the highway” approach drove off other developers and did damage to the community keeping Korn Shell alive.

In 2020, his “enhancements” metastasized into “ksh2020”, which was greeted much like COVID was given how buggy it was. Things as basic as filename globbing wasn’t reliable.

And then, from what I can tell, he either burned out or was demoralized by the negative response around his changes, or maybe some of both. Linux distros that had embraced ksh2020 reverted to other ksh93 forks, and after a bit, some folks started building a new team to take responsibility for ksh. They backed up a few years to a known-stable version, then reviewed patches since then, carefully reintegrating changes.

To those guys: Thank you for stepping up, for being sane, and for saving the best shell from the brink.

Replacement: zsh

ksh2020 drove me to find a replacement for interactive use; I settled on zsh. It isn’t a drop-in replacement because there are incompatibilities, but it’s somewhat close. There are various quirks moving to zsh:

• To void frequent “shell history corrupted” issues, don’t export HISTFILE; if you do and ksh gets invoked as a subshell, it appends to the history file and gives zsh a concussion.
• Disable ^D handling: bindkey -r '^D'
• Disable escape-slash handing: bindkey -r '^[/'
• Write a handler for ‘v’ in vi command mode to edit the command line in vi
• Update ^L handler, which should clear the screen and the scrollback buffer, thank you very much.

There were also scripts that required updates:

• I had to rewrite bits of my shell library (functions in my $FPATH) to accommodate variations, especially the lack of a -q option to whence.
• Script variables named $path need to be renamed, as zsh sets it to a space-separated version of $PATH, a crappy unexpected behavior imported from csh
• Similarly, $PROMPT is linked to $prompt.
• Array indexing starts at 1, not at 0.

zsh documentation is a hassle: there’s a lot of it, it’s split across several man pages. Losing the --help and --man options to the built-ins is inconvenient.

Some things are improved, like the column-counting: there are escape sequences that direct zsh to stop and resume counting. It means a little more work to set the prompt, but avoids an ugly, unreliable counting algorithm hard-coded in the shell.

For shell scripting, though, I still rely on Korn shell. Nothing else compares.

Conclusion

Admittedly, I’m more familiar with Korn shell, and that’s an influence. While I can use Bash just fine interactively, I find it frustrating to try to script with. After ksh, bash seems just broken—and in some cases, it’s probably because I’m not used to it. But in other cases, Bash requires jumping through hoops that aren’t necessary when working in Korn shell.

So, in programming features, ksh seems far ahead to me. I am glad to see it’s being maintained again, although it doesn’t have the popularity of Bash, which makes its future unclear.

Should Korn shell be abandoned in the future, zsh isn’t perfect but shows more promise as a sanctuary for Korn shell users than Bash.

For certain small projects and “glue”, shell is convenient and a good choice. But Korn shell makes it possible to tackle larger needs, and just because we can doesn’t mean we should: for some of those, Python might be a better choice.

Korn Shell’s self-documenting getopts

Insanely old Korn shells or other shells portending to be Korn Shell might not support the advanced getopts. If you are worried about this, you can detect availability with this construct:

if [[ $(getopts '[-][12:abc]' flag --abc; print -- 0$flag) == "012" ]]
then
    GETOPTS_STRING=$'
        ...new-style-getopts-document
        here...'
else
    GETOPTS_STRING="old-style-getopts-string here"
fi

The $'' quoting uses C-style quoting within, to keep you sane.

In your parsing loop, provide the selected string:

while getopts "$GETOPTS_STRING" flag
do
    case "$flag" in
        ... handle your options flags here
    esac
done

So what do you put in the new-style getopts?

A section title and a paragraph for it in the resulting man page:

[+TITLE?example - how to use the \bksh\b(1) getopts string]

If you need a new paragraph, leave out the title:

[+?This is another paragraph under the last title.]

To insert a definition list instead of a paragraph, wrap the entries in braces:

[+EXIT STATUS]
{
    [+0?The command succeeded.]
    [+1?The file was not mangled correctly.]
    [+2?The file could not be found.]
}

To define an option:

[s?Flag with short option only.]
[f:long-flag-name?Description of the flag.]
[p:option-with-argument-name?Description]:[argument-name]
[n:numeric-option-name?Explanation]#[argument-name]
[x:?This flag has no long-option name]
[12:flag-with-only-long-name?There is no short-form of this flag.]

The letter after the opening bracket is the short option. If a long option is used, it’s converted to the short form when provided to you. If you don’t want a short form, provide a 2 or more digit number to be used instead. Numeric argument specifications are enforced, but note 0xhex is allowed and decimal numbers (0.5) are an error.

For options with arguments, you can include a list of values. These are rendered in documentation, but they are not used for getopts processing. So don’t expect any restrictions they document to be applied by getopts.

[l:flag-with-list?Flag that takes only one of the defined values.]:[argument-name:type:attributes:=default-value]{
    [+one?This is the \afirst\a choice.]
    [+two?This is the \asecond\a choice.]
    [+three?This is the \athird\a choice.]
}

To include information about the script’s creators:

[-title?Name and contact information]

For example, [-author?Perette Barella] or [-license?This is free software released under the MIT license.]

Parameters go after a blank line following all your other text:

from-file to-file

If you have multiple forms, you can list them on separate lines:

from-file to-file
from-file ... to-directory

Lastly, there’s a handful of useful escape sequences: \b (backspace) toggles strong emphasis (bold). \a (bell) toggles emphasis (italic/underline). \v (vertical tab) toggles fixed-width display.

Putting it all together, here’s a demo program you can tinker with:

#!/bin/ksh
while getopts -a getopts_info $'
[+NAME?getopts info - document describing the ksh long-form getopts]
[+TITLE?This is a section and a paragraph in the resulting document.]
[+SOME OTHER SECTION?This is a spare section]
[+?Here is an example of a \adefinition list\a]
    {
        [+SOME] ?Text inside this \vdefinition list\v.]
        [+OTHER]?And text for another \vdefinition list\v entry.]
    }
[+?This is a continuation paragraph]
[s?Flag with short option only.]
[f:long-flag-name?Description of the flag.]
[p:option-with-argument-name?Description]:[argument-name]
[n:numeric-option-name?Explanation]#[argument-name]
[x:?This flag has no long-option name]
[12:flag-with-only-long-name?There is no short-form of this flag.]
[l:flag-with-list?Flag that takes only one of the defined values.]:[argument-name:type:attributes:=default-value]{
    [+one?This is the \afirst\a choice.]
    [+two?This is the \asecond\a choice.]
    [+three?This is the \athird\a choice.]
}
[-Author?Perette Barella]
[-Copyright?This document is public domain.]

argument argument2 ...
Other forms of this command
[Note that at this point, this is plain text]' option "$@"
do
        print -- "Argument: '$option' optarg:'$OPTARG'"
done