Recently, I wanted to implement a decorator that re-tries any function with an (optional) exponential backoff. Obviously, the settings for the backoff should be configurable, but users shouldn’t have to do that if they are fine with the defaults.

Essentially, I wanted both of the following applications to work:

@retrying
def may_fail_occasionally(*args, **kwargs):
     ...

@retrying(max_retries=10, initial_sleep=0.1)
def may_also_fail_occasionally(*args, **kwargs):
     ...

This looks innocent enough, right? I’m sure you’ve seen decorators with and without parameters before. Let’s start with the first case without parameters. An implementation may look like the following:

def retrying(f):
    max_retries = 5
    initial_sleep = 1
    multiplier = 2
    max_sleep = 15 * 60
    def wrapper(*args, **kwargs):
        exceptions = []
        sleep_time = initial_sleep
        for i in range(max_retries):
             try:
                 return (f(*args, **kwargs), exceptions)
                 sleep_time = min(max_sleep, sleep_time * multiplier)
                 sleep(sleep_time)
             except Exception as e:
                 exceptions.append(e)
    return wrapper

Type hints

So far so well, but let’s look at the types a bit more in detail. As you can see, I’ve decided to modify the return type of the wrapped function, adding the exceptions that we silently ignored as a second return value. It would be nice if we could indicate that in the function signature with proper type hints. For that, we have to think a bit more about what a decorator is. Essentially, we have a higher-oder function that takes a function f as argument and returns a modified version of that function:

P = ParamList("P")
R = TypeVar("R")

def retrying(f: Callable[P, R]) -> Callable[P, tuple[R, list[Exception]]]:
    ...

We use the handy TypeVar and ParamList classes to indicate that we keep the parameters and return type of the wrapped function intact, just adding something to the latter.

Adding parameters to the mix

Now that we have covered the case without paramers, we just need to move the local variables like initial_sleep into the function signature, and we’re good, right?

Unfortunately, that’s wrong. Consider our use case again:

@retrying
def may_fail_occasionally(*args, **kwargs):
     ...

@retrying(max_retries=10, initial_sleep=0.1)
def may_also_fail_occasionally(*args, **kwargs):
     ...

In the decorator application @retrying, the retrying refers to the function itself. We now want to replace that by something like @retrying(max_retries=10), which is a function call. So if we want to have parameters in our decorator, we need to build a function A that takes the parameters as an input and returns a function B that takes a function C as an argument and returns a modified version of C. 😵‍💫 With that, our return type changes to the following:

Callable[
    [Callable[P, R]],
    Callable[P, tuple[R, list[Exception]]]
]

Now we’re in Haskell-level type signature land. :laughing: Armed with that information, let’s re-implement retrying with parameters while keeping the types straight (buckle in!).

# Function A
# Returns the actual decorator.
def retrying(max_retries: int=5, initial_sleep: int=1, multiplier: int=2, max_sleep: int=15 * 60) -> Callable[
    [Callable[P, R]],
    Callable[P, tuple[R, list[Exception]]]
]:
    # Function B
    # The decorator, very similar to our initial version or retrying.
    def decorator(f: Callable[P, R]) -> Callable[P, tuple[R, list[Exception]]]:
        # Function C
        # The wrapper that replaces the function the decorator is applied to.
        def wrapper(*args, **kwargs) -> tuple[R, list[Exception]]:
            exceptions = []
            sleep_time = initial_sleep
            for i in range(max_retries):
                try:
                    return (f(*args, **kwargs), exceptions)
                    sleep_time = min(max_sleep, sleep_time * multiplier)
                    sleep(sleep_time)
                except Exception as e:
                    exceptions.append(e)
        return wrapper
    return decorator

Yeah, what the fuck, right? We only wanted to have a decorator with parameters and what we had to implement is essentially a decorator factory which returns a decorator that is itself a factory for creating wrapper functions. It looks crazy, but that’s what has to go on under the hood to make this work.

At this point, you might be tempted to just keep the factory monster we have right now, use @retrying() instead of @retrying to apply the decorator and call it a day. But we don’t do things halfway on this blog, so please fasten your seatbelts and brace yourself for a rough landing. We’re going to marry the two versions of the decorator into one. ✈️

Supporting both use without and with parameters

We have two type signatures for the retrying function now, which we need to support both in one function. Fortunately Python has the @overload decorator for that since version 3.5:

@overload
def retrying(f: Callable[P, R]) -> Callable[P, tuple[R, list[Exception]]]:
    ...

@overload
def retrying(max_retries: int=5, initial_sleep: int=1, multiplier: int=2, max_sleep: int=15 * 60) -> Callable[
    [Callable[P, R]],
    Callable[P, tuple[R, list[Exception]]]
]:
    ...

Now we have some type hints that don’t immediately make the eyes of our users bleed, but how do we get those types together for the actual implementation? Is there a way? Fortunately, yes. Since f is provided as a positional argument, and we want our optional parameters to be supplied as keyword arguments, we can even have a super clean definition using the * in the argument, which enforces that all parameters after it must be supplied as keyword arguments.

def retrying(
    f: Callable[P, R] | None = None,
    *,  # everything after this _must_ be given as keyword argument
    max_retries: int=5,
    initial_sleep: int=1,
    multiplier: int=2,
    max_sleep: int=15 * 60
) -> Callable[
    [Callable[P, R]],
    Callable[P, tuple[R, list[Exception]]]
] | Callable[
    P,
    tuple[R, list[Exception]]
]:
    def decorator(f: Callable[P, R]) -> Callable[P, tuple[R, list[Exception]]]:
        def wrapper(*args, **kwargs) -> tuple[R, list[Exception]]:
            exceptions = []
            sleep_time = initial_sleep
            for i in range(max_retries):
                try:
                    return (f(*args, **kwargs), exceptions)
                    sleep_time = min(max_sleep, sleep_time * multiplier)
                    sleep(sleep_time)
                except Exception as e:
                    exceptions.append(e)
    if f is None:
        # no function argument given => we are called with parameters
        return decorator
    # function argument is there => we are called without parameters
    # => we need to return the wrapper directly instead of the decorator
    return decorator(f)

There we are. Safe and sound in decorator land. Please clap for the captain. ✈️ :laughing:

If we call retrying(f), as python will do internally when we use the decorator @retrying without parameters, we essentially get the same solution as we had in the first version of the implementation: We define a decorator function, but immediately apply that function, essentially stripping away the extra layer again. If we call retrying(initial_sleep=10) instead, which is what happens when we use the decorator @retrying(initial_sleep=10), we use the extra wrapping layer to provide the decorator function as a closure that already encapsulates the value we gave for the extra argument intial_sleep.

Final remarks

I have mixed feelings about types like Callable[[Callable[P, R]], Callable[P, tuple[R, list[Exception]]]] | Callable[P,tuple[R, list[Exception]]. Let’s be honest: Nobody will understand this type when they come across it in the code or in an error message of the type checker. Even in functional programming languages, this is the point where I would say “Just let me put auto as the type and figure it out yourself!”. However, spending some time to think about the actual type that is returned by the decorator was what ultimately helped me to understand why I had to implement it that way and that, yes, there just is no easier way. So I think I’m still in favor of typing everything you can, but I do understand if you don’t want to touch types like that with a 10-foot pole. 🙈

Why COBOL?

COBOL is like a dragon. Every programmer has heard of it, and that it is formidable. We’ve heard stories about people fighting the beast, but very few of us have actually seen a live COBOL program out in the wild. If that isn’t reason enough to be intrigued, maybe its historical sgnificance brought you here, wondering about how programming was like in the days of Grace Hopper - inventor of the first compiler, rear admiral, and DPMA man of the year 1969, who democratized coding, was a passionate teacher and genuinely kept her staff happy as a manager. Whatever your angle is, I assume you have a reason for being here, so I won’t jabber any longer about why COBOL is still interesting.

Why this guide?

A lot of people have written amazing and detailed COBOL guides and here I am, having solved one exercise of Advent of Code 2024 in COBOL. Why should you listen to me introducing a language I barely know? Well, the COBOL guides I found value precision and proper contextualization over speed. This is the opposite: If you want to spend no less than 30 minutes but be able to write a rudimentary COBOL program after that, I’m here for you.

Hello world

Enough of the introduction. I promised speed, so here is “Hello World” in COBOL:

000001 IDENTIFICATION DIVISION.                                         HELLO
000002 PROGRAM-ID. HelloWorld.                                          HELLO
000003 PROCEDURE DIVISION.                                              HELLO
000004     DISPLAY "Hello World!".                                      HELLO

If you want to run this, you can install GnuCOBOL and run the following:

cobc -xjO hello.cob

COBOL file format

The above hello world program is written in fixed format, which was used for COBOL programs prior to 2002. There is also a free format now, which removes the restrictions about which columns source code has to start (and end) in. However, for the purpose of this guide, we go old-school because that’s more fun.

In fixed format, the actual code resides in columns 8-11 (Area A) and 12-72 (Area B). This seems crazy from today’s coding standards, but was actually pretty reasonable back in the day. You see, each of these lines of code would be written on a punchcard—yes, just one line of code per punchcard. Why punchcards? Because they were way cheaper than any form of electronic storage back in the day. So imagine you’ve finally written your 300 LOC program. You’re on the way to the computer room to test it, but on the way you bump into a colleague, also holding a stack of 300 punchcards. The cards go flying all over the place and now what? Don’t fret! COBOL has you covered (if you followed best practices, that is). You can just stack them in any order, put them into a sorting machine, and instruct it to first sort by column 73-80 (Program Name Area) and then by column 1-6 (Sequence Number Area). Voilà, now both of your programs are in the right order again, and can be separated from each other! Today, we don’t need this anymore, and we can just leave these areas empty, but I’m sure this feature saved countless work hours.

The only things we need to remember now are these:

• Column 7 can be used to designate a line as a comment (*) or code that’s only used for debugging (d). There are a few other symbols that appear here, but they are not relevant for this guide.
• Actual code starts in Area A (column 8), or sometimes Area B (column 12).
• Code must end on column 72. Everyting beyond that will be ignored.

Side note: If you’re weird like me and this makes you want to dive deeper into how punchcards worked, feel free to have some fun with this interactive editor for IBM 5081 punch cards:

COBOL syntax

You might have already noticed that the hello world program above only uses characters that we would also use when writing text in English. This is very much by design. Grace Hopper herself heavily pushed for using English over symbols because she wanted to democratize programming and make it more accessible.

It’s important to keep that in mind, because unlike most modern programming languages, COBOL doesn’t aim to have a minimal number of syntactic elements and keywords (if, for, …). Instead, it provides a plethora of keywords, which sometimes consist of more than one word and can have alternatives, to make code sound as natural as possible to a non-technical person that just writes their first program.

Time for another example: Let’s calculate how a user-defined amount of money grows with 1% interest over 10 years:

       IDENTIFICATION DIVISION.
       PROGRAM-ID. Interest.
       DATA DIVISION.
       WORKING-STORAGE SECTION.
       01 balance PICTURE s9(7)V99.
       PROCEDURE DIVISION.
       calculate-interest.
           DISPLAY "Please enter starting balance:"
           ACCEPT balance.
           PERFORM 10 TIMES
               MULTIPLY balance BY 1.01 GIVING balance
               DISPLAY "New balance: " balance
           END-PERFORM.

Hierarchy

You probably already noticed in the hello world example that COBOL code has a document-like hierarchy. There are DIVISIONs, and SECTIONs that seem to structure the code. In fact, there are the following hierarchical levels:

• Divisions
• Sections
• Paragraphs
• Sentences
• Statements

Divisions, sections, and paragraphs have pre-defined names and structures, with the exception of sections and paragraphs in the PROCEDURE DIVISION, which can be named freely by the programmer. Sentences are groups of statements (which again have pre-defined structures) that are terminated with a dot. One effect of this hierarchy is that the definition of variables (which are called data items in COBOL) in the WORKING-STORAGE SECTION of the DATA DIVISON is separated from the code in the PROCEDURE DIVISION.

Data items

The WORKING-STORAGE SECTION looks bizarre from the standpoint of modern programming languages:

      * level             signed
      * |   name          |digit
      * |   |             || 7 times
      * |   |             || | decimal point
      * |   |             || | |
       01 balance PICTURE s9(7)V99.

The number in the beginning is called the level. A level of 01 designates an elementary data item. A higher number would be used for an aggregate data item that belongs to a group item of the next lower level above it. This allows to define and reference nested data types of arbitrary complexity. We’ll go into a bit more detail about that later.

The name of the data item is pretty self-explanatory, but the PICTURE seems weird again. COBOL actually never forces you to think in binary or any low-level data types for that matter. Instead, you define your data types by their “picture”, i.e. by the format how you would write them in a text file. The syntax for this picture part almost looks like a form of proto-regex:

• s denotes a sign (either the character + or -).
• 9 denotes a single decimal digit.
• (7) repeats the character in front of it 7 times.
• V denotes where the decimal point is placed.

With that, the picture s9(7)V99 stands for a signed 7-figure number with two decimal places - a pretty reasonable data item for the account balance of most people.

Procedures

Our example program has a single procedure defined by the named paragraph calculate-interest. Naming the paragraph is only necessary if we aim to call it as a procedure later in the code. By default, the first paragraph (named or unnamed) of the PROCEDURE DIVISION will be called as the main procedure.

Inside a procedure, you can create sentences out of an arbitrary number of statements. The statements that we use here are:

• DISPLAY some-value some-other-value ... to display an arbitrary number of values as a concatenated string on the terminal.
• ACCEPT data-item to read user input and store it in a data item. Note how the PICTURE definition of that item gives you input validation for free.
• PERFORM 10 TIMES [...] END PERFORM to repeat the statements in [...] in a loop.
• MULTIPLY x BY y GIVING z to multiply the value of x by y and store the result in z.

As already mentioned, COBOL uses many more keywords and statements than most modern programming languages, so learning to program in COBOL consists largely of searching for the right keywords and associated statements. There are also the classical named functions that we are more used to, but those will be covered in the next section.

Solving a non-trivial problem in COBOL

We could end our quick-start instructions here, but I think the step from a toy example to one that solves an actual non-trivial task still brings a lot of insights with a good cost-benefit ratio. The following COBOL program solves the first part of day 2 from Advent of Code 2024.

For this exercise, we have to read a text file with a list of numbers on each line, separated by spaces and count how many of those are “safe”. A list is safe if:

• The numbers are sorted in ascending or descending order.
• And the absolute difference between adjacent numbers is between 1 and 3 (inclusive).

       IDENTIFICATION DIVISION.
       PROGRAM-ID. AoC-2024-Day2.
       ENVIRONMENT DIVISION.
       INPUT-OUTPUT SECTION.
       FILE-CONTROL.
           SELECT input-file
           ASSIGN TO "input"
           ORGANIZATION IS LINE SEQUENTIAL.
       DATA DIVISION.
       FILE SECTION.
       FD input-file.
       01 input-file-line PICTURE x(1024).
       WORKING-STORAGE SECTION.
       01 FILLER PICTURE a.
           88 at-eof VALUE 'Y' FALSE 'N'.
       01 line-cursor PICTURE 999.
       01 previous-line-cursor PICTURE 999.
       01 number-pair.
           02 previous PICTURE 999.
           02 current PICTURE 999.
       01 line-length PICTURE 999.
       01 FILLER PICTURE a.
           88 all-increasing VALUE 'Y' FALSE 'N'.
       01 FILLER PICTURE a.
           88 all-decreasing VALUE 'Y' FALSE 'N'.
       01 safe-count PICTURE 999.
       01 difference-safe PICTURE a.
           88 difference-is-safe VALUE 'Y' FALSE 'N'.
       01 difference PICTURE 999.
       PROCEDURE DIVISION.
       parse-file.
           OPEN INPUT input-file.
           PERFORM UNTIL at-eof
               READ input-file INTO input-file-line
               AT END
                   SET at-eof TO TRUE
               NOT AT END
                   PERFORM process-line
               END-READ
           END-PERFORM.
           DISPLAY "Total count of safe sequences: " safe-count.
           CLOSE input-file.
           STOP RUN.
       process-line.
           MOVE FUNCTION LENGTH(FUNCTION TRIM(input-file-line))
           TO line-length.
           MOVE 1 TO line-cursor.
           SET all-increasing TO TRUE.
           SET all-decreasing TO TRUE.
           SET difference-is-safe TO TRUE.
           PERFORM UNTIL line-cursor GREATER line-length
               MOVE line-cursor TO previous-line-cursor
               PERFORM read-next-number
               PERFORM check-if-increasing
               PERFORM check-if-decreasing
               PERFORM check-difference
           END-PERFORM.
           IF (all-increasing OR all-decreasing) AND difference-is-safe
               ADD 1 to safe-count.
       read-next-number.
           MOVE current TO previous.
           UNSTRING input-file-line
           DELIMITED BY " " INTO current
           WITH POINTER line-cursor.
       check-if-increasing.
           IF previous-line-cursor IS GREATER THAN 1
           AND previous IS GREATER THAN current
               SET all-increasing TO FALSE.
       check-if-decreasing.
           IF previous-line-cursor IS GREATER THAN 1
           AND previous IS LESS THAN current
               SET all-decreasing TO FALSE.
       check-difference.
           COMPUTE difference EQUAL FUNCTION ABS(previous - current).
           IF previous-line-cursor IS GREATER THAN 1
           AND (
               difference IS LESS THAN 1
               OR difference IS GREATER THAN 3
           )
               SET difference-is-safe TO FALSE.

Reading an input file

The first challenge that we haven’t tackled yet is file access. The definition of what file is accessed is handleed in the INPUT-OUTPUT SECTION of the ENVIRONMENT DIVISION:

       FILE-CONTROL.
           SELECT input-file
           ASSIGN TO "input"
           ORGANIZATION IS LINE SEQUENTIAL.

All of this is pretty straightforward: We give our file the name input-file, provide the file name on the local file system ("input"), and tell COBOL that it should read the file line by line. If you’re wondering “how else would I read a file?”, consider that COBOL mostly operates on fixed-length records. This has the benefit of allowing random access to files, skipping to the nth record in the file in O(1) instead of having to read O(n) variable-width lines. In that regard, our file structure is the worst case scenario from an efficiency perspective.

There is one more part of the definition in the FILE SECTION of the DATA DIVISION that is a little less straightforward:

       FD input-file.
       01 input-file-line PICTURE x(1024).       

Here, we first define what type of tile input-file is. The type FD designates a normal file used for reading and writing data. Another alternative would be SD, which designates a file that is used as a working memory area for COBOL’s sorting and merging functionality. The next line, defines a data item that represents one record from the file. As our file has ORGANIZATION IS LINE SEQUENTIAL, we effectively define what data type should be used for reading a line. As we want to be agnostic about line size and content, we just pick 1024 alphanumerical characters. The downside of this is obviously that if a line is less than 1024 characters long, we will still fill 1024 bytes of memory padded with spaces.

The code for actually reading a line from the file has again an interesting property:

       READ input-file INTO input-file-line
       AT END
           SET at-eof TO TRUE
       NOT AT END
           PERFORM process-line
       END-READ

As you can see, the READ statement has an option to execute other statements depending on whether there was something to read or we already reached the end of the file. Here, we use this to set the condition at-eof, which controls the outer loop of the program. But more on conditions in the next section.

Defining a condition name

Let’s look a bit closer at the definition of at-eof:

       01 FILLER PICTURE a.
           88 at-eof VALUE 'Y' FALSE 'N'.

The first thing that’s new is the FILLER in place of the name of the top-level data item. This is just a way of saying that we will never access that data item itself, so it doesn’t need a name. We still need to give it a picture, which is just one alphabetic character.

Now it gets interesting: The second line defines a subitem with level 88. Normally, you can only use levels 01 through 49, but there are a few special levels in COBOL. Level 88 is used for defining so-called “condition names”, which can be used as booleans. If set to TRUE, a condition name will assume the value given after VALUE (or the first possible value in case a range of values is specified there). If set to FALSE, it assumes the value given after FALSE. So essentially, we have an alphabetic data item that can either be Y or N, but we can use at-eof directly in place of a condition in an IF or UNTIL, and we can assign it to TRUE or FALSE with the SET statement.

Defining a group item

We’ve seen the special level 88 for condition names, but we haven’t seen a true group item yet:

       01 number-pair.
           02 previous PICTURE 999.
           02 current PICTURE 999.

Here, number-pair is a group item and previous and current are its subitems. The level 02 for the subitems is arbitrary. The only rule is that the level has to be larger than the parent item and both subitems have to have the same level. With this, we have a number-pair consiting of two numbers. If we wanted to swap pairs around as a whole, we could access number-pair directly, but for the purpose of this task we actually don’t need that. I just created the group item for learning purposes. You can access the subitems just by their name, but in case you would have subitems with the same name in differently named group items, you could also reference them explicitly as previous OF number-pair, for example.

Calling a procedure

In the code for reading a line from our file, you might have already noticed the PERFORM process-line statement. This is how you call a procedure in COBOL. As long as it’s in the same file, there is no need to pass any arguments, as all data is shared between procedures. This makes it harder to track which procedure is responsible for changing which data items, but it allows you to divide your code into procedures at virtually zero cost: Just add one line in Area A that introduces a new paragraph and thus gives the procedure a name.

In this program, we make heavy use of this feature to make the code more readable. For example, consider the loop over the numbers in a line:

           PERFORM UNTIL line-cursor GREATER line-length
               MOVE line-cursor TO previous-line-cursor
               PERFORM read-next-number
               PERFORM check-if-increasing
               PERFORM check-if-decreasing
               PERFORM check-difference
           END-PERFORM.

If all the procedures were written out here instead of calling them, the code would become quite convoluted. There is the downside of losing track where the data item line-cursor in the loop condition is changed, but to some degree such problems can be alleviated by choosing speaking names for the procedures.

As you might have noticed, procedures are not the only way of breaking down code into smaller elements. COBOL has functions, which are called similar to how we are used to in modern languages. For example, FUNCTION TRIM(input-file-line) calls the function TRIM and passes input-file-line as an argument. You can, of course, define your own functions in COBOL, but this is beyond the scope of this quick start guide.

Overview of program logic

With all the syntactic peculiarities out of the way, let’s briefly discuss what the program actually does:

• The main loop PERFORM UNTIL at-eof just ensures that process-line is called for each line in the input file.
• process-line reads the numbers found in the line one by one into the number-pair data item to check for each pair whether one of the safety conditions is violated.
• read-next-number advances the line-cursor by reading the next number in the line and storing it in current while saving the previous content of current in previous to do the comparions later on.
  • I admit that it took me a while to find out that UNSTRING is the statement I needed here.
• check-if-increasing and check-if-decreasing update the data items all-increasing and all-decreasing if they find a pair of numbers that violates the condition that the numbers in the line are sorted in ascending or descending order respectively.
• check-difference calculates the absolute difference between the numbers in the pair and sets difference-is-safe to false if the difference is too small or too large.
• Based on the values of all-increasing, all-decreasing, and difference-is-safe, the counter safe-count is incremented if the line violated none of the safety conditions.
• At the very end, we use DISPLAY to print the final value of safe-count and thus solve the Advent of Code exercise.

Learnings and outlook

So, what have we learned? By this point, you are no more of a COBOL expert than I am. You understood a very basic program that can be solved with a handful of lines in Python or Ruby, and we haven’t even touched on any advanced concepts such as copybooks, tables, or sorting. However, I hope that now that the very basic questions are out of the way, you are able to dig into the GnuCOBOL Programmer’s Guide or your COBOL manual of choice to find the statements you need for your program. And I hope that this will be much quicker for you than if you had to start reading that fine but long document from the beginning.

Alternatively (or maybe additionally), I hope you just had a bit of fun with a quirky old language and maybe learned to appreciate where it got its quirks from. Maybe COBOL doesn’t look so much like a dragon now but more like a bear. Sure, it can still kill you, and you might not want to get too close to it, but you can see how it has its place in nature. If you just let it sleep and don’t disturb it too much, it’ll probably be okay.

Think about a very simple AI that classifies emails into benign “ham” or unwanted “spam”, very much like the one we built in a previous article: How do we measure how good our AI performs?

Until now, we just looked at a few examples manually and determined whether we felt that the outcome was correct. This works if our question is just “Can the AI distinguish between a typical ‘Nigerian prince’ spam mail and a genuine mail to a friend from Nigeria?”. But what if the question becomes “Can the AI detect all kinds of typical spam mails that are in my inbox?” or if we also want to check for the kind of spam our friends and colleagues are getting? The more general we want our AI to be, the more examples we will need to check. Wouldn’t it be great if we could just automate this? After all, all we need for that a list of emails for which we already know whether they are “spam” or “not spam”. This is the same kind of labeled data that we needed for building our AI in the first place. To distinguish both datasets, we will call the data we use for testing the test set.

Having this data lets us rephrase the question to “What percentage of these examples does the AI get right?”. This measure is called accuracy, and it can be calculated simply like this:

1. Classify all example emails in the test set with the spam detection AI.
2. Count how often the AI outputs the right choice of “spam” or “not spam”.
3. Divide that number by the total number of examples in the test set.

The possible outcomes are between 0 (the AI does not get any of the examples right → 0% accuracy) and 1 (the AI gets every example right → 100% accuracy). In general, an AI with 80% accuracy is better than an AI with 70% accuracy, for example.

A cautionary note about data separation

In the previous example, I just assumed that we do not use the same data for building the AI and for testing it. Let’s have a brief look at why we do this and what will go wrong if we fail to separate between training and test data:

Remember the instructions for our spam detection AI:

1. For all labeled emails in the database, calculate the number of matching words between that email and the query.
2. Find the database entry with the maximum number of matching words.
3. Output the label attached to this database entry.

What happens if we try to classify an email that was already in the database? Well, the maximum possible number of matching words between one text and another is of course all the words in the text. So if we ask the AI for the best label for an entry that it has already stored in its database, it will always find the exact copy of that entry and output the label attached to that copy. In other words: If we use our training data to calculate accuracy, we will end up with 100% accuracy. Always. By definition.

The same is true for many other algorithms that you can use to build an AI. This is why AI researchers and developers always stash away a part of their data as test data that the AI is never allowed to see until the time comes to evaluate its performance.

Different kinds of errors

Let’s say we have two separate spam classification AIs with 70% accuracy. Can there still be differences between them?

Well, let’s have a look at the different kinds of errors our AI can make by going through all possibilities:

• True positive: If it classifies a mail as spam that actually was a spam mail, that’s good. No error here.
• False positive: If it classifies a mail as spam that actually was not spam, that’s one error to make. The AI was too eager in finding spam.
• False negative: If it classifies a mail as not spam that actually was spam, that is also an error, but in the opposite direction. It was too lazy and did not catch all the spam mails.
• True negative: If it classifies a mail as not spam that actually was not spam, that’s fine again.

So, we end up with two kinds of errors: “eagerness errors” and “laziness errors”. As you may have noticed, one of the two is a little more dangerous: Being too lazy just means we still have to delete a few spam mails ourselves. Being too eager might mean that a mail from our Nigerian friend or maybe from the company in Nigeria where we applied for a job lands in the spam folder, and we might never notice it.

This leads to two new questions about the AI’s performance:

1. What percentage of the examples that end up in the spam folder actually are spam.
2. What percentage of the spam mails that I receive in my inbox will be sent to the spam folder.

The first measure is called precision and the second is called recall. As you might already have guessed, they can be calculated as follows:

Precision

1. Count the number of emails that get a “spam” label from the AI and actually are spam.
2. Count the number of emails that get a “spam” label from the AI, regardless of whether they were spam or not.
3. Divide the number from step 1 by the number of step 2.

Recall

1. Count the number of emails that get a “spam” label from the AI and actually are spam.
2. Count the number of emails that are spam, regardless of whether the AI classifies them as such.
3. Divide the number from step 1 by the number of step 2.

To put it in simple terms, higher precision means less eagerness errors and higher recall means less laziness errors.

Precision and recall are, however, not the only measures that help to distinguish between those two kinds of errors. Imagine a medical setting where you test for a disease like COVID-19: On the one hand, you want to know how good the test is at detecting sick people as sick. But on the other hand, you also want to know how good it is at detecting healthy people as healthy.

For the first part, you can just use recall because that is exactly what recall measures: The percentage of all sick people that will test positive. In this setting, this is called sensitivity, however, because it measures how sensitive the test is to finding the disease.

For the second part, we use a different measure that we call specificity.

Specificity

1. Count the number of healthy people who are tested negative.
2. Count the number of healthy people in the whole test group, regardless of whether they were tested negative.
3. Divide the number from step 1 by the number of step 2.

Again, you can think of a test that is more specific of having less eagerness errors and a test that is more sensitive of having less laziness errors. It is just a slightly different definition and terminology that helps to make the right decisions in a medical setting.

It is important to note here that both for precision and recall and for sensitivity and specificity, only knowing one of these two measures will tell you nothing about the actual quality of the AI or COVID-19 test. This is because you can easily cheat them by just classifying every sample as spam/sick (100% recall, 100% sensitivity) or classifying every as no spam/healthy (100% specificity). Precision is a little harder to trick, since we would divide by zero if we classify every sample as spam. However, that just means we need to find that one spam mail that we are really sure about, and we still can get 100% precision.

Dealing with more than two categories

Until now, we only looked at the spam example, where there was only a yes/no decision to make by our AI. How about the image classification task where we tried to recognize handwritten digits from 0 to 9? Here, we have ten possible classification outcomes and ten possible true labels.

We can still calculate the overall accuracy, which tells us how close we are to our goal in a single number. We can also calculate precision and recall for each digit, which will tell us which digit we recognize too often or too seldom. However, there is a new question that becomes interesting: “Which digit is confused with which?”. If our AI sometimes confuses a 1 with a 7 or an 8 with a 0, that might be acceptable, but if it starts confusing a 4 and a 1 really often, something weird is going on.

To diagnose these issues, we can just count:

1. How often does the AI classify a 0 as a 0?
2. How often does the AI classify a 0 as a 1?
3. How often does the AI classify a 0 as a 2?
4. …
5. How often does the AI classify a 1 as a 0?
6. How often does the AI classify a 1 as a 1?
7. …

And so on. This gives us 100 numbers for all the ten times ten possible outcomes. To visualize this, you can build what is called a confusion matrix that puts the label predicted by the AI (p:) on the columns and the true class label (t:) on the rows of a table. The result looks like this.

To look for mistakes, we search for the largest numbers outside the diagonal, since the diagonal shows us the samples that were classified correctly. For this particular classifier, we can see that the most common mistake is to predict a 3 (p:3) for images that actually showed a 5 (t:5). If we roughly calculate the sum over the rows, we can also see that the dataset used for the test contained fewer examples for the digit 5 than for the digit 3. If the same was true for the training data, this might already indicate why the AI makes exactly this kind of mistake. It could be that it just hasn’t seen enough examples of the digit 3. When it is in doubt, it errs on the side of the class label that is more likely to occur in the data.

As you can see, confusion matrices may be confusing (heh) to look at at first, but they can tell you a lot about the performance of an AI that is supposed to classify data into multiple options.

Final remarks

Let’s sum up what we have learned:

• There are automatic measures that can tell you how good an AI is.
• Some of these measures (accuracy) are just one number, others are number pairs (recall/precision, sensitivity/specificity). Never trust anyone, who just boasts a high score in one of the numbers belonging to a pair!
• When you want to look at what kind of errors an AI used for classification makes in detail, you can build a confusion matrix.
• It’s important not to test an AI on the data it has already seen when it was trained, since that makes it easy for the AI to cheat.

Even if you won’t remember any more details from this post than those bullet points, you are already in a powerful position to judge AI systems. You know what numbers to look out for, be suspicious if they are not or only partly reported, and can compare different AIs with each other based on those numbers.

A hash table allows storing and retrieving values, which are identified by associated keys. They are the data structure behind the types that we know as map, dict, or hash. The main idea of a hash table is to calculate an integer value for each key that can be used to determine the index where the associated value should be stored in an array. This magic integer is called a hash and the array structure in which the values are stored is sometimes called a table—hence the name hash table. Since the number of elements we want to store is usually smaller than the maximum integer value, the actual array index is determined by using the modulo operation to obtain index = hash(key) % size.

The main difficulty in finding a good hash() function is that the range of available hashes is limited by the number of possible (positive) integer values while the theoretical number of possible key objects is often infinite or at least much larger. Take the example of using strings as keys: A string s can have aribitrary length, but there are only 2³² possible different results of hash(s). This implies that some Strings will need to have the exact same hash value and will therefore be put into the same index in the hash table, even though they are completely different. Even if two different strings have a different hash value, they can still end up with the same index through the modulo operation. For example, lets say the string "foo" has a hash of 5 and the string "bar" has a hash of 9. If both strings are put into a hash table of size 4, they both are assigned the index 1 since 5 % 4 = 1 and 9 % 4 = 1.

There are different possibilities to solve these collisions, the most common of which is called separate chaining. It works by actually storing a linked list in each of the entries of the array and chaining the values that are assigned to the same index in this list. Each time you retrieve a value from the hash table you then have to do a sequential search in this “bucket” of values. Regardless of which mechanism is used to avoid collisions, a high number of collisions in a hash table will mean more searching and therefore a slower access to the individual values. A good hash function is therefore one that minimizes collisions by spreading out hash values as uniformly as possible across the integer value range, avoiding any clusters of similar keys that end up with the same hash value.

The question

To use a data type as a key in a hash table, we need a hash function for that particular data type. Finding a good hash function is hard, and usually the best bet is to resort to using predefined functions in the standard library. However, it is never a good idea to blindly trust an algorithm without having a general understanding how it works and why we should use it in place of other alternatives. At the very least, this knowledge will help to anticipate and debug issues that may occur in our application.

In Java, two methods are therefore of particular interest:

• String.hashCode() implements the hashing function for the data type String, which is both often used as key for hash tables and is very flexible since it can have an arbitrary length. Ignoring particularities like internal caching of hash codes and the Java 9 feature to compact strings to store them in latin-1 encoding if possible, we arrive at StringUTF16.hashCode(byte[]):

    public static int hashCode(byte[] value) {
        int h = 0;
        int length = value.length >> 1; // two bytes per character
        for (int i = 0; i < length; i++) {
            h = 31 * h + getChar(value, i);
        }
        return h;
    }
  

• Objects.hash(Object ...) is a helper method for Java developers who want to overwrite Object.hashCode() for their custom data types. Instead of creating a hashing function from scratch, developers can simply pass all components of the object that are relevant for equality as argument to the helper method. It then delegates its work to Arrays.hashCode(Object[]):

    public static int hashCode(Object a[]) {
        if (a == null)
            return 0;
  
        int result = 1;
  
        for (Object element : a)
            result = 31 * result + (element == null ? 0 : element.hashCode());
  
        return result;
    }
  

As you can see, both methods essentially use the same idea of multiplying the parts of the objects successively with the prime number 31. So our questions for today are: Why the successive multiplication? Is it important that 31 is a prime number? And why the magic number 31 in particular?

Non-answers

I noticed that searching for this question online yields a lot of answers which are either false, incomplete, or assume too much mathematical background knowledge. Examples include…

• … suggesting that the multiplication is important since it scales up values, which is moot since scaling a number that is distributed over a small range of possible values does not increase that number of possible values;
• … using phrases like “because of the nature of maths” in the explanation;
• … saying that actually larger primes are better, because they are less likely to produce collisions with small numbers, which ignores the fact that the modulo operation can also introduce collisions even if the hash codes of all keys in the hash table are unique;
• … suggesting that we use a prime number as the bucket size of the hash table, which is cumbersome since hash tables would need a list of prime numbers to choose their bucket size from;
• … capitulating before the question, stating that “The value 31 was chosen because it is an odd prime.” (all primes other than two are odd) and “The advantage of using a prime is less clear, but it is traditional.” as is true for Effective Java 2nd Edition, which sadly is one of the most cited references on the topic;
• … explainig the choice of 31 with the fact that multiplication with 31 can be expressed as a bit shift and a subtraction, which has some truth in it but still does not explain why 31 and not another Mersenne prime like 127 or 8191, which have the same property;
• … just stating that it is good to keep the number of prime factors low to avoid collisions;
• … stating that 31 is chosen because it is close to the number of letters in the latin alphabet or because it is close to the 32 bits of the int data type, which both is utter nonsense;
• or claiming that the product of a prime with any other number has the best chance of being unique, whatever that means.

All these non-answers serve to show that this is an area where computer scientists tend to be out of their comfort zone, because it involves number theory, i.e. hardcore math. I do not think that anyone in the above examples is to blame for not knowing a better answer. After all, they are programmers and computer scientists and not mathematicians, but I do question the lack of curiosity that leads to such shallow answers being given and accepted. You cannot learn anything if you do not acknowledge the gaps in your knowledge first. Put this way, the answer from Effective Java might be the most honest one.

There are also good answers that explain parts of the problem really well:

• The StackOverflow user advait shows an example how a prime modulus yields a more uniform distribution than a non-prime modulus.
• Steve Jessop and ILMTitan clarify that the mathematical property that we want does not require either the multiplication factor or the number of buckets to be prime, but just that both are coprime, meaning that they have orthogonal prime factorizations, i.e. no common divisor other than 1.
• JohnZaj gives a reference to an experiment testing which multiplication factors result in the least collisions for a list of 50,000 English words, in which 31 is one of the winners.
• David Ongaro and Flow both demonstrate that they have mastered necromany to level 100 by conjuring up an old JDK bug report in which Joshua Bloch explains his reasoning for choosing the prime 31 in the implementation of String.hashCode(), which turns out to be part empirical testing and part literature research.
• coolblog published a Chinese blog article in which they perform an experiment using 230,000 english words, also finding that 31 as a multiplier gives good results.

However, I did not find any single source that pieces these bits of information together to a fully comprehensive answer.

Ask a mathematician

As a teacher at the THM, I wanted to do better for my students and therefore did the only thing that came to my mind after the internet and books failed me: ask a mathematician. The following explanation is therefore fully owed to Prof. Dr. Bettina Just, who sacrificed her coffee break to change my view of prime numbers in hashing forever by drawing a few numbers on a whiteboard from the top of her head. Any errors and obscurities in this explanation of course remain entirely my own.

Let’s start by what Prof. Just wrote on the whiteboard on that fateful day:

The thing that you should note is what the multiplication with 5 and 7 does to the last digit of x, i.e. the result of x % 10. For the factor 5, which is not coprime to the modulus 10, the last digit can only be either 0 or 5 meaning that we lost some diversity. For 7, which is coprime to 10, we get all the values we had before from 1 to 9—just in a different order.

It turns out that this is a general mathematical property that is true whenever the multiplicative factor and the modulus are coprime. I will not pretend to understand the deeper mathematical reason behind this or attempt to provide a proof here. Instead I will just leave you with this StackExchange question, and the reference to Euler’s theorem, which I believe implies this unnamed theorem, and which is interestingly also central for the RSA algorithm used for encrypted communication. Putting aside the mathematical origins, let’s try to state our key theorem in a more understandable way:

Theorem: If a and n are coprime, then multiplying all possible (nonzero) values of x % n = 1, 2, 3, …., n-1 with a and again applying the modulus n to the result yields a permutation of these values.

Or in even less mathematical terms:

Multiplying x with a number a that shares no divisors with n “shuffles” the possible outcomes of x % n.

Going back to our hash function, this means that multiplying a key value with a number coprime to the number of buckets will never make the distribution of values into buckets less uniform.

Ok, this means that we do no harm with the multiplication, but what good does it actually do? To explore this question, we must first take a look at the algorithm in which this multiplication takes place.

Benefits of polynomial rolling hashes

The algorithm that we see both in String.hashCode() and Objects.hash(Object...) is called a polynomial rolling hash. It is used to calculate hashes of composite objects in a way that each component has an equally important influence on the hash value. To understand how it does this and why this is imporant, we first look at the following prototype:

public static void hash(int[] data, int p) {
    hash = 1
    for(int x: data) {
        hash = hash * p + x;
    }
}

At first glance it is odd that the whole hash is successively multiplied with p, but this becomes easier to see if we write down the result for data.length = 3:

hash(data) = ((1 * p + data[0]) * p + data[1]) * p + data[2]
           = data[2] + p * data[1] + p * p data[0] + p * p * p

So what hash(int[], int) is doing is actually just building a polynomial in which the elements of data are the coefficients and p is the base. We could write this in a more straightforward way, but exponentiation is more expensive than multiplication and since hash tables are often used for performance reasons and we have to build a hash each time we retrieve a value from the table, we want to save a little time here.

So why are polynomials a good idea? To understand this, let us first look at the most naive solution we could think of. Essentially, we want to reduce an array of numbers to a single number. So why not just take the sum of the elements?

The main issue here is that addition is a commutative operation. a + b is the same as b + a, which means that the order of the elements does not matter for the end result. If we think about hashing strings, all anagrams will collide with each other, since, for example, the string “one” will have the same hash value as the string “neo”. Additionally, if the range of possible values for the individual elements is smaller than the range of integers, a sum does not grow quickly enough to yield a good distribution of hash values across the integer range. This is again the case for strings. For ASCII strings of length 4, we have 128 possible characters with a maximum sum of 4 · 127 = 508, but we have 128⁴ = 268,435,456 possible four-character strings, which means an expected collision rate of at least 1 - 508 / 268,435,456 = 99.9998%.

This is where the polynomial rolling hash comes in. By using polynomials, we both avoid the commutativity of addition, because each summand is multiplied with a different value before summation, and even when the data only consists of very small values, the exponentiation quickly spreads these values over the integer range.

Now for the choice of p, remember our theorem:

Multiplying x with a number a that shares no divisors with n “shuffles” the possible outcomes of x % n.

Here, p is the factor a and the amount of buckets in the hash table is n. To choose a good value for p, we have to know how the scaling of the hash table is implemented. If we don’t, the safest bet is a prime number, since it will not share any divisors other than itself with any other number.

Since we are in Java, we can look up the implementation of HashMap<K,V>, which always uses powers of two as the table length. The reason for this is that instead of using the modulo operator we can just calculate (n - 1) & h when h is the hash value. Since n = 2^e, the binary representation of n - 1 are zeros followed by a series of e ones and the bitwise and operator gives us just the last e bits of the hash.

This also means that the only restriction we have to follow for p is that it cannot be even. All odd numbers will be coprime to 2^e for all e.

Remaining sources of collisions

By now we have ensured that neither the multiplicative factor nor the table size may introduce any particular pattern of collisions. Assuming a uniform distribution of the content of the data array across the possible number of options (be it 128 ASCII chars, 2¹⁶ UTF-16 chars, or 2³² integers), this would give us a perfect hash function with minimal collisions.

The only remaining point of concern is our data itself. Remember that multiplication with a number coprime to the number of buckets does not make the spread across buckets worse, but it also cannot do anything if the spread was already bad before the multiplication. For example, if we would start with hash = 0 instead of hash = 1, we would be able to easily produce collisions if all elements of data were divisible by two for all or at least many of the keys that we want to store. By starting with hash = 1 we add the summand p^data.length to the calculation, which cannot be divisible by two.

At this point we have ruled out the most easily predictable patterns of our data that may cause collisions, but in real data there might be all kinds of unpredictable patterns. Unless we run tests with a particular p and a particular data set that is representative of real world applications, we cannot be sure that the hash function really works well for our use case.

Just to give you an example of what kind of patterns we might be talking about, the letter ‘e’ is far more likely to occur in an english text than the letter ‘k’. For texts in other languages the probability distribution might be similar or entirely different. And what about other kinds of data such as file system paths, points in a 2D coordinate system, or dates? All of these are not uniformely distributed by any means, so some choices for p might work well and others quite poorly.

History of 31

This brings us to the magic number 31. As already hinted at, you can read up the whole history of how this number was proposed by Joschua Bloch in an old JDK bug report. In short, there are two main arguments: One is the previous use of 31 by other programmers, and the other is a test performed by Bloch himself.

The references for the choice of the number 31 go back to Kerninghan and Ritchies “The C programming language”, but when Bloch called them and asked them about its origin, neither of the authors could remember it.

Since Bloch was aware that 31 was by far not the only candidate, he performed tests evaluating collision probabilities with the following kind of data:

• All of the words and phrases with entries in Merriam-Webster’s 2nd Int’l Unabridged Dictionary (311,141 strings, avg length 10 chars).
• All of the strings in /bin/, /usr/bin/, /usr/lib/, /usr/ucb/ and /usr/openwin/bin/* (66,304 strings, avg length 21 characters).
• A list of URLs gathered by a web-crawler that ran for several hours last night (28,372 strings, avg length 49 characters).

The composite number 33 performed a little better than the prime 31, but since Bloch was unsure about the mathematical foundations, he was more comfortable choosing the candidate that was indeed a prime number.

Let’s put aside for a minute that this testing procedure leads to a preferential treatment of English speaking countries over others and Linux users over Windows users, and just acknowledge that choosing any single number that would work well for all application scenarios across all existing and future Java applications is an extremely hard problem. It may very well be that choosing a number that “works well enough” for some popular applications might be the best that Bloch could do in this situation.

However, while this is true for String.hashCode() it does seem that the 31 was copied over to Object.hash() without much thought or testing. After all, we should assume different patterns to occur in arbitrary composite objects than in strings, which are heavily shaped by natural and formal languages.

Performing a bit of necromancy myself, I could indeed find other old JDK bug reports, which report bad performance of Arrays.hashCode(long[]) for small arrays and for arrays only containing the values 0 and -1. This was acknowledged by the Java developers, but no change was implemented since the internal algorithm is already fully specified in the documentation of the Java API and thus some applications might rely on this specific implementation.

Bonus: Evaluating different primes

Other people have already tried to come up with “better” primes than 31. In general I find the quest for the best prime number for a general hash function moot, because—as explained earlier—different application scenarios will have different patterns leading to a different performance of individual prime numbers.

However, there are a few questions left that still tickled me:

• Do larger prime numbers really perform better than smaller ones?
• Do Mersenne primes perform better in speed and/or collision avoidance than other primes?
• Will the prime 31 perform as well for strings in other languages as it does for English?

So in a small myth busting experiment, I put together a few very small test cases which I think are fairly realistic keys for a hash table:

• The 1000 most common English words
• The 1000 most common German words
• The 1000 most common Russian words
• 100000 random pixels in a 1024x768 pixel image consisting of two integers for the x- and y-coordinate
• the same pixels, but converted to strings of the form "(x, y)"
• 10000 random dates between 1983 and 2022 consisting of three integers for year, month, and day
• the same dates, but converted to strings in ISO 8601 format ("YYYY-MM-DD").

For the pixel example I also assessed the performance of a custom hash function that just adds the coordinates after shifting the y coordinate left by 10 bits, effectively just enumerating the pixels line by line.

To make the comparison as realistic as possible, I calculated the number of buckets as the lowest power of two such that the stored values fill only 75% of the available space at maximum. In Java this is assured by the “load factor” in the HashMap implementation. I also did not use the polynomial rolling hash directly, but applied a second hash function that XORs the hash with itself shifted by 16 bits to the right as in the protected method HashMap.hash(Object). Unlike Bloch, I did not calculate the average number of items per bucket in the hash table but the collision percentage, because this allows comparison across different tests with different table sizes.

I also performed a very crude performance test by running the whole simulation once to ensure that the JVM is warmed up and then taking time measures with System.nanoTime(), since I did not want to set up a JMH project for such a small test.

Let’s have a first look at the prime values below 50 to see how the prime 31 performs in comparison to its direct neighbors:

The dotted vertical lines show prime numbers and the solid vertical lines indicate the Mersenne primes 3, 7, and 31. The colored lines represent the different test scenarios and the faded area in the background is the result of the crude performance test.

We can see a few things here that were expected and also a few that were surprising. For one, the prime 31 seems to be neither a particularly bad nor a particularly good candidate for our test cases. If we had to determine a “winner” in this range, it would probably be the prime 29 due to its exceptional performance for the date test. However, if you care more about Russian than about English or German, you might also be interested in the prime 41.

Both the raw versions of the date and points tests show symptoms of having a too narrow value range when the multiplicative factor is small. Most surprisingly, converting dates and points to strings yields much better hashing performance in both cases. Even the custom idea of just enumerating pixels performs well but still worse than the string variant. Memo to myself: Stop scolding students for implementing hashCode() as return this.toString().hashCode()!

Another result that is surprising at first glance is that we actually don’t see more collisions for even factors as we would have predicted. This can be explained by the second hash function applied by HashMap, which unfortunately makes our results less predictable. I have run the test without it and the result is indeed much more jagged with worse performance for even factors.

Finally, there is no noticeable performance gain in terms of computation speed when using Mersenne primes. This can be explained by the fact that the Java compiler does not automatically perform the required optimization step to transform the multiplication into a shift and subtract operation. Looking a little closer into this, there is a JDK bug report about changing the code to improve the performance of String.hashCode(), but it turns out that the performance gain on modern machines is a whooping 0.01%. Myth busted!

The only myth that remains is that larger primes would perform significantly better. For this, let’s redo this calculation around the other Mersenne primes within the integer range: 127, 8191, 131071, 524287, and 2147483647.

And for good measure one that is quite large but not near any power of two, which I chose by calling new Random(31).nextInt(1 << 15, 1 << 25). (The 31 is only in there for funsies, because I needed some number to fix the seed of the random number generator.)

Apparently, Mersenne primes are not only not better, but actually worse than other primes. It seems like there is some kind of “gravity well” around powers of two, which drags the adjacent Mersenne primes into the performance abyss along with them. I suspect that this has something to do with the fact that all values in this area will have a quite large number of zeros in their binary representation. The effect we see is, however, extremely dependent on the test case. For the raw date and point tests it is much more pronounced than for words, and for the string version of points and dates it does not occur at all.

In this regard, Mersenne primes might have been a poor choice to look at the effect of the size of the multiplicative factor on the collision performance, but the last test of a pseudorandomly generated number shows that there are fluctuations everywhere and that the only effect of size that we can reliably determine is that there are some use cases where extremely small and extremely large multiplicative factors perform equally badly. Therefore: “Large numbers are better” myth (mostly) busted!

That being said, I have no good explanation for many of the observed phenomena in this test. Maybe I chose weird test cases, maybe I have some bugs in my simulation? If you want, you can have a look at the source code. I would be grateful to hear if you found any errors, but currently I think the most likely explanation is the one I gave earlier: Finding a multiplicative factor that performs well for all test cases is extremely hard and predicting which patterns will occur in the keys of hash tables is next to impossible. After all, I bet you could think of at least a dozen other test cases that would be interesting to investigate right now.

What do we take away from this? I would suggest four things:

• It’s always more complicated than you think at first glance.
• In Java, it seems to be OK if not outright beneficial to implement hash functions based on String representations of objects.
• Sometimes, rolling a die is the best way to avoid running into patterns that degenerate algorithmic performance.
• If you really want to know if your hash function performs well for your application scenario, test it with data that is representative of this scenario.

Recap

In this series we have previously classified emails in “spam” and “ham” and recognized hand-drawn digits from images. In both cases, we used the nearest neighbor algorithm, which simply finds the “most similar” example from a large database and copies the label (“ham”/”spam” or the digits from 0 to 9) attached to that element as classification output.

One interesting observation was that the only difference between the spam detection AI and the image recognition AI was the data that we used. For emails, we counted the number of matching words to measure similarity and for images we counted the number of matching pixels. In this post, I want to give another example of the broad applicability of this approach by looking at stock market prediction.

A new task: Stock market prediction

Today I will teach you how to get rich quick—just kidding, please do not use the techniques presented here for actual stock trading. That being said, it is interesting to think about the problem how one could predict the price of a stock on the following day or in the following week given the price history. After all, very real money is being made with artificial intelligence in this area.

Our data might look as follows:

In this case we see the price of a single share of a company’s stock¹ change from week to week. I have chosen weeks instead of days here because then we do not have to consider that markets are closed on the weekend. In principle, however, we could also formulate this problem for daily or monthly prices.

To earn money with stock trading, you have to buy shares of a stock when they have a low value and then sell them when their value has risen. The core question we have to ask is therefore: “What will the price of the stock be next week?” This prediction of a future trend is the inherent risk in stock trading and the attack point that we want to tackle with our AI.

Data series as database entries

The first difference that you might notice between this task and the previous ones is that our dataset is no longer composed of individual items that are independent of each other. Instead, it is now a series of interconnected data points. This poses the question what kind of “neighbors” we might consider in our nearest neighbor approach.

In principle, we want to predict an unknown, future share value based on a history of known, past share values. The easiest way to transform the data series given above into a list of individual database entries that we can use for the nearest neighbor approach is therefore to only consider, for example, the history of the last three weeks. We can then split the data series into subseries of four successive weeks, where the values at the first three weeks are the database entry and the value at the fourth week is the “label” that we want to predict:

In this table I have used overlapping individual histories (week 1–3, week 2–4, and week 3-5) to obtain as many database entries as possible.

Neighbors of data series

Now that we know how our database entries look, the question remains how to determine the similarity between two of these entries in order to find our beloved nearest neighbors. In our previous examples, we always just counted the number of matching items (words, pixels) between the two database entries. Since our database entries now consist of arbitrary numbers, that does not work anymore. Instead, we have to calculate how close these numbers are to each other. The easiest way to do so is to take the absolute difference between them, that is, subtract the one from the other and if the result is negative, just flip the sign to obtain a positive number. For example, the individual differences between the numbers in the first two database entries would look like this:

The smaller these differences are, the higher the similarity between the two entries. However, each difference still consists of three numbers. Is a difference of (4, 6, 1) larger or smaller than a difference of (3, 5, 2)? To answer this problem we need to condense the difference down to a single number, for example by simply taking the sum of the individual differences. For our example, we can then say that 4 + 6 + 1 = 11 is indeed larger than 3 + 5 + 2 = 10.

In a real application, one would use more complex formulas such as the Euclidean distance, but for our purposes this simple sum of differences is perfectly fine.

Predicting numbers instead of classes

When discussing our database format, I have put the term “label” in quotes, since we actually do not want to predict distinct labels (such as “ham”, “spam”, “digit 1”, or “digit 2”) but an arbitrary number. After a week the share value could drop to zero, or it could double or quadruple. While some answers are of course more probable than others, the theoretical number of possibilities is endless.

How can we deal with this infinite number of possible labels? The answer is surprisingly simple: We do the same as before. Once we have found the nearest neighbor to a given series of share values, we simply copy the share value at the fourth week as our prediction.

This has the disadvantage that we will never produce any numbers that are not already in the database, but this can be compensated by having a really large database with a good coverage of different cases, which we would need anyway to make good predictions. Alternatively, we could also store trends (by how much did the value rise/fall between weeks) instead of absolute values, which would also allow us to predict an infinitely rising or falling trend, but for now we just want a simple solution that works.

Putting together the algorithm

With this, we have all the information that we need to build an AI using the nearest neighbor approach to predict a share value based on a three-week history:

Algorithm: Simple share value predictor

Inputs:

• Database: list of three-week share value histories with separate fourth-week value
• Query: a three-week share value history

Output:

• Prediction: the most likely share value in the next week

Steps:

1. For all histories in the database, calculate the sum of absolute differences between that history and the query.
2. Find the database entry with the smallest sum of differences.
3. Output the fourth-week value attached to this database entry.

Again, we have successfully converted the human task “Find the share price that this stock will have next week.” into a set of instructions that can be automatically performed by a computer. While you will definitely not get rich quick with this particular algorithm, real trading algorithms do follow the same principle to get as much information out of existing historical data in order to predict share values in the future—and they are as susceptible to unforeseen patterns that do not appear in their database.

If you imagine a much more sophisticated algorithm than this one running on years and years of data from thousands of stocks, maybe also analyzing the behavior of other traders and then making sub-second decisions, it is easy to see why high-frequency trading is so effective at making the rich even richer and why people who do not have access to these tools tend to lose out in comparison.

To paint a less bleak picture, we have unlocked a whole new class of problems that we can now tackle with our AI approach, namely all problems that require us to predict a numeric value that depends on a list of other numeric values. This general task is called regression analysis, and it can also, for example, be used to predict crop yield in agriculture or the effect of medication on blood pressure based on patient-specific data.

Beyond one week

You might have already asked yourself what we would do if we wanted to predict the share price for a duration of more than one week into the future. For this, we basically have two possible approaches:

1. We can predict more than one week by storing more values in the “label” part of our database. Instead of just the fourth-week value we could also store the value for the fifth, sixt week, and so on. This approach increases the problem that we only can predict trends that we have already seen as we now have even more possible outputs and need even more data in the database.
2. We can simply run our algorithm multiple times. If we predicted the value 59 for the history [50, 54, 60], we can then see what our AI yields for the new query [54, 60, 59]. This does not require to change the algorithm or to acquire more data, but it also means that our errors will accumulate, and our predictions will get increasingly unreliable the longer the desired output.

Next steps

We have now seen that the nearest neighbor algorithm is very flexible and in principle behaves similar to state-of-the-art machine learning approaches across a large variety of tasks. To dive deeper into the core principles of artificial intelligence, we just need to take one additional step moving from one to an arbitrary number k of neighbors to obtain the k-nearest neighbor algorithm. This will be established in the next post in this series.

Building unit tests for GUIs is not entirely straightforward. Usually, you want to separate your application logic from your GUI as much as possible precisely because this makes the code easier to test and debug. However, what if the graphical display is the main part of the application? This is true for FCanvas, a project that I started in my first years of teaching at the THM in 2013 and now recently uploaded to GitHub. FCanvas is a library that allows programming novices to draw on a canvas and create simple animations, so the core functionality is closely tied to what can be seen on the canvas.

While brushing the dust off the project I also wanted to add a few unit tests to ensure that none of my refactoring attempts would introduce any bugs or visual glitches. And since I love a good CI/CD pipeline, I also wanted to add respective GitHub actions workflows. I already had a hunch that those two things in combination might be a problem, and sure enough my unit test, which ran fine on my local machine, threw the following exception in GitHub actions:

java.lang.ExceptionInInitializerError
    at de.thm.mni.oop.fcanvas.FCanvasTest.testRectangle(FCanvasTest.java:20)

    Caused by:
    java.awt.HeadlessException:
    No X11 DISPLAY variable was set,
    but this program performed an operation which requires it.
        at java.desktop/java.awt.GraphicsEnvironment.checkHeadless(GraphicsEnvironment.java:166)
        at java.desktop/java.awt.Window.<init>(Window.java:553)
        at java.desktop/java.awt.Frame.<init>(Frame.java:428)
        at java.desktop/java.awt.Frame.<init>(Frame.java:393)
        at java.desktop/javax.swing.JFrame.<init>(JFrame.java:180)
        at de.thm.mni.oop.fcanvas.FCanvasGUI.<init>(FCanvasGUI.java:18)
        at de.thm.mni.oop.fcanvas.FCanvas.<clinit>(FCanvas.java:85)
        ... 1 more

The culprit was a call to new FCanvasGUI();, because FCanvasGUI inherits from JFrame, whose constructor can throw the java.awt.HeadlessException, which we see here. The term “headless” means that no display device is attached to the machine that runs the Java code, which in consequence means that there is no way to actually display the JFrame we have just created.

Solution 1: Avoid code that can throw java.awt.HeadlessException

Searching for the term “headless” in the Oracle documentation reveals that this affects all graphical components except for Canvas, Panel, and Image. My simple test for drawing a rectangle on a JPanel could therefore also be performed in headless mode if I skipped the creation of the JFrame in which the panel is displayed. However, I also plan to add fancier tests down the road, which will involve simulating user input through java.awt.Robot, so this was not an option for me.

Solution 2: Create a dummy display

Instead, I searched for a way to fix the issue on the side of the operating system. Surely the Linux community has found some way to run X11-applications on a headless server, right? Right! There is the X virtual framebuffer (Xvfb), which holds an image buffer in memory that behaves like an X server display but does not require an actual display device.

Xvfb comes with a simple tool xvfb-run, which runs a single command with such an Xvfb server and closes the server right after the command exits. The solution for my GitHub workflow was therefore simply to install the xvfb package using apt install xvfb and then exchanging ./gradlew build with xvfb-run ./gradlew build. Works like a charm. 🎉

Notes

While searching for these solutions I came across a few misleading tips, which I want to discuss here in order to save you the trouble should you run into the same issues.

Running a gradle task through xvfb-run vs running xvfb-run from within gradle

Searching for the terms “xvfb” and “gradle” leads to an Ask Ubuntu answer that states the following:

You likely don’t want to run a gradle task through Xvfb, but rather execute something within an X Windows Virtual Frame Buffer FROM a gradle task.

The context is using xvfb-run for a gradle run configuration, not for gradle test or gradle build. In that context the suggestion is somewhat sensible: Gradle itself does not need the display, only the sub-process that it starts does. Nonetheless, I thought that there might be some deeper meaning to why simply using xvfb-run directly with Gradle would be a bad idea (the Ask Ubuntu post unfortunately does not explain this). Maybe the display is not properly passed through to sub-processes? Maybe xvfb-run only works with programs that actually ask for a display or there is some other incompatibility with the gradle executable?

Nope! None of this is true. The only downside of using xvfb-run for the whole Gradle task is that the framebuffer will exist for a slightly longer duration than it is needed. In contrast to gradle run, there also simply is no option in Gradle to alter the call used to start the unit tests beyond setting system properties and other JVM args. So in the sense of simplicity over premature optimization you can quote me on this: “You likely do want to run a Gradle task through Xvfb.” 😉

Don’t blindly use examples from man pages

Due to the aforementioned Ask Ubuntu answer, I first tried to avoid xvfb-run and instead set up xvfb manually. The first example in the man page for xvfb (or at least the version that is online on die.net) is the following:

Xvfb :1 -screen 0 1600x1200x32

The problem with that is the x32, which sets the color depth to 32 bit. While there are pixel formats with 32 bit, they technically only use 24 bits for the color information and the last 8 bits for transparency. Therefore, Xvfb crashes with the following error message:

Fatal server error:
Couldn't add screen 0

If, for some reason, you want to set up Xvfb manually without xvfb-run, you can do the following:

export DISPLAY=:1
Xvfb :1 -screen 0 1600x1200x24 &
# your call using the Xvfb display goes here
killall Xvfb

Which would look like this in a GitHub actions workflow:

- name: Setup xvfb for screen 0
  run: Xvfb :1 -screen 0 1600x1200x24 &
- run: # your program call goes here
  env:
    DISPLAY: :1
- name: Tear down xvfb
  run: killall Xvfb

By the way, there is a bug report for the misleading line in the man page, which was opened in 2008 and lead to a change in the man page that was implemented in 2018. Such is the way of low-priority issues, I guess.

I wanted to write a simple Tk application that would assist my students in running unit tests for their exercises in an introductory Python course. I choose Tk, because it can be accessed from the Python standard library with the tkinter module, which would allow my students to run the app without installing additional Python packages. Inside the app, I needed a Text widget to display the standard output and standard error streams of the unit test process. The students should be able to copy text from this widget into a search engine to learn about the meaning of Python error messages, but they should not be able to alter the content of the widget accidentally by typing something into the widget. Like most GUI toolkits, Tk allows to “disable” widgets so that they do not react to user input.

However, from there on it got a little more complicated. First, the insert() method used for setting the content of the text widget programmatically also becomes disabled when the widget state is set to "disabled". This can be fixed by wrapping calls to insert() with an activation and deactivation call as follows:

text_widget.config(state="normal")
text_widget.delete("1.0", "end")
text_widget.insert("1.0", res)
text_widget.config(state="disabled")

Once this was fixed, however, my tutors noticed that copying text with the usual shortcut CTRL-C did not work in Ubuntu. The curious thing: It did work on my Manjaro machine. What could be the reason of this platform dependent behavior?

Things I learned along the way

Before I tell you about the solution, I want to share a few discoveries with you that I made along the way: My first instinct was check whether removing the lines containing .config(state="disabled") would remove the problem. According to my tutor it did not, so I searched for the problem online learning that…

• … there might be an issue related to language settings,
• … there is an issue involving ScrolledText,
• … there are a lot of solutions for building a read-only Text widget, but all of them somehow feel wrong.

It seemed like disabling all keybindings manually by binding them to a function that returns 'break' might be an easy solution, but this seemed a little too hacky for me, so I went on to do some research on the interna of Tk.

The first interesting thing that I found was a mention of a file called tk.tcl, which apparently is responsible for setting up the key bindings for copy-paste commands. I did find the file at /usr/lib/tk8.6/tk.tcl in Manjaro and /usr/share/tcltk/tk8.6/tk.tcl in Ubuntu and by looking at the contents I could confirm that…

• … the only variable that controls how the “virtual event” <<Copy>> is set up, is tk windowingsystem, which can be x11, win32, or aqua,
• … if you put puts "<insert frustrated slur here>" somewhere into the file, you are indeed insulted by every Tk app you start afterwards, including git gui,

To find an easier way to check the value of tk windowingsystem than meddling with the tk.tcl file, I then searched for a way to execute Tcl/Tk commands. It turns out that there are two shells tclsh and wish, the former being a REPL for plain Tcl and the latter already including the Tk library. In wish, you can then simply type tk windowingsystem and as suspected it returned x11 for both Linux systems, ruling out the tk.tcl file as possible culprit.

Since the next possible offender was ScrolledText, I looked at the source code for the tkinter.scrolledtext module and found out that the relevant class is only a little more than 20 lines of code, which in passing alleviated my concerns how hard it would be to write a version of ScrolledText that used themed ttk widgets.

The solution

After a lot of back and forth changing search terms, I finally arrived at a StackOverflow post that mentioned the following with regard to using the disabled state for a read-only widget.

On some platforms, you also need to add a binding on <1> to give the focus to the widget, otherwise the highlighting for copy doesn’t appear:

text_widget.bind("<1>", lambda event: text_widget.focus_set())

Having set up a VM with a vanilla Ubuntu installation in the meantime, this led me to re-investigate the issue regarding the disabled state again myself.

Sure enough, the copy-paste problem did vanish when I disabled the two relevant lines that set the disabled state. So there is another lesson in passing here: If you can only communicate via text, make absolutely sure that what you think is being tested is actually what is being tested. Back to the problem, I re-enabled the code lines that set the disabled state and used the suggestion from Bryan Oakley from StackOverflow instead and it did also work.

So the problem was solved, but I was still not satisfied, because I wanted to know the cause of this difference. Who is responsible here? Is it Tk, is it Linux? Are there really such fundamental differences between Ubuntu and Manjaro? My journey took me to the release notes of Tcl/Tk and I found the following bullet point in the notes for version 8.6.11:

• Allow for select/copy from disabled text widget on all platforms

So it was just a bug in Tk after all and the suggested fix to set the focus when the mouse button is clicked is the correct solution. Whew! Sure enough, I checked the package versions and my Manjaro used version 8.6.11.1-1 of the tk package while Ubuntu used version 8.6.9+1. As a last bit of information, if you want to find out your Tk version from within a Tk app, you can query root.tk.call("info", "patchlevel") where root is your root-level Tk object.

I hope you enjoyed this little murder mystery as much as I did, even if it was just the copy-paste function that was murdered and at the end of the day it turned out that it only did not find the way home, since it was lacking focus. 😆

In the last post we constructed our very first AI algorithm that was able to mimic the human decision whether a newly received email is unwanted “spam” or benign “ham”. We also established that the technical term “algorithm” actually only means a step-by-step description of how to translate input values into output values. For our spam detection algorithm we needed to find the database entry that has the maximum number of matching words between itself and the email that needs to be decided upon. Then we could just look up whether that database email was considered “spam” or “ham” and copy the same label for our output.

This algorithm was effective, but it only detected spam, so how could we possibly use it as a proxy for understanding all the different and complex AI systems out there? Well, in fact the nearest neighbor algorithm that we used as the basis for our spam detection AI, can be applied to a wide range of problems. In this post, I want to show you how it can be used for a problem that has nothing at all to do with spam detection, and that is image recognition.

A new task: Image recognition

Imagine you work in mail distribution center and have to find a way to sort letters by the handwritten zip codes on letter envelopes. This can and has for a long time been done by manual labor, but I imagine this is not anybody’s dream job. Instead, it would be nice if we could just pass the mail through a scanner, and then use a computer program to automatically extract the zip code from the scanner image. There are a whole lot of messy details in this process: The program needs to align the image, find the area on the envelope where the address is written, find the zip code within the address and separate the code into individual digits that it then needs to recognize. To keep things simple, we only focus on the last step: Recognizing a handwritten digit from a black and white scanner image.

Some examples may look like this (which are converted samples from an actual AI dataset called MNIST by Corinna Cortes, Christopher JC Burges, and Yann LeCun):

The “intelligent” decision we are looking for in this situation is to take one of these images and recognize which number is shown on it. For most of the above examples, this would be simple for a human, but there are also some instances that are a little more tricky:

• could be both a 1 or a 7.
• is somewhere between a 2 and a 3.
• might be confused for a 9 instead of an 8.

How computers see images

In order to automate this digit recognition task, we need to know how images are represented in a computer. You probably already know this, but to start from the beginning: Digital images are made up of small squares with uniform color and size, which are called pixels (short for “picture element”). To build up a whole image, these pixels are arranged in a regular grid of rows and columns. The images above all have 28 rows and 28 columns of pixels, which each can be either fully black or fully white. Let’s scale up one of these images to ten times its size to actually see the pixels.

Now we are still talking about images in terms of colors and geometric terms instead of text or numbers that a computer can read and manipulate. One simple way of storing the above image in a machine-readable format is to create a text file and write a zero for each white pixel and a one for each black pixel, arranged in rows and columns and separated by spaces:

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0
0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0
0 0 0 0 1 1 1 1 1 1 1 1 1 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0
0 0 0 0 1 1 1 1 1 1 1 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0
0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0
0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0
0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

If you squint your eyes a little, you can even still see the picture, but now it is just a bunch of zeros and ones. This image format is called a Portable BitMap (PBM), and it can actually be read by open-source image editing tools like GIMP. If you want to, you can try it out by downloading the above image in PBM format.

The image formats that you are used to, like JPEG, PNG, or GIF, are much more complicated, but this is just because they are designed to save storage space. Whenever images are displayed on the screen or opened in an image editor, it is in some bitmap-like format.

Comparing images

Now that we know how represent images in a format that can be understood by a computer program, we can start thinking about how to adjust our nearest neighbor algorithm to cope with 28x28 pixel images of digits. First, let’s start by revisiting our spam detection algorithm:

Algorithm: Simple spam detector

Inputs:

• Database: list of labeled emails
• Query: unlabeled email that should be classified

Output:

• Label: the most fitting label for the query (either “spam” or “ham”)

Steps:

1. For all labeled emails in the database, calculate the number of matching words between that email and the query.
2. Find the database entry with the maximum number of matching words.
3. Output the label attached to this database entry.

There are a few things that we have to change to make this work with images. First, we have different inputs and outputs now. The Database is now a list of labeled images, where the label is the digit shown in the image. The Query is an unlabeled image that has to be recognized. And finally, the output is again a Label, but instead of only two options we now have to choose one of ten different labels that correspond to the ten digits from zero to nine.

Moving to the steps of the algorithm, the “number of matching words” is, of course, meaningless for images. However, we can match something else to find the “nearest neighbor” of an image: the pixels. We can count the number of matching pixels by moving through both pictures at the same time, starting at the top left, and moving in “reading” order from left to right and from top to bottom. Every time the pixels at corresponding positions of the two images match, we increase the count of matching pixels by one. When we reach the bottom right of the image, we will have the total number of matching pixels between both images.

To see that this works, let’s look at a small example with tiny 3x4 pixel images:

[Database entry 1:]

Label: 1

0 1 0
0 1 0
0 1 0
0 1 0


[Database entry 2:]

Label: 7

1 1 1
0 0 1
0 0 1
0 0 1


[Query image:]

1 1 1
0 0 1
0 1 0
1 0 0

We have two images in the database of the digits one and seven, both consisting of straight lines. The query image that we want to know about is another seven, but this time with a slanted lower line. Our AI now compares the query image to both database images to find the following:

Matches for entry 1:
0 1 0     1 1 1      x ✓ x
0 1 0  =  0 0 1  ->  ✓ x x
0 1 0     0 1 0      ✓ ✓ ✓
0 1 0     1 0 0      x x ✓

  Total number of matching pixels: 6


Matches for entry 2:
1 1 1     1 1 1      ✓ ✓ ✓
0 0 1  =  0 0 1  ->  ✓ ✓ ✓
0 0 1     0 1 0      ✓ x x
0 0 1     1 0 0      x ✓ x

  Total number of matching pixels: 8

Since database entry 2 has more matching pixels than database entry 1, our AI chooses to copy the label of entry 2 and correctly decides that the query image is a seven and not a one.

Now that we know that our idea works, the only thing that is left to do is to turn it into a new formal algorithm description:

Algorithm: Simple digit recognizer

Inputs:

• Database: list of labeled images
• Query: unlabeled image that should be recognized

Output:

• Label: the most fitting label for the query (0-9)

Steps:

1. For all labeled images in the database, calculate the number of matching pixels between that image and the query.
2. Find the database entry with the maximum number of matching pixels.
3. Output the label attached to this database entry.

As you can see, we only hat to change a few words. The resulting algorithm is still a nearest neighbor algorithm, it only needed to be adapted to find “neighbors” of images instead of emails.

Again, we have turned the human task “recognize the handwritten number in this image” into a set of instructions for a machine that is now able to mimic this human decision-making process and thus can be considered an AI. This also means that all implications that we have drawn for the spam detection algorithm are also valid for image recognition tasks. In particular, the “intelligence” of the algorithm still only comes from having good samples in the database that represent the set of possible inputs well. While the recognition of the slanted seven worked out fine in our simplified example, a database featuring larger images should definitely include both straight and slanted sevens, and possibly also sevens with an additional horizontal bar in the center. If it does not have enough of these examples, some sevens could be falsely recognized as ones or fours. In our mail distribution center setting, this could result in a delay in package delivery, as the package would initially end up in the wrong district, but real-world examples of image recognition systems based on too simple datasets can have far worse outcomes: If you have dark skin, you have a tenfold chance to be misidentified by state-of-the-art facial recognition software. This can lead to you not being able to pass through biometric passport validation at the airport or even a false identification as a criminal. A major source for this problem is likely the fact that the databases used to train these algorithms can have as much as 80% light-skinned persons. Of course these systems use more complicated algorithms than our nearest neighbor approach, but as in cooking, it is all about the ingredients. If you give a bunch of moldy vegetables to a Michelin star chef and to a home-cook, the chef might be able to produce a more tolerable looking dish than the home-cook, but you probably would not want to eat either one. This problem is endearingly called garbage in, garbage out. We will go into detail on this issue in one of the next posts of the series, but first we will take a little more time to explore the variety of possible application areas of the nearest neighbor algorithm.

Edits and Acknowledgements

Since it was first published, this article has undergone the following changes due to reader feedback:

• Added more detailed explanation of “garbage in, garbage out” (thanks to Annina)

There are already countless answers to the question “What is artificial intelligence?”—most of them written by people that are far more experienced than I am. So why write the umpteenth blog post about it? The answer is that I tried to find a good introductory text about the topic for my students online and was disappointed with the search results. For my taste, they were either too shallow to produce real understanding, leaving the reader with a large list of half-explained terms, or too technical to be approachable, focusing only on the tools and applications that the writer is most enthusiastic about.

This post is the first in a series with which I hope to bridge that gap by using (and to some extent abusing) the example of a very simple artificial intelligence (AI). At the end of this first post, I want you to truly and fully understand how this AI works whether you are a computer scientist, a hairdresser, a physician or a baker. With the whole series, I want to enable you to use the simple method behind it as a proxy for understanding other AI systems, approaches, and general questions and issues.

What is AI?

I could start this section by introducing a lot of fancy words that describe different kinds of artificial intelligence (AI) to carefully contrast the term with other related terms such as “machine learning” and with more philosophical approaches. Instead, I will just assume that for the layperson AI means “the stuff that companies like Google do that detects faces, plays chess, drives cars, recommends movies on Netflix, and powers Alexa and Siri”.

All those highly successful and widely used AI systems are programs that make predictions or guesses about some real-world problem based on a set of existing examples. These predictions and guesses should be “intelligent” in the sense that they emulate the choices a skilled human being would make.

A typical AI problem

To make this a little more concrete, think about your mail inbox and that nice Nigerian prince that wants to share his inherited fortune with you. Most humans know that mails from nice Nigerian princes are usually just scams to get your money (sorry to all honest Nigerian princes out there 🙈), but how do we teach this knowledge to a machine? After all we do not want to delete all those spam emails by ourselves! There are a lot of ways to do this. For example, we could set up a rule to delete all mails that contain the words “Nigerian” and “prince”. But what if you want to talk to your Nigerian friend about Prince Harry and the latest gossip about the British royals? This example shows that such simple keyword-based rules are not “intelligent” at all, but rather stupid. I am sure most of you will easily recall a moment where your spam filter accidentally flagged an email that was obviously not spam—or the other way around. As it turns out, acting “intelligent” is quite hard for a machine.

The basic idea of AI is that this elusive “intelligence” can be harnessed from existing data. Each time you flag an email as spam, leave it in your inbox, or rescue that email address validation link from your spam folder, you generate examples of the human decisions that an AI spam filter should mimic. These examples contain two components: An email, and a label, which is the result of your decision if that mail was either “spam” or “not spam” (also called “ham”). It is important to have this label, because this is where information about “intelligence” is stored.

Comparing emails

Once we have a set of labeled data, we can rephrase our initial task from “classify whether this email is spam or not” to “decide the label of this email like humans did in these previous examples”. This is, of course, still not understandable for a machine. For some spam mails, a lookup in our data set might be enough, because we already received the exact same email, but usually there is a little variance in spam in order to avoid detection by such simple means. Therefore, we cannot avoid dealing with some form of uncertainty and as we all learned in school, uncertainty can best be tackled by statistics. For our task this means that we do not look for exact matches in the database, but rather for similar emails, and we have to find some kind of statistic calculations that expresses the similarity of two emails as a number. Expressing similarity between texts is in itself a highly complex task that requires a lot of intelligence, but there is a simple approach that we can take as first approximation: Just count the number of words that both emails have in common.

Consider this example:

[Database entry 1:]

Label: spam
Subject: Need trustworthy business partner

I am Mohammed Abacha, prince of Nigeria. I am the son of the late Nigerian
Head of State who died on the 8th of June 1998. I have secretly deposited
the sum of $30,000,000.00 with a security firm abroad. I shall be grateful
if you could receive this fund into your Bank account for safekeeping.


[Database entry 2:]

Label: ham
Subject: The tabloids are at it again

This is a great one: Woman&home titles "Meghan Markle and Prince Harry
weren't in 'great shape' mentally - reveals Tom Bradby who interviewed them
on Africa tour". And in that SAME article talking about mental health, they
end with "In other royal news, the Duchess of Cambridge stuns in red as she
steps out in London to promote photography book launch." Can you believe
it?! In other news ;), how are things in Nigeria?


[New email:]

Label: ???
Subject: The Guardian Nigeria

I just learned that Nigeria has a newspaper called "The Guardian Nigeria".
You don't find Prince Harry on the front page here, but you sure do find
him: https://guardian.ng/tag/prince-harry/ :D

In this example, our simple AI would find the following sets of matching words (assuming we split by punctuation and whitespace and transform letters to lowercase).

Matches entry 1: ["the", "nigeria", "i", "a", "you", "prince", "on"]
Matches entry 2: ["the", "nigeria", "that", "a", "you", "prince", "harry", "on"]

With this, we have seven matches for database entry one and eight matches for database entry two. Since the second entry has more matching words and was labeled as “ham” in the database, our spam detection AI will correctly copy this label for our friend’s email about the Guardian Nigeria. As it turns out, "harry" was our savior after all.

My first AI algorithm

With this example, we have defined our first AI algorithm. In fact, let’s look a little closer at the word “algorithm”. It is used a lot in conjunction with AI or with any complex automated system—often to the point that it sounds a little arcane and ominous. However, at the end of the day an algorithm is nothing more than a formal set of instructions that have to be carried out to calculate a result based on some input data. Our algorithm in this article could be described as follows:

Algorithm: Simple spam detector

Inputs:

• Database: list of labeled emails
• Query: unlabeled email that should be classified

Result:

• Label: the most fitting label for the query (either “spam” or “ham”)

Steps:

1. For all labeled emails in the database, calculate the number of matching words between that email and the query.
2. Find the database entry with the maximum number of matching words.
3. Output the label attached to this database entry.

Usually, algorithmic definitions tend to be a bit more detailed and technical, but the phrases that we used here like “for all entries in a database” or “find the maximum” are universal and simple enough that they can be directly translated into code that can be understood by a computer. The only remaining “difficult” part is the term “number of matching words”. We will translate this into a more detailed algorithmic form in one of the next posts, but for now let’s just accept that it too can be made understandable to a computer.

We therefore have turned the instruction for the human task “decide whether this email is spam or not” into a set of instructions for a machine that can now mimic human decisions—we have created our first actual artificial intelligence. There is, of course, a lot of room for improvement, and we will discuss some approaches in the next posts. However, if we generalize our approach by replacing the term “maximum number of matching words” with “maximum similarity” or “minimum distance”, we obtain the so-called nearest neighbor algorithm. This algorithm is a standard tool in any AI researcher’s inventory and if you do not only look at the single closest match but at the k closest matches (k just being any number like 5 or 100) you get the k-nearest neighbors algorithm, which often already is surprisingly accurate even for complex tasks.

This also highlights a curious fact about this algorithm: Notice that the instructions do not contain any information describing the nature of what a spam email actually looks like. We could use the very same algorithm to distinguish between “work” and “private” mail, by simply using a different database with a different set of labels. The algorithm itself is very generic and all the “intelligence” comes from the data. This is true for almost any artificial intelligence currently out there on the consumer market. Sef-driving cars drive based on sensory data collected from humans driving these vehicles in test scenarios; Alexa’s and Siri’s speech recognition is based on human speech samples that were manually transcribed by other humans; and maybe the fact that computers can beat top-level players at chess isn’t that mind-boggling anymore when you consider that those computers were able to use data from thousands of the world’s best chess players—probably even including the very players they were able to beat.

If you could follow my explanation to this point, let me congratulate you and formally bestow you the title “apprentice AI researcher”. If not, I would be very grateful if you could email me and tell me which parts you did not understand—ideally with a suggestion of how the text could be improved.

Edits and Acknowledgements

Since it was first published, this article has undergone the following changes due to reader feedback:

• Properly introduced the acronym AI (thanks to Franzi)
• Removed mentions of the word “algorithm” before it was explained (thanks to Fabi)
• Re-wrote the algorithmic definition, removing unnecessary low-level detail and adding context (thanks to Franzi)

You probably know that git uses SHA-1 hashes to identify a commit along with its full history of commits. This is a neat trick to ensure that you can absolutely be sure that two versions of a repository contain the exact same files with the exact same history. You can see these hashes in your git log or on GitHub and they are commonly used to identify versions of a repository that have no corresponding tag. They look like this e78e40aa4a2c03bf469ef842d37ec5eeaf49f37b, but usually a prefix like e78e40a is enough to uniquely identify them since there are already over 200 million of these prefixes and SHA-1 ensures a quite even distribution of hashes across these possibilities.

Git hashes in Julia’s package manager

However, recently I stumbled upon another kind of hash that git uses and that can also be used to identify the current version of a repository. In many of my Modelica projects, I need the latest version of my Julia package MoST.jl, which leads to entries like the follwing in the Manifest.toml that Julia uses to identify the versions of all packages installed for the current project.

[[ModelicaScriptingTools]]
deps = ["CSV", "DataFrames", "Documenter", "Markdown", "OMJulia", "PyCall", "Test", "ZMQ"]
git-tree-sha1 = "950e41b2d5a6aacd24d541cb95a172b0bc2a0230"
repo-rev = "main"
repo-url = "https://github.com/THM-MoTE/ModelicaScriptingTools.jl.git"
uuid = "9bd7ba1c-b518-491f-8f72-7efe190322aa"
version = "1.1.0"

The interesting part here is the git-tree-sha1 entry, which has clearly something to do with Git and with SHA-1, but what does the tree mean? This question became important for me, because I wanted to identify the actual version of my package that I used to produce the results for one of my papers. I knew I had this information stored in the Manifest.toml, so I did not bother to write it down at the time. This meant I was left with the task to obtain the SHA-1 hash identifying the commit version from this ominous “tree” hash.

Git tree hashes

It turns out that Git uses hashes for a lot of things and it also always uses SHA-1. The “tree” hashes are used to capture the contents of a directory tree including file contents and usage rights. Each commit actually has three SHA-1 hashes associated with it:

• The commit hash identifies the current commit with all of its history.
• The parent hash is a pointer to the parent of the current commit.
• The tree hash, which is used by Julia, captures the state of the whole directory tree of all the files in the repository.

Getting the commit hash associated with a tree hash

You can see these, by using git log --pretty=raw, which gives an output like the following:

commit a8a3dd68a37e21b1c2c835b7797e004940b2d9e6
tree 950e41b2d5a6aacd24d541cb95a172b0bc2a0230
parent 17577427038d695c4bce8696f36d2df726eb1c4d
author Christopher Schölzel <christopher.schoelzel@gmx.net> 1615040546 +0100
committer Christopher Schölzel <christopher.schoelzel@gmx.net> 1615040546 +0100

    adds documenter key for deploying docs

...

So one easy way to get the commit hash associated to a particular tree hash is to use

git log --pretty=raw | grep -B 1 <tree-hash>

Note that the association of tree hashes to commit hashes is not unique. If, for example, a commit reverts the changes of the previous commit, the new tree hash will be the same as the tree hash before the first commit.

Why use tree hashes to identify a commit?

Now the only question that remains is why the heck does Julia’s package manager use tree hashes instead of commit hashes, when only the latter can be used to uniquely identify the package version on GitHub, for example?

The answer is as simple as it is unpleasant: Because people like to rewrite history far more than they should. The main advantage of tree hashes is that they stay the same after git rebase has been used to “clean up” the history (which in my opinion is more accurately described as messing up the history 99% of the time). So if a Julia package developer rebases their repository, the Manifest.toml file of all people using this package stays valid as long as there is some commit in the new repository history that provides the exact same file content. This is, of course, perfectly safe for the sake of identifying a dependency to a precise version of a package to install, but it’s one of these features where you are kinda doing it wrong if you need it.