Counting things is an apparently trivial computational problem. In this article series — more of a journey than a short single-topic read — I’ll go over the process of producing a letter-based histogram from a piece of text, as a motive to progressively introduce several useful built-in and Standard Library tools.

Along the way I’ll share a few tips and thoughts on coding principles, related topics, like unicode text processing, and, for completeness, ideas on possible output production.

The key takeaway is that Python has many built-in and Standard Library tools that, used in the right combination, lead to simpler, shorter code, that being correct and easier to understand and manage, will also be faster.

The Task

Say we want to produce histogram representing the letter frequency — or counts — in a given piece of text. In a simple text analysis scenario, following good coding practices, we’d like to:

• Determine per-letter counts, regardless of the original letter case in the text.
  A capital “A” should be accounted for just like a lower case “a”, as yet another occurrence of the letter “A”.
• Discard any non-letters, like punctuation and whitespace.
• Produce a relatively nice human-readable result.
• Have clean, fast, robust and easy to read and modify code.

Being a “Journey on Counting”, we’ll mostly focus on counting and the many possible solutions (and challenges!) around it. Leaving the output production to a later stage, separating the counting process from the output handling, is a good mental and code design practice.

First Takes

As first approach to a solution we create a letter_counts.py module with the letter_counts_take1 function:

def letter_counts_take1(text):
    """
    Returns a dict where:
    - keys are letters (strings of length 1).
    - values are letter counts, from the text argument.
    """
    letter_counts = {}
    for letter in text:
        if not letter in letter_counts:
            # Haven't seen this letter yet: start counting at 1.
            letter_counts[letter] = 1
        else:
            # Already tracking this letter's count: increment it.
            letter_counts[letter] += 1
    return letter_counts

We then give it a test run, in the Python interactive prompt:

$ python3 -i letter_counts.py
>>> letter_counts_take1('Hello there!')
{'H': 1, 'e': 3, 'l': 2, 'o': 1, ' ': 1, 't': 1, 'h': 1, 'r': 1, '!': 1}
>>>

It seems to be working, but it is still pretty limited:

• It’s counting 'H' and 'h' separately.
• It’s not discarding whitespace, nor punctuation.

Let’s continue by creating an improved letter_counts_take2 function in our module:

import string

def letter_counts_take2(text):
    """ ... """
    letter_counts = {}
    for letter in text:
        if letter in string.whitespace or letter in string.punctuation:
            # Don't count whitespace and punctuation.
            continue
        # Combine lower-/upper-case letter occurrences.
        letter = letter.upper()
        if not letter in letter_counts:
            letter_counts[letter] = 1
        else:
            letter_counts[letter] += 1
    return letter_counts

A few notes:

• I’m using whitespace and punctuation, from the Standard Library’s string module to determine whether a given letter is to be discarded or not; these are useful, predefined character strings containing common characters in the categories their names suggest.
• Converting letter to its upper-case variant, and using it in the counting process, solves the problem of conflating the counts of upper- and lower-case letters; what you may not be aware of is that the .upper() method of Python strings handles all letters, including accented- and non-latin, for example, with 'resumé'.upper() returning 'RESUMÉ' and 'δ'.upper() returning 'Δ'.

Being orthogonal to the counting process itself, these improvements on text processing reflect common, real-world constraints on counting things: they often need to be filtered and/or pre-processed before the actual counting is done. Let’s do a quick check on how letter_counts_take2 behaves, using the Python interactive prompt, and bring our attention back to counting:

$ python3 -i letter_counts.py
>>> letter_counts_take2('Hello there!')
{'H': 2, 'E': 3, 'L': 2, 'O': 1, 'T': 1, 'R': 1}
>>>

It’s definitely looking better: we no longer have non-letters in the result, and the 'H' count is now 2, as expected — problem solved!

Can we do better than this? We certainly can.

One common source of code complexity is branching. In simple terms, the more ifs in a code block, the more complex it is, making it harder to reason about, making it more difficult — if not completely unfeasible — to test all possible execution paths, etc. We could argue that the code is small and simple enough and that it should be left as is. We could also, of course, counter argue that, one day, in the future, it will need changes and improvements, to support new capabilities, or to address edge cases it may not handle correctly. That is the day when this small and simple code will grow. And we’d like to have it as simple as possible. All the time.

One improvement would convert this if, checking whether or not a given letter is already accounted for, …

        if not letter in letter_counts:
            letter_counts[letter] = 1
        else:
            letter_counts[letter] += 1

…into something simpler and linear, like:

        running_count = letter_counts.get(letter, 0)
        letter_counts[letter] = running_count + 1

Using the dictionary’s .get(key[, default]) method we pass in 0 as the default running count for letter, which ensures we get the actual running count for any letters already in the dictionary, and 0 for the ones not accounted for yet. Updating letter counts is then a simple addition and assignment away.

With this change, we could create letter_counts_take3 as:

import string

def letter_counts_take3(text):
    """ ... """
    letter_counts = {}
    for letter in text:
        if letter in string.whitespace or letter in string.punctuation:
            continue
        letter = letter.upper()
        running_count = letter_counts.get(letter, 0)
        letter_counts[letter] = running_count + 1
    return letter_counts

Under the same simplification tone, avoiding branches, we might feel tempted to find a solution to eliminate the remaining if in the counting loop. We will look at a possible solution, as we progress. For now, let’s dive into the exploration of built-in and Standard Library tools that can be of use.

Bringing in some Tools

The letter counting solutions we’ve created use a Python dictionary to track the per-letter counts, where the keys are letters and the associated values are the respective letter counts. This is, after all, the perfect use case for dictionaries. One thing we had to be careful about, though, was the problem of per-letter count initialization:

• We started off using an if statement.
  It initialized the per-letter count to 1, if the given letter wasn’t account for yet, or incremented it, otherwise.
• We later changed it to a more linear approach.
  Getting letter counts using the dictionary’s .get() method, defaulting to 0 when unaccounted for, then incrementing it.

The Standard Library’s collections module includes a very useful type — I’d say, unknown to many — with the sole purpose of simplifying dictionary value initialization — the defaultdict. Let’s take a short break from counting things and learn a little bit about it.

>>> from collections import defaultdict
>>> def my_factory():
...     return 42
...
>>> dd = defaultdict(my_factory)

>>> len(dd)                 # Just created, no keys.
0
>>> dd['A'] = 21            # Add a key as in a regular dictionary.
>>> dd['A']
21
>>> len(dd)                 # One key, now, as expected.
1

>>> 'B' in dd               # The dictionary does not contain the 'B' key.
False
>>> dd['B']                 # It is created, with the default value, on first access.
42
>>> 'B' in dd               # The dictionary now contains the 'B' key.
True

>>> 'C' in dd               # The dictionary does not contain the 'C' key.
False
>>> dd['C'] += 1000         # Three-steps: get dd['C'], add 1000, assign result to dd['C'].
>>> dd['C']
1042

>>> from collections import defaultdict
>>> int()
0
>>> counter = defaultdict(int)
>>> counter['A']                    # How many 'A's have we counted so far?
0
>>> counter['B'] += 1               # Count one 'B'.
>>> counter['B'] += 1               # Count another 'B'.
>>> counter['B']                    # How many 'B's have we counted so far?
2

>>> from collections import defaultdict
>>> list()
[]
>>> grouper = defaultdict(list)
>>> grouper['A']                    # What's in group 'A'?
[]
>>> grouper['B'].append('bee')      # Add 'bee' to group 'B'.
>>> grouper['B'].append('bear')     # Add 'bear' to group 'B'.
>>> grouper['B']                    # What's in group 'B'?
['bee', 'bear']
>>> grouper['C'].append('cat')      # Add 'cat' to group 'C'.
>>> grouper['C']                    # What's in group 'C'?
['cat']

With the defaultdict interlude completed, let’s get back to the letter counting function and see how we can simplify it further. Here’s what we had so far (copied here, no need to scroll up/down):

import string

def letter_counts_take3(text):
    """ ... """
    letter_counts = {}
    for letter in text:
        if letter in string.whitespace or letter in string.punctuation:
            continue
        letter = letter.upper()
        running_count = letter_counts.get(letter, 0)
        letter_counts[letter] = running_count + 1
    return letter_counts

With what we now know about defaultdicts we can make it shorter and more readable, which is always a good thing. More, relying on the existing Standard Library code, widely used and tested, is also a good principle: the more we get from it, the less we need to code and, importantly, the less we need verify and test. Here’s our take, with letter_counts_take4:

from collections import defaultdict
import string

def letter_counts_take4(text):
    """ ... """
    letter_counts = defaultdict(int)            # Better counter than a plain dictionary.
    for letter in text:
        if letter in string.whitespace or letter in string.punctuation:
            continue
        letter_counts[letter.upper()] += 1      # No need to initialize: just increment the count.
    return letter_counts

Ok, it’s looking better and better. Can it be improved?

Well, it probably can, in several ways. There are two intertwined things that stand out, to me:

• On one hand, we have an if statement, used to discard non-letters. Code with no ifs is always simpler, how could we change it?
• On the other, is the fact that we have a pretty much hard coded non-letter condition on the if. Could we avoid it?

Then, there is at least another one, not so obvious to many, but that we have alluded to before:

• We’re using a hard-coded letter transformation with letter.upper() to combine upper-/lower-case letters into common counts. This does not support, for example, conflating accented letter counts with their non-accented variations. How could it be improved?

These are possible improvements both in the code design and capabilities domains, not strictly Python things. Let’s explore them and see what we can come up with, starting with the idea of replacing the if with something else that doesn’t involve branching.

A cursory glance at the code tells us that if we are to eliminate the non-letter detection if statement in the loop, we will need to do the filtering beforehand. That means something along the lines of creating a filtered_text object, copied from text, where any non-letters are discarded. We could try to do that with a list comprehension…

filtered_text = [l for l in text if l not in string.whitespace and l not in string.punctuation]

…but that would be misleading and wrong (and not really computationally efficient, but that’s a whole other topic): the fact is that the if statement we’re trying to eliminate is still somewhat half-hidden in the list comprehension. If we are to get rid of branches, this is not it.

A possible solution is in the little-used str.translate method — from a sneak peek at the docstring we gather that:

• It takes a translation table as its single argument.
• Returns a copy of the string in which each character has been mapped through the given translation table.
• Characters mapped to None are deleted.

Let’s take another short break, now to explore and learn how str.translate can help.

>>> xlate_table = {
...     ord('.'): ord('!'),
...     ord('w'): ord('W'),
... }
>>> 'Hello world...'.translate(xlate_table)
'Hello World!!!'

>>> replace_these_ones = '.w'
>>> with_these_instead = '!W'
>>> xlate_table = str.maketrans(replace_these_ones, with_these_instead)
>>> 'Hello world...'.translate(xlate_table)
'Hello World!!!'

>>> xlate_table = str.maketrans('.w', '!W', ' ')
>>> 'Hello world...'.translate(xlate_table)
'HelloWorld!!!'

>>> filter_table = str.maketrans('', '', ' .')
>>> 'Hello world...'.translate(filter_table)
'Helloworld'

With what we’ve just seen about str.translate and str.maketrans, we’re ready to take a stab at eliminating the remaining if in our code. Here’s how letter_count_take5 can look like:

from collections import defaultdict
import string

def letter_counts_take5(text):
    """ ... """
    letter_filter = str.maketrans('', '', string.whitespace + string.punctuation)
    filtered_text = text.translate(letter_filter)
    letter_counts = defaultdict(int)
    for letter in filtered_text:
        letter_counts[letter.upper()] += 1
    return letter_counts

Could we say it’s better now? I say we can: we have the same amount of code, in line count, but we got rid of a branching statement inside the loop — simpler code, easier to test, easier to manage. The cost is consuming additional RAM for the filtered_text string and using str.translate and str.maketrans which may not be 100% familiar to every Pythonista out there (but always a help(str) away, thus, hardly a cost). The benefit is less branching (in our code, of course, str.translate will need to branch somehow, but we don’t care) and, probably, increased performance given that str.translate is implemented in C, very well suited to tight loop processing, as is the case.

What next? Well, expanding on the str.translate idea, we can avoid the letter.upper() call in every loop iteration, striving for a single-pass, upper-casing and character filtering operation. Here’s a possible implementation:

from collections import defaultdict
import string as s

def letter_counts_take6(text):
    """ ... """
    # Upper-case the 26 english alphabet letters and discard whitespace and punctuation.
    xlate_table = str.maketrans(s.ascii_lowercase, s.ascii_uppercase, s.whitespace + s.punctuation)
    just_letters = text.translate(xlate_table)
    letter_counts = defaultdict(int)
    for letter in just_letters:
        letter_counts[letter] += 1
    return letter_counts

Whether this is a good idea or not depends on our ultimate purpose, and established requirements. Attentive readers may notice that, now using str.translate from string.ascii_lowercase to string.ascii_uppercase, this version does not combine lower-/upper-case variations of accented letters — it now counts 'é' and 'É' separately, while the previous one combined them (but neither one merges the 'é', 'É' and 'e' counts, though). We could build a translation table that, in a single pass, would have str.translate upper-casing and stripping accents from all letters in a given domain, while also filtering out unwanted whitespace and punctuation, but there’s no immediate solution to that (more on this, later).

What this formulation suggests, given the barebones letter counting loop, is simplifying it further using yet another tool in the Standard Library’s collection module — the Counter.

>>> from collections import Counter
>>> numbers = [1, 1, 2, 2, 2, 2, 3]
>>> c = Counter(numbers)
>>> c[2]                            # What's the count for 2?
4
>>> c                               # Give me all the counts.
Counter({2: 4, 1: 2, 3: 1})

>>> from collections import Counter
>>> letters = 'aabbbc'
>>> c = Counter(letters)
>>> c['a']                          # What's the count for 'a'?
2
>>> c                               # Give me all the counts.
Counter({'b': 3, 'a': 2, 'c': 1})
>>> c['x']                          # The count of an unknown is 0.
0

With this knowledge, the plain letter counting loop and its associated defaultdict — from letter_counts_take6 — can be replaced with a Counter object, doing precisely the same thing:

from collections import Counter
import string as s

def letter_counts_take7(text):
    """ ... """
    xlate_table = str.maketrans(s.ascii_lowercase, s.ascii_uppercase, s.whitespace + s.punctuation)
    just_letters = text.translate(xlate_table)
    return Counter(just_letters)

This is definitely shorter and simpler: there is no branching or looping in our code. It’s hidden away, and handled by existing, widely tested, probably faster code, in the built-in and Standard Library tools. What’s not to like about it?

There certainly still is the fact that the letter grouping, via upper-casing, and filtering is hard-coded in the str.maketrans call. That’s something we’d probably like to tackle. But let’s step back for a moment, take a deep breath, and review our progress.

$ python3 -m timeit \
  -s 'from letter_counts import letter_counts_take1; text="Hello there!"' \
  -c 'letter_counts_take1(text)'

So, where do we stand? We have a very short and clean implementation in letter_counts_take7, that is also the fastest so far, as long as the input text is large enough: ~200 characters or more, per the medium sized input text measurements above. Importantly, it does meet the initially defined requirements — let’s not forget that.

Before calling the counting process done, and moving on to exploring ideas on output generation, there is one more thing we should address (yes, even though our code is short and fast and meets the requirements). The fact that the letter filtering and grouping is based on non-changable, hard-coded values, using string.ascii_lowercase and its friends. This is unnecessarily limiting and, in general, a bad coding practice.

In the next article in this series we’ll explore just that, hopefully getting to a point where we’re satisfied with the letter-counting process, finally moving on to producing nice and accessible human-readable output. As we will see, eliminating the hard-coded parts in our function will lead us to a path of Unicode and API design exploration: two topics with a lot to be said by themselves.

We’ll see where that takes us.

[ 2018-04-29 UPDATE: Follow-up article here. ]

Wrap up

This article kicks-off a Python journey, motivated by the topic of counting letters. We’ve built several versions of letter counting functions, progressively simpler, using powerful built-in and Standard Library tools. Knowing them and how they can be applied to a given problem is a valuable skill — like with other skills, mastering it takes time; don’t feel rushed, just keep at it, at your own pace. We’ve also highlighted a few universal coding principles: these are not related to knowing about, or using, any specific tools, they’re not even Python specific.

Summarizing, I would highlight:

• Counting things, like many computational problems, looks simple.
  It often is, but the need for filtering and processing is common, which introduces complexity.
  The devil is in the detail, a very appropriate idiom here: the more requirements, the more prone to complexity a given problem becomes. Creating simple code to solve complex problems is, by definition, not a simple task, but one we should strive for. If possible, agreeing on the simplest requirements possible, is a good idea, while coding with an eye into the future to accommodate change.

• The Python Standard Library includes many useful tools, always at hand.
  Learning its ways is more of a process than an objective in itself. The collections module contains, among other useful types, the defaultdict and the Counter which we explored and learned about: they both help writing simpler, shorter and faster code. While the Counter is very much a single-purpose thing, the defaultdict has many useful applications, including counting, grouping, and more. The string module — not to be confused with the str built-in type, also very powerful — includes a few little nuggets that, again, we can build upon and use to simplify existing code.

• The simpler the code, the better.
  Less code is always better code: there’s less to read and understand, less to be executed, less to be maintained and changed in the future. It will also tend to have fewer bugs, assuming that, statistically, bug counts per line are constant, in a given setting. Simpler code also means less branching and looping: linear code — with no ifs, fors or whiles — is easier to grasp and test. We can’t always create it like so, but trying to minimize those is a good long term practice. Simpler code is faster: for both humans and computers.

Thanks for reading. See you soon.