- Mon 24 September 2018
- Python
- #intermediate
The other day, while reviewing the Assignment Section exercises for a training course I was about to deliver, wanting to type dir()
into a Python REPL, my fingers went for del()
instead. At first I didn’t even notice it but then, something in the back of my head called for attention: I had indeed typed del()
which, to my surprise, Python happily executed, producing no exceptions.
Since I knew that del
is a Python statement and a reserved keyword, there was no way I was calling a del
function that I could have accidentally created. This left me wondering, hmmm… What’s going on here?!
In this article I’ll describe the steps I went through in the process of grasping why del()
is valid Python1 and exactly what it does, sharing a few techniques that can prove to be useful to others in understanding funny looking Python code tidbits, like this one felt to me.
Let the games begin: del()
works
It all started with a freshly launched CPython 3.7 REPL, where I entered:
>>> del()
>>>
Observations:
- No exception was raised.
- Whatever
del()
did, no value was returned.
Conclusion:
- Whatever Python did, it did something I wasn’t grasping.
Since del
is a reserved keyword and a statement, I immediately went for the following hypothesis “The Python tokenizer and parser are interpreting del()
as del ()
“ — note the space between del
and ()
. That seemed reasonable but, given that ()
is an empty tuple literal and that del
deletes references to objects (like variables, attributes, and more), the logic behind deleting an object literal still made no sense to me.
Deleting a literal makes no sense
From the hypothesis that del()
was being parsed as del ()
, I then confirmed my second expectation: you really can’t delete a literal.
>>> del {}
File "<stdin>", line 1
SyntaxError: can't delete literal
>>> del 42
File "<stdin>", line 1
SyntaxError: can't delete literal
>>>
Interestingly, these two forms work:
>>> del ()
>>> del []
Knowing how Python assignment works and what the del
statement does, I was positive that deleting literals should fail, like it did for the dict
and int
literals. It still provocatively seemed to work with tuple
and list
literals… What was going on here?
Digging Deeper
It was time to bring in a powerful tool. I decided to disassemble the mysterious statement and see what kind of byte-code was being generated which, apparently, was being happily executed by the Python VM. To do that, I created the mystery
function containing a single del ()
statement, and then used the dis
module, from the Standard Library:
>>> import dis
>>>
>>> def mystery():
... del ()
...
>>> dis.dis(mystery)
2 0 LOAD_CONST 0 (None)
2 RETURN_VALUE
>>>
Interesting! The del
statement in the mystery
function seemed to have been completely eliminated… Was I getting it right? I went further with two other tests. I first created a do_nothing
function, disassembling it…
>>> def do_nothing():
... pass
...
>>>
>>> dis.dis(do_nothing)
2 0 LOAD_CONST 0 (None)
2 RETURN_VALUE
>>>
…and then the triple_mystery
function, calling del ()
three times in a row and, again, disassembled it:
>>> def triple_mystery():
... del ()
... del ()
... del ()
...
>>>
>>> dis.dis(triple_mystery)
4 0 LOAD_CONST 0 (None)
2 RETURN_VALUE
>>>
Hopefully, even someone who is not familiar with Python byte-code2 can verify that both the mystery
and triple_mystery
functions compile down to the same exact byte-code, which does precisely nothing, much like the do_nothing
function I created for comparison3.
I confess that these results, while consistent with the initial observations — nothing is returned and not exception is raised — puzzled me even more, leading me to direct my quest somewhere else…
Has the compiler gone crazy?
When in doubt, the answer to such a question is most probably “no, the compiler has not gone crazy, there’s something you’re misunderstanding”. Especially when dealing with widely-used, time-tested compilers like the one in CPython.
To figure out why the del ()
statement was generating no byte-code at all, I went for a divide and conquer diagnostic approach. Here’s what we know, at this point:
- The input code is
del ()
. - It is fed to the compiler.
- The output byte-code is empty.
What’s going on in-between, inside the compiler itself? Well, between the source Python code and the executable Python byte-code, the Python compiler generates an intermediate representation called an Abstract Syntax Tree — knowing that, in simple terms, we can break Step 2., above, into:
- The input code is
del ()
. - It is fed to the compiler:
- The input code is parsed into an Abstract Syntax Tree.
- The Abstract Syntax Tree is used to generate byte-code.
- The output byte-code is empty.
Fortunately, the Python Standard Library includes the ast module which allows us to easily create an Abstract Syntax Tree from a given piece of valid Python code. That’s precisely where I went next:
>>> import ast
>>> node = ast.parse('del ()')
The node
object is an ast.Module
object which includes a body
attribute…
>>> node.body
[<_ast.Delete object at 0x10c1b1b00>]
…which is a list of nodes containing, in our case, unsurprisingly, an instance of a ast.Delete
object. The ast.Delete
object, in turn, has a targets
attribute which is a list of the targets for the delete statement:
>>> delete = node.body[0]
>>> delete.targets
[<_ast.Tuple object at 0x10c1bc8d0>]
From the above, it seems clear that the parsed del ()
statement has been understood to mean: a) delete a single target, b) the target is a tuple. Let’s check that tuple’s AST representation by taking a look at the elts
attribute in the ast.Tuple
object:
>>> delete_target = delete.targets[0]
>>> delete_target.elts
[]
It comes out as an empty list, pretty much inline with what we could expect to represent the elements in our source empty tuple.
Compacting the exploratory code a bit, we could write:
>>> ast.parse('del ()').body[0].targets
[<_ast.Tuple object at 0x10c424c88>]
>>> ast.parse('del ()').body[0].targets[0].elts
[]
Preliminary Conclusion
The above observations clearly confirm that parsing del ()
results in an AST representing the deletion of a single target, which is a tuple with no elements. Thus, something is going on, in the later stages of the compiler, that is “clever enough” to somehow discard such statement, given that no byte-code is produced from it… The question is: should it? Does it make any sense? It probably does, let’s see.
Stepping back for a while, let’s see what kind of AST’s we get from more common delete statements:
>>> ast.parse('del n').body[0].targets
[<_ast.Name object at 0x10c424278>]
>>> ast.parse('del n, m').body[0].targets
[<_ast.Name object at 0x10c424c88>, <_ast.Name object at 0x10c424860>]
The targets are ast.Name
objects, which makes sense, even if we don’t dig any deeper. What about deleting targets in a tuple?
>>> ast.parse('del (n,)').body[0].targets
[<_ast.Tuple object at 0x10b9b5908>]
>>> ast.parse('del (n,)').body[0].targets[0].elts
[<_ast.Name object at 0x10ba50cf8>]
>>> ast.parse('del (n, m)').body[0].targets
[<_ast.Tuple object at 0x10b9b58d0>]
>>> ast.parse('del (n, m)').body[0].targets[0].elts
[<_ast.Name object at 0x10ba50cf8>, <_ast.Name object at 0x10ba50d68>]
Interesting, the target is a single ast.Tuple
, containing ast.Name
objects as elements. Do these forms even work?
>>> n = 42
>>> del (n,)
>>> n
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'n' is not defined
>>>
>>> n, m = 42, 24
>>> del (n, m)
>>> 'n' in dir(), 'm' in dir()
(False, False)
Yes, they certainly do.
What about something somewhat crazy like…
>>> n, m, o, p = range(4)
>>> del [n, m,], (o,), p
>>> dir()
['__annotations__', '__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__', 'ast']
…or…
>>> n, m, o, p, q, r = range(6)
>>> del (n, [m, o, (p, (q, (r,)))])
>>> dir()
['__annotations__', '__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__', 'ast']
It all works. As I progressed through these attempts, I developed an intuition like I believe some of the readers will. This all felt familiar, especially in the context of Python assignment, around which the del
statement is an important tool.
That’s when I decided to do something I probably never did before: I went reading the del
statement documentation, where the first line reads del_stmt ::= "del" target_list
, and target_list
links here, under the Assignment Statements section of the documentation.
Conclusion
The del
statement grammar rule clearly states that the del
keyword is to be followed by a target list which, in turn, can be as simple as a single name, or as complex as an arbitrarily nested structure of tuples, lists, and more, with lots of possibilites in-between.
Such target lists are used in many other places, such as on the left side of an assignment statement, in for
statements, in function arguments, and eventually more.
The specification for target lists includes empty tuple
s and list
s as valid targets. That’s why del ()
and its funny cousin del []
work the way they do: both include valid target lists with no targets to delete.
Wrap-up
In retrospect, the reason why del()
works, and what it does, is pretty obvious and natural to me. Some people, faced with the same code, might have understood it immediately, while others might have taken a completely different path in understanding what could be going on.
I don’t believe in a right vs. wrong way of understanding things and, more importantly, of investigating unexpected behaviours. I decided to write these words sharing the process I took and how useful the dis
and ast
modules were for me. Maybe these tools will be useful to someone else having the need to understand some funny behaving code they don’t immediately understand.
Lastly, something I usually say a lot during my training sessions: “The Python documentation is very good. When in doubt, refer to it!”
Now, why didn’t I go there straight away? 😊
Post Scriptum
Interestingly, the del ()
statement does not seem to run on CPython 2.7, CPython 3.5, PyPy 6.0.0 2.7.13, nor PyPy 6.0.0 3.5.3, resulting in SyntaxError: can't delete ()
all around. I did not dig further, but after my findings, and even though del ()
is obviously a no-op, I kind of like its consistency.
[ 2018-09-27 UPDATE: Fixed typo on third paragraph. Thanks to Alex for pointing it out. ]
-
After having written these words I went ahead and tried a few different Python interpreters, empirically finding out that this only holds for recent versions of CPython. The exploratory techniques I illustrate are, nonetheless, certainly applicable to a much wider range of Python interpreters and versions. ↩
-
Python byte-code is the result of compiling Python code and what the
dis.dis
function prints. More importantly, it is what the Python VM actually executes under the covers. ↩ -
For the sake of correctness, but without diving into Python byte-code material, the two operations we’re seeing here —
LOAD_CONST
andRETURN_VALUE
— are actually implicitly added by the Python compiler, given that the disassembled functions omit an explicit finalreturn
statement; thedel ()
statements, much like thepass
statement, lead to no byte-code output from the compiler. ↩