An Apprentice Experiment in Python Programming

A couple weeks ago Zvi made an apprentice thread. I have always wanted to be someone’s apprentice, but it didn’t occur to me that I could just …ask to do that. Mainly I was concerned about this being too big of an ask. I saw gilch’s comment offering to mentor Python programming. I want to level up my Python skills, so I took gilch up on the offer. In a separate comment, gilch posed some questions about what mentors and/or the community get in return. I proposed that I, as the mentee, document what I have learned and share it publicly.

Yesterday we had our first session.

Background

I had identified that I wanted to fill gaps in my Python knowledge, two of which being package management and decorators.

Map and Territory

Gilch started by saying that “even senior developers typically have noticeable gaps,” but building an accurate map of the territory of programming would enable one to ask the right questions. They then listed three things to help with that:

Documentation on the Python standard library. “You should at least know what’s in there, even if you don’t know how to use all of it. Skimming the documentation is probably the fastest way to learn that. You should know what all the operators and builtin functions do.”

Structure and Interpretation of Computer Programs for computer science foundation. There are some variants of the book in Python, if one does not want to use Scheme.

CODE as “more of a pop book” on the backgrounds.

In my case, I minored in CS, but did not take operating systems or compilers. I currently work as a junior Python developer, so reading the Python standard library seems to be the lowest hanging fruit here, with SICP on the side, CODE on the back burner.

Decorators

The rest of the conversation consisted of gilch teaching me about decorators.

Gilch: Decorators are syntactic sugar.

@foo
def bar():
    ...

means the same thing as

def bar():
    ...
bar = foo(bar)

Decorators also work on classes.

@foo
class Bar:
    ...

is the same as

class Bar:
    ...
Bar = foo(Bar)

An Example from pytest

At this point I asked if decorators were more than that. I had seen decorators in pytest:

@pytest.fixture
def foo():
    ...
    
def test_bar(foo):  # foo automatically gets evaluated inside the test
    ...

Does this mean that, when foo is passed in test_bar as a variable, what gets passed in is actually something like pytest.fixture(foo)?

Gilch identified that there might be more than decorators involved in this example, so we left this for later and went back to decorators.

Decorators, Example 1

I started sharing my screen, gilch gave me the first instruction: Try making a decorator.

def test_decorator(foo):
    return 42

@test_decorator
def bar():
    print('hi')
    
print(bar())

Then, before I ran the program, gilch asked me what I expected to happen when I run this program, to which I answered that hi and 42 would be printed to console. At this point, gilch reminded me that decorators were sugar, and asked me to write out the un-sugared translation of the function above. I wrote:

def bar():
    bar = test_decorator(bar)
    return bar

I ran the program, and was surprised by the error TypeError: 'int' object is not callable. I expected bar to still be a function, not an integer.

Gilch asked me to correct my translation of my program based on the result I saw. It took me a few more tries, and eventually they showed me the correct translation:

def bar():
    print('hi')

bar = test_decorator(bar)

Then I realized why I was confused—I had the idea that decorators modify the function they decorate directly (in the sense of modifying function definitions), when in fact the actions happen outside of function definitions.

Gilch explained: A decorator could [modify the function], but this one doesn’t. It ignores the function and returns something else. Which it then gives the function’s old name.

Decorators, Example 2

Gilch: Can you make a function that subtracts two numbers?

Me:

def subtract(a, b):
    return a - b

Gilch: Now make a decorator that swaps the order of the arguments.

My first thought was to ask if there was any way for us to access function parameters the same way we use sys.argv to access command line arguments. But gilch steered me away from that path by pointing out that decorators could return anything. I was stuck, so gilch suggested that I try return lambda x, y: y-x.

Definition Time

My program looked like this at this point:

@swap_order
def subtract(a, b):
    return a - b

def swap_order(foo):
    return lambda x, y: y - x

PyCharm gave me an warning about referencing swap_order before defining it. Gilch explained that decoration happened at definition time, which made sense considering the un-sugared version.

Interactive Python

Up until this point, I had been running my programs with the command python3 <file>. Gilch suggested that I run python3 -i <file> to make it interactive, which made it easier to experiment with things.

Decorators, Example 2

Gilch: Now try an add function. Decorate it too.

Me:

def swap_order(foo):
    return lambda x, y: y - x

@swap_order
def subtract(a, b):
    return a - b

@swap_order
def add(a, b):
    return a + b

Gilch then asked, “What do you expect the add function to do after decoration?” To which I answered that the add function would return the value of its second argument subtracted by the first argument. The next question gilch asked was, “Can you modify the decorator to swap the arguments for both functions?”

I started to think about sys.argv again, then gilch hinted, “You have ‘foo’ as an argument.” I then realized that I could rewrite the return value of the lambda function:

def swap_order(foo):
    return lambda x, y: foo(y, x)

I remarked that we’d see the same result from add with or without the decorator. Gilch asked, “Is addition commutative in Python?” and I immediately responded yes, then I realized that + is an overloaded operator that would work on strings too, and in that case it would not be commutative. We tried with string inputs, and indeed the resulting value was the reverse-ordered arguments concatenated together.

Gilch: Now can you write a decorator that converts its result to a string?

I wrote:

def convert_to_str(foo):
    return str(foo())

It was not right. I then tried

def convert_to_str(foo):
    return str

and it was still not right. Finally I got it:

def convert_to_str(foo):
    lambda x, y: str(foo(x, y))

There was some pair debugging that gilch and I did before I reached the answer. Looking at the mistakes I’ve made here, I see that I still hadn’t grasped the idea that decorators would return functions that transform the results of other functions, not the transformed result itself.

Gilch: Try adding a decorator that appends ”, meow.” to the result of the function.

I verbalized the code in my head out loud, then asked how we’d convert the types of the function return value to string before appending ", meow" to it. Gilch suggested f"{foo(x, y)}, meow" and we had our third decorator.

We then applied decorators in different orders to show that multiple decorators were allowed, and that the order of decorators decided the order of application.

Splat

When we were writing the convert_to_str decorator, I commented that this would only work for functions that take in exactly 2 arguments. So gilch asked me if I was familiar with the term “unpacking” or “splat.” I knew it was something like ** but didn’t have more knowledge than that.

How Many Arguments

Gilch asked me, “How many arguments can print() take?” To which I answered “infinite.” They then pointed out that it was different from infinite—zero would be valid, or one, or two, and so on. So the answer is “any number, ” and the next challenge would be to make convert_to_str work with any number of arguments.

print()

We tried passing different numbers of arguments into print(), and sure enough it took any number of arguments. Here, gilch pointed out that print actually printed out a newline character by default, and the default separator was a space. They also pointed out that I could use the help(print) command to access the doc in the terminal without switching to my browser.

type(_)

Gilch pointed out that I could use the command type(_) to get the type of the previous value in the console, without having to copy and paste.

Splat

To illustrate how splat worked, gilch gave me a few commands to try. I’d say out loud what I expected the result to be before I ran the code. Sometimes I got what I expected; sometimes I was surprised by the result, and gilch would point out what I had missed. To illustrate splat in arrays, gilch gave two examples: print(1,2,3,*"spam", sep="~") and print(1,2,*"eggs",3,*"spam", sep="~"). Then they showed me how to use ** to construct a mapping: (lambda **kv: kv)(foo=1, bar=2)

Dictionary vs. Mapping

We went off on a small tangent on dictionary vs. mapping because gilch pointed out that dictionary was not the only type of mapping and tuple is no the only type of iterable. I asked if there were other types of mapping in Python, and they listed OrderedDict as a subtype and the Mapping abstract class.

Parameter vs. Argument, Packing vs. Unpacking

At this point gilch noticed that I kept using the word “unpacking.” I also noticed that I was using the term “argument” and “parameter” interchangeably here. Turns out the distinction is important here—the splat operator used on a parameter packs values in a tuple; used on an argument unpacks iterable into separate values. For example, in (lambda a, b, *cs: [a, b, cs])(1,2,3,4,5), cs is a parameter and * packs the values 3, 4, 5 into a tuple; in print(*"spam", sep="~"), "spam" is an argument and * unpacks it into individual characters.

Dictionaries

Gilch gave me another example: Try {'x':1, **dict(foo=2,bar=3), 'y':4}. I answered that it would return a dictionary with four key-value pairs, with foo and bar also becoming keys. Gilch then asked, “in what order?” To which I answered “dictionaries are not ordered.”

“Not true anymore,” gilch pointed out, “Since Python 3.7, they’re guaranteed to remember their insertion order.” We looked up the Python documentation and it was indeed the case. We tried dict(foo=2, **{'x':1,'y':4}, bar=3) and got a dictionary in a different order.

Hashable Types

I asked if there was any difference in defining a dictionary using {} versus dict(). Gilch compared two examples: {42:'spam'} works and dict(42='spam') doesn’t. They commented that keys could be any hashable type, but keyword arguments were always keyed by identifier strings. The builtin hash() only worked on hashable types.

I don’t fully understand the connection between hashable types and identifier strings here, it’s something that I’ll clarify later.

Parameter vs. Argument, Packing vs. Unpacking

Gilch gave another example: a, b, *cs, z = "spameggs"

I made a guess that cs would be an argument here, so * would be unpacking, but then got stuck on what cs might be. I tried to run it:

>>> a, b, *cs, z = "spameggs"
>>> a
's'
>>> b
'p'
>>> cs
['a', 'm', 'e', 'g', 'g']
>>> z
's'

Gilch pointed out that cs was a store context, not a load context, which made it more like a parameter rather than an argument. Then I asked what store vs. load context was.

Context

Gilch suggested, import ast then def dump(code): return ast.dump(ast.parse(code)). Then something like dump("a = a") would return a nexted object, in which we can locate the ctx value for each variable.

This reminded me of lvalue and rvalue in C++, so I asked if they were the same thing as store vs. load context. They were.

Splat

Gilch tied it all together, “So for a decorator to pass along all args and kwargs, you do something like lambda *args, **kwargs: foo(*args, **kwargs). Then it works regardless of their number. Arguments and keyword arguments in a tuple and dict by keyword. So you can add, remove, and reorder arguments by using decorators to wrap functions. You can also process return values. You can also return something completely different. But wrapping a function in another function is a very common use of decorators. You can also have definition-time side effects. When you first load the module, it runs all the definitions—This is still runtime in Python, but you define a function at a different time than when you call it. The decoration happens on definition, not on call.”

We wrapped up our call at this point.

Observations

As we were working through the examples, we’d voice out what we expect to see when we run the code before actually running to verify. Several times gilch asked me to translate a decorated function into an undecorated one. This was helpful for me to check my understanding of things.
Another thing I found valuable were the tips and tricks I picked up from gilch throughout the session, like interactive mode; and the clarification of concepts, like the distinction between parameter and argument.
Gilch quizzed me throughout the session. This made things super fun! I haven’t had the opportunity for someone to keep quizzing me purely for learning (as opposed to giving me a grade or deciding whether to hire me) for the longest time! I guess that reading through well-written text tends to be effective for familiarizing oneself with concepts, while asking/answering questions is effective at solidifying and synthesizing knowledge.
In this post, I tried to replicate the structure of my conversation with gilch as much as possible (the fact that gilch’s mic was broken so they typed while I talked made writing this post so much easier—I had their half of the transcript generated for me!) since we went off on some tangents and I wanted to provide context for those tangents. I think of a conversation as a tree structure—we start with a root topic and go from there. A branch would happen when we go off on a tangent and then later come back to where we left off before the tangent. Sometimes two sections of this post would have the same section headings; a second time a section heading is used indicates that we stopped the tangent and went back to where we branched off.