A couple weeks ago Zvi made an apprentice thread. I have always wanted to be someone’s apprentice, but it didn’t occur to me that I could just …ask to do that. Mainly I was concerned about this being too big of an ask. I saw gilch’s comment offering to mentor Python programming. I want to level up my Python skills, so I took gilch up on the offer. In a separate comment, gilch posed some questions about what mentors and/or the community get in return. I proposed that I, as the mentee, document what I have learned and share it publicly.
Yesterday we had our first session.
Background
I had identified that I wanted to fill gaps in my Python knowledge, two of which being package management and decorators.
Map and Territory
Gilch started by saying that “even senior developers typically have noticeable gaps,” but building an accurate map of the territory of programming would enable one to ask the right questions. They then listed three things to help with that:
Documentation on the Python standard library. “You should at least know what’s in there, even if you don’t know how to use all of it. Skimming the documentation is probably the fastest way to learn that. You should know what all the operators and builtin functions do.”
In my case, I minored in CS, but did not take operating systems or compilers. I currently work as a junior Python developer, so reading the Python standard library seems to be the lowest hanging fruit here, with SICP on the side, CODE on the back burner.
Decorators
The rest of the conversation consisted of gilch teaching me about decorators.
Gilch: Decorators are syntactic sugar.
@foo
def bar():
...
means the same thing as
def bar():
...
bar = foo(bar)
Decorators also work on classes.
@foo
class Bar:
...
is the same as
class Bar:
...
Bar = foo(Bar)
An Example from pytest
At this point I asked if decorators were more than that. I had seen decorators in pytest:
@pytest.fixture
def foo():
...
def test_bar(foo): # foo automatically gets evaluated inside the test
...
Does this mean that, when foo is passed in test_bar as a variable, what gets passed in is actually something like pytest.fixture(foo)?
Gilch identified that there might be more than decorators involved in this example, so we left this for later and went back to decorators.
Decorators, Example 1
I started sharing my screen, gilch gave me the first instruction: Try making a decorator.
Then, before I ran the program, gilch asked me what I expected to happen when I run this program, to which I answered that hi and 42 would be printed to console. At this point, gilch reminded me that decorators were sugar, and asked me to write out the un-sugared translation of the function above. I wrote:
def bar():
bar = test_decorator(bar)
return bar
I ran the program, and was surprised by the error TypeError: 'int' object is not callable. I expected bar to still be a function, not an integer.
Gilch asked me to correct my translation of my program based on the result I saw. It took me a few more tries, and eventually they showed me the correct translation:
def bar():
print('hi')
bar = test_decorator(bar)
Then I realized why I was confused—I had the idea that decorators modify the function they decorate directly (in the sense of modifying function definitions), when in fact the actions happen outside of function definitions.
Gilch explained: A decorator could [modify the function], but this one doesn’t. It ignores the function and returns something else. Which it then gives the function’s old name.
Decorators, Example 2
Gilch: Can you make a function that subtracts two numbers?
Me:
def subtract(a, b):
return a - b
Gilch: Now make a decorator that swaps the order of the arguments.
My first thought was to ask if there was any way for us to access function parameters the same way we use sys.argv to access command line arguments. But gilch steered me away from that path by pointing out that decorators could return anything. I was stuck, so gilch suggested that I try return lambda x, y: y-x.
Definition Time
My program looked like this at this point:
@swap_order
def subtract(a, b):
return a - b
def swap_order(foo):
return lambda x, y: y - x
PyCharm gave me an warning about referencing swap_order before defining it. Gilch explained that decoration happened at definition time, which made sense considering the un-sugared version.
Interactive Python
Up until this point, I had been running my programs with the command python3 <file>. Gilch suggested that I run python3 -i <file> to make it interactive, which made it easier to experiment with things.
Decorators, Example 2
Gilch: Now try an add function. Decorate it too.
Me:
def swap_order(foo):
return lambda x, y: y - x
@swap_order
def subtract(a, b):
return a - b
@swap_order
def add(a, b):
return a + b
Gilch then asked, “What do you expect the add function to do after decoration?” To which I answered that the add function would return the value of its second argument subtracted by the first argument. The next question gilch asked was, “Can you modify the decorator to swap the arguments for both functions?”
I started to think about sys.argv again, then gilch hinted, “You have ‘foo’ as an argument.” I then realized that I could rewrite the return value of the lambda function:
I remarked that we’d see the same result from add with or without the decorator. Gilch asked, “Is addition commutative in Python?” and I immediately responded yes, then I realized that + is an overloaded operator that would work on strings too, and in that case it would not be commutative. We tried with string inputs, and indeed the resulting value was the reverse-ordered arguments concatenated together.
Gilch: Now can you write a decorator that converts its result to a string?
There was some pair debugging that gilch and I did before I reached the answer. Looking at the mistakes I’ve made here, I see that I still hadn’t grasped the idea that decorators would return functions that transform the results of other functions, not the transformed result itself.
Gilch: Try adding a decorator that appends ”, meow.” to the result of the function.
I verbalized the code in my head out loud, then asked how we’d convert the types of the function return value to string before appending ", meow" to it. Gilch suggested f"{foo(x, y)}, meow" and we had our third decorator.
We then applied decorators in different orders to show that multiple decorators were allowed, and that the order of decorators decided the order of application.
Splat
When we were writing the convert_to_str decorator, I commented that this would only work for functions that take in exactly 2 arguments. So gilch asked me if I was familiar with the term “unpacking” or “splat.” I knew it was something like ** but didn’t have more knowledge than that.
How Many Arguments
Gilch asked me, “How many arguments can print() take?” To which I answered “infinite.” They then pointed out that it was different from infinite—zero would be valid, or one, or two, and so on. So the answer is “any number, ” and the next challenge would be to make convert_to_str work with any number of arguments.
print()
We tried passing different numbers of arguments into print(), and sure enough it took any number of arguments. Here, gilch pointed out that print actually printed out a newline character by default, and the default separator was a space. They also pointed out that I could use the help(print) command to access the doc in the terminal without switching to my browser.
type(_)
Gilch pointed out that I could use the command type(_) to get the type of the previous value in the console, without having to copy and paste.
Splat
To illustrate how splat worked, gilch gave me a few commands to try. I’d say out loud what I expected the result to be before I ran the code. Sometimes I got what I expected; sometimes I was surprised by the result, and gilch would point out what I had missed. To illustrate splat in arrays, gilch gave two examples: print(1,2,3,*"spam", sep="~") and print(1,2,*"eggs",3,*"spam", sep="~"). Then they showed me how to use ** to construct a mapping: (lambda **kv: kv)(foo=1, bar=2)
Dictionary vs. Mapping
We went off on a small tangent on dictionary vs. mapping because gilch pointed out that dictionary was not the only type of mapping and tuple is no the only type of iterable. I asked if there were other types of mapping in Python, and they listed OrderedDict as a subtype and the Mapping abstract class.
Parameter vs. Argument, Packing vs. Unpacking
At this point gilch noticed that I kept using the word “unpacking.” I also noticed that I was using the term “argument” and “parameter” interchangeably here. Turns out the distinction is important here—the splat operator used on a parameter packs values in a tuple; used on an argument unpacks iterable into separate values. For example, in (lambda a, b, *cs: [a, b, cs])(1,2,3,4,5), cs is a parameter and * packs the values 3, 4, 5 into a tuple; in print(*"spam", sep="~"), "spam" is an argument and * unpacks it into individual characters.
Dictionaries
Gilch gave me another example: Try {'x':1, **dict(foo=2,bar=3), 'y':4}. I answered that it would return a dictionary with four key-value pairs, with foo and bar also becoming keys. Gilch then asked, “in what order?” To which I answered “dictionaries are not ordered.”
“Not true anymore,” gilch pointed out, “Since Python 3.7, they’re guaranteed to remember their insertion order.” We looked up the Python documentation and it was indeed the case. We tried dict(foo=2, **{'x':1,'y':4}, bar=3) and got a dictionary in a different order.
Hashable Types
I asked if there was any difference in defining a dictionary using {} versus dict(). Gilch compared two examples: {42:'spam'} works and dict(42='spam') doesn’t. They commented that keys could be any hashable type, but keyword arguments were always keyed by identifier strings. The builtin hash() only worked on hashable types.
I don’t fully understand the connection between hashable types and identifier strings here, it’s something that I’ll clarify later.
Parameter vs. Argument, Packing vs. Unpacking
Gilch gave another example: a, b, *cs, z = "spameggs"
I made a guess that cs would be an argument here, so * would be unpacking, but then got stuck on what cs might be. I tried to run it:
>>> a, b, *cs, z = "spameggs"
>>> a
's'
>>> b
'p'
>>> cs
['a', 'm', 'e', 'g', 'g']
>>> z
's'
Gilch pointed out that cs was a store context, not a load context, which made it more like a parameter rather than an argument. Then I asked what store vs. load context was.
Context
Gilch suggested, import ast then def dump(code): return ast.dump(ast.parse(code)). Then something like dump("a = a") would return a nexted object, in which we can locate the ctx value for each variable.
This reminded me of lvalue and rvalue in C++, so I asked if they were the same thing as store vs. load context. They were.
Splat
Gilch tied it all together, “So for a decorator to pass along all args and kwargs, you do something like lambda *args, **kwargs: foo(*args, **kwargs). Then it works regardless of their number. Arguments and keyword arguments in a tuple and dict by keyword. So you can add, remove, and reorder arguments by using decorators to wrap functions. You can also process return values. You can also return something completely different. But wrapping a function in another function is a very common use of decorators. You can also have definition-time side effects. When you first load the module, it runs all the definitions—This is still runtime in Python, but you define a function at a different time than when you call it. The decoration happens on definition, not on call.”
We wrapped up our call at this point.
Observations
As we were working through the examples, we’d voice out what we expect to see when we run the code before actually running to verify. Several times gilch asked me to translate a decorated function into an undecorated one. This was helpful for me to check my understanding of things.
Another thing I found valuable were the tips and tricks I picked up from gilch throughout the session, like interactive mode; and the clarification of concepts, like the distinction between parameter and argument.
Gilch quizzed me throughout the session. This made things super fun! I haven’t had the opportunity for someone to keep quizzing me purely for learning (as opposed to giving me a grade or deciding whether to hire me) for the longest time! I guess that reading through well-written text tends to be effective for familiarizing oneself with concepts, while asking/answering questions is effective at solidifying and synthesizing knowledge.
In this post, I tried to replicate the structure of my conversation with gilch as much as possible (the fact that gilch’s mic was broken so they typed while I talked made writing this post so much easier—I had their half of the transcript generated for me!) since we went off on some tangents and I wanted to provide context for those tangents. I think of a conversation as a tree structure—we start with a root topic and go from there. A branch would happen when we go off on a tangent and then later come back to where we left off before the tangent. Sometimes two sections of this post would have the same section headings; a second time a section heading is used indicates that we stopped the tangent and went back to where we branched off.
An Apprentice Experiment in Python Programming
A couple weeks ago Zvi made an apprentice thread. I have always wanted to be someone’s apprentice, but it didn’t occur to me that I could just …ask to do that. Mainly I was concerned about this being too big of an ask. I saw gilch’s comment offering to mentor Python programming. I want to level up my Python skills, so I took gilch up on the offer. In a separate comment, gilch posed some questions about what mentors and/or the community get in return. I proposed that I, as the mentee, document what I have learned and share it publicly.
Yesterday we had our first session.
Background
I had identified that I wanted to fill gaps in my Python knowledge, two of which being package management and decorators.
Map and Territory
Gilch started by saying that “even senior developers typically have noticeable gaps,” but building an accurate map of the territory of programming would enable one to ask the right questions. They then listed three things to help with that:
Documentation on the Python standard library. “You should at least know what’s in there, even if you don’t know how to use all of it. Skimming the documentation is probably the fastest way to learn that. You should know what all the operators and builtin functions do.”
Structure and Interpretation of Computer Programs for computer science foundation. There are some variants of the book in Python, if one does not want to use Scheme.
CODE as “more of a pop book” on the backgrounds.
In my case, I minored in CS, but did not take operating systems or compilers. I currently work as a junior Python developer, so reading the Python standard library seems to be the lowest hanging fruit here, with SICP on the side, CODE on the back burner.
Decorators
The rest of the conversation consisted of gilch teaching me about decorators.
Gilch: Decorators are syntactic sugar.
means the same thing as
Decorators also work on classes.
is the same as
An Example from
pytestAt this point I asked if decorators were more than that. I had seen decorators in
pytest:Does this mean that, when
foois passed intest_baras a variable, what gets passed in is actually something likepytest.fixture(foo)?Gilch identified that there might be more than decorators involved in this example, so we left this for later and went back to decorators.
Decorators, Example 1
I started sharing my screen, gilch gave me the first instruction: Try making a decorator.
Then, before I ran the program, gilch asked me what I expected to happen when I run this program, to which I answered that
hiand42would be printed to console. At this point, gilch reminded me that decorators were sugar, and asked me to write out the un-sugared translation of the function above. I wrote:I ran the program, and was surprised by the error
TypeError: 'int' object is not callable. I expectedbarto still be a function, not an integer.Gilch asked me to correct my translation of my program based on the result I saw. It took me a few more tries, and eventually they showed me the correct translation:
Then I realized why I was confused—I had the idea that decorators modify the function they decorate directly (in the sense of modifying function definitions), when in fact the actions happen outside of function definitions.
Gilch explained: A decorator could [modify the function], but this one doesn’t. It ignores the function and returns something else. Which it then gives the function’s old name.
Decorators, Example 2
Gilch: Can you make a function that subtracts two numbers?
Me:
Gilch: Now make a decorator that swaps the order of the arguments.
My first thought was to ask if there was any way for us to access function parameters the same way we use
sys.argvto access command line arguments. But gilch steered me away from that path by pointing out that decorators could return anything. I was stuck, so gilch suggested that I tryreturn lambda x, y: y-x.Definition Time
My program looked like this at this point:
PyCharm gave me an warning about referencing
swap_orderbefore defining it. Gilch explained that decoration happened at definition time, which made sense considering the un-sugared version.Interactive Python
Up until this point, I had been running my programs with the command
python3 <file>. Gilch suggested that I runpython3 -i <file>to make it interactive, which made it easier to experiment with things.Decorators, Example 2
Gilch: Now try an add function. Decorate it too.
Me:
Gilch then asked, “What do you expect the add function to do after decoration?” To which I answered that the add function would return the value of its second argument subtracted by the first argument. The next question gilch asked was, “Can you modify the decorator to swap the arguments for both functions?”
I started to think about
sys.argvagain, then gilch hinted, “You have ‘foo’ as an argument.” I then realized that I could rewrite the return value of the lambda function:I remarked that we’d see the same result from
addwith or without the decorator. Gilch asked, “Is addition commutative in Python?” and I immediately responded yes, then I realized that+is an overloaded operator that would work on strings too, and in that case it would not be commutative. We tried with string inputs, and indeed the resulting value was the reverse-ordered arguments concatenated together.Gilch: Now can you write a decorator that converts its result to a string?
I wrote:
It was not right. I then tried
and it was still not right. Finally I got it:
There was some pair debugging that gilch and I did before I reached the answer. Looking at the mistakes I’ve made here, I see that I still hadn’t grasped the idea that decorators would return functions that transform the results of other functions, not the transformed result itself.
Gilch: Try adding a decorator that appends ”, meow.” to the result of the function.
I verbalized the code in my head out loud, then asked how we’d convert the types of the function return value to string before appending
", meow"to it. Gilch suggestedf"{foo(x, y)}, meow"and we had our third decorator.We then applied decorators in different orders to show that multiple decorators were allowed, and that the order of decorators decided the order of application.
Splat
When we were writing the
convert_to_strdecorator, I commented that this would only work for functions that take in exactly 2 arguments. So gilch asked me if I was familiar with the term “unpacking” or “splat.” I knew it was something like**but didn’t have more knowledge than that.How Many Arguments
Gilch asked me, “How many arguments can
print()take?” To which I answered “infinite.” They then pointed out that it was different from infinite—zero would be valid, or one, or two, and so on. So the answer is “any number, ” and the next challenge would be to makeconvert_to_strwork with any number of arguments.print()We tried passing different numbers of arguments into
print(), and sure enough it took any number of arguments. Here, gilch pointed out thatprintactually printed out a newline character by default, and the default separator was a space. They also pointed out that I could use thehelp(print)command to access the doc in the terminal without switching to my browser.type(_)Gilch pointed out that I could use the command
type(_)to get the type of the previous value in the console, without having to copy and paste.Splat
To illustrate how splat worked, gilch gave me a few commands to try. I’d say out loud what I expected the result to be before I ran the code. Sometimes I got what I expected; sometimes I was surprised by the result, and gilch would point out what I had missed. To illustrate splat in arrays, gilch gave two examples:
print(1,2,3,*"spam", sep="~")andprint(1,2,*"eggs",3,*"spam", sep="~"). Then they showed me how to use**to construct a mapping:(lambda **kv: kv)(foo=1, bar=2)Dictionary vs. Mapping
We went off on a small tangent on dictionary vs. mapping because gilch pointed out that dictionary was not the only type of mapping and tuple is no the only type of iterable. I asked if there were other types of mapping in Python, and they listed
OrderedDictas a subtype and theMappingabstract class.Parameter vs. Argument, Packing vs. Unpacking
At this point gilch noticed that I kept using the word “unpacking.” I also noticed that I was using the term “argument” and “parameter” interchangeably here. Turns out the distinction is important here—the splat operator used on a parameter packs values in a tuple; used on an argument unpacks iterable into separate values. For example, in
(lambda a, b, *cs: [a, b, cs])(1,2,3,4,5),csis a parameter and*packs the values3, 4, 5into a tuple; inprint(*"spam", sep="~"),"spam"is an argument and*unpacks it into individual characters.Dictionaries
Gilch gave me another example: Try
{'x':1, **dict(foo=2,bar=3), 'y':4}. I answered that it would return a dictionary with four key-value pairs, withfooandbaralso becoming keys. Gilch then asked, “in what order?” To which I answered “dictionaries are not ordered.”“Not true anymore,” gilch pointed out, “Since Python 3.7, they’re guaranteed to remember their insertion order.” We looked up the Python documentation and it was indeed the case. We tried
dict(foo=2, **{'x':1,'y':4}, bar=3)and got a dictionary in a different order.Hashable Types
I asked if there was any difference in defining a dictionary using
{}versusdict(). Gilch compared two examples:{42:'spam'}works anddict(42='spam')doesn’t. They commented that keys could be any hashable type, but keyword arguments were always keyed by identifier strings. The builtin hash() only worked on hashable types.I don’t fully understand the connection between hashable types and identifier strings here, it’s something that I’ll clarify later.
Parameter vs. Argument, Packing vs. Unpacking
Gilch gave another example:
a, b, *cs, z = "spameggs"I made a guess that
cswould be an argument here, so*would be unpacking, but then got stuck on whatcsmight be. I tried to run it:Gilch pointed out that
cswas a store context, not a load context, which made it more like a parameter rather than an argument. Then I asked what store vs. load context was.Context
Gilch suggested,
import astthendef dump(code): return ast.dump(ast.parse(code)). Then something likedump("a = a")would return a nexted object, in which we can locate thectxvalue for each variable.This reminded me of lvalue and rvalue in C++, so I asked if they were the same thing as store vs. load context. They were.
Splat
Gilch tied it all together, “So for a decorator to pass along all args and kwargs, you do something like
lambda *args, **kwargs: foo(*args, **kwargs). Then it works regardless of their number. Arguments and keyword arguments in a tuple and dict by keyword. So you can add, remove, and reorder arguments by using decorators to wrap functions. You can also process return values. You can also return something completely different. But wrapping a function in another function is a very common use of decorators. You can also have definition-time side effects. When you first load the module, it runs all the definitions—This is still runtime in Python, but you define a function at a different time than when you call it. The decoration happens on definition, not on call.”We wrapped up our call at this point.
Observations
As we were working through the examples, we’d voice out what we expect to see when we run the code before actually running to verify. Several times gilch asked me to translate a decorated function into an undecorated one. This was helpful for me to check my understanding of things.
Another thing I found valuable were the tips and tricks I picked up from gilch throughout the session, like interactive mode; and the clarification of concepts, like the distinction between parameter and argument.
Gilch quizzed me throughout the session. This made things super fun! I haven’t had the opportunity for someone to keep quizzing me purely for learning (as opposed to giving me a grade or deciding whether to hire me) for the longest time! I guess that reading through well-written text tends to be effective for familiarizing oneself with concepts, while asking/answering questions is effective at solidifying and synthesizing knowledge.
In this post, I tried to replicate the structure of my conversation with gilch as much as possible (the fact that gilch’s mic was broken so they typed while I talked made writing this post so much easier—I had their half of the transcript generated for me!) since we went off on some tangents and I wanted to provide context for those tangents. I think of a conversation as a tree structure—we start with a root topic and go from there. A branch would happen when we go off on a tangent and then later come back to where we left off before the tangent. Sometimes two sections of this post would have the same section headings; a second time a section heading is used indicates that we stopped the tangent and went back to where we branched off.