Let’s call two broad clusters of problem-solving “systemizing” and “hacking.” Systemizing roughly refers to building general, robust solutions. Hacking refers to quickly building one-offs. You would expect to accrue technical debt while hacking and pay it down while systemizing.
Both are useful in different situations. A big win for systemizing would be finding patterns in code and dramatically reducing the complexity by factoring out those patterns. A big win for hacking would be quickly testing several UIs to find one that potential customers love.
As you can probably guess, I lean very heavily towards systemizing, and have typically sucked at hacking – I generally relied on coworkers for that skill. But I started working on solo projects where I couldn’t easily rely on someone else, so naturally, I developed a system for hacking things together.
The concrete example here is that I wanted to be able to run thousands of completely independent machine learning jobs quickly in parallel across a lot of computers, and I was looking into AWS Batch to do this.
Here’s how I would have approached this in “systemizing” mode:
Break down all the parts involved, e.g. AWS Batch, Docker, boto3, etc.
Study them in detail until I grok each one, writing small bits of code primarily designed to help me understand
Assemble a solution
Here’s how I did this in hacking mode:
Find a valuable point of high uncertainty (e.g., running Python scripts through Docker)
Find something that does something similar to what I want (e.g., an online introduction to Docker that involved running a particular Python script)
Modify that until it does what I want (successfully running my Python script and uploading data to AWS)
Find the next valuable point of uncertainty and repeat (running a single Docker image on AWS Batch)
This was really effective – previously, I might have spent ~40 hours deeply learning all the tools involved before I got the system up and running. Using the hacking approach, it only took me about 3 hours – literally an order of magnitude faster.
Of course, the hacking approach has clear disadvantages – I don’t deeply understand Docker or any of the tools involved, and my experience here won’t save me a ton of time if I work on a similar project. But in this case, I don’t expect to do anything similar in the near-term, so hacking was a huge win. More generally, having the skill of hacking things together gives me many more options for projects in the future.
Systemizing and Hacking
Let’s call two broad clusters of problem-solving “systemizing” and “hacking.” Systemizing roughly refers to building general, robust solutions. Hacking refers to quickly building one-offs. You would expect to accrue technical debt while hacking and pay it down while systemizing.
Both are useful in different situations. A big win for systemizing would be finding patterns in code and dramatically reducing the complexity by factoring out those patterns. A big win for hacking would be quickly testing several UIs to find one that potential customers love.
As you can probably guess, I lean very heavily towards systemizing, and have typically sucked at hacking – I generally relied on coworkers for that skill. But I started working on solo projects where I couldn’t easily rely on someone else, so naturally, I developed a system for hacking things together.
The concrete example here is that I wanted to be able to run thousands of completely independent machine learning jobs quickly in parallel across a lot of computers, and I was looking into AWS Batch to do this.
Here’s how I would have approached this in “systemizing” mode:
Break down all the parts involved, e.g. AWS Batch, Docker, boto3, etc.
Study them in detail until I grok each one, writing small bits of code primarily designed to help me understand
Assemble a solution
Here’s how I did this in hacking mode:
Find a valuable point of high uncertainty (e.g., running Python scripts through Docker)
Find something that does something similar to what I want (e.g., an online introduction to Docker that involved running a particular Python script)
Modify that until it does what I want (successfully running my Python script and uploading data to AWS)
Find the next valuable point of uncertainty and repeat (running a single Docker image on AWS Batch)
This was really effective – previously, I might have spent ~40 hours deeply learning all the tools involved before I got the system up and running. Using the hacking approach, it only took me about 3 hours – literally an order of magnitude faster.
Of course, the hacking approach has clear disadvantages – I don’t deeply understand Docker or any of the tools involved, and my experience here won’t save me a ton of time if I work on a similar project. But in this case, I don’t expect to do anything similar in the near-term, so hacking was a huge win. More generally, having the skill of hacking things together gives me many more options for projects in the future.