Re-configuring my Python development workflow

I have been working on re-configuring my development workflow this week.

So my main (or to say only) programming language is Python, I have used uv widely across my projects¹. However, I found I consistently have to keep creating new projects with manual setup of repository structure. uv does help a bit to speed up thing (like using uv init, uv add and uv run), but I still have to manually create the project structure and files. For example, if I want to start a data project, then I need type in (at least) the following commands:

gh repo create my-data-project --private
gh repo clone my-data-project
cd my-data-project
uv init
mkdir data src tests config
curl -o .gitignore https://raw.githubusercontent.com/github/gitignore/main/Python.gitignore >> .gitignore
uv add --dev ipykernel pandas httpx2
touch .playground.ipynb
# Manually create the .pre-commit-config.yaml and paste the content as instructed in the [commitizen](https://commitizen-tools.github.io/commitizen/) documentation
prek install --hook-type commit-msg --hook-type pre-push
# Then copy the powerful CLAUDE.md by multica-ai/andrej-karpathy-skills to the project root
... (other steps I might have already forgotten)

This really spent me a lot of time, doing it manually and inconsistently.

Eventually I found out copier that fits what I need.

Copier

In a nutshell copier is for generating projects from templates. It allows you to populate a project structure base on a CLI questionnaire. The questionnaire is defined in a copier.yml file. The tool then uses jinja2 to populate the files and directories based on the answers you provide.

Practically this is how it works:

1. First create a project directory on the local machine

mkdir my-data-project

2. Then, run the following command to initialize the project with copier using uvx:

uvx copier copy gh:${YOUR_TEMPLATE_REPO} my-data-project

3. Then answer the questions in the CLI questionnaire.

That’s it! The project structure and files will be populated base on the template.

Things I like about copier

1. It supports template updates, that allow you to update the project structure with the same template. This is especially helpful as I in a discovery stage right now, that I might want to edit my configuration over time. What I need to do is just run copier update in the project directory, and answer the questions again. The files will be updated base on the new answers.

2. The project is actively maintained, good documentation, and has a very active community. You can find the templates with the copier-template tag on GitHub.

3. It allows reference a specific branch or commit in a template (GitHub) repository, which is very useful for testing out the template before using it in production.

My own copier template

I kept configuring my own copier template this week. You can check it in kenlhlui/substrate.

My template was developed base on superlinear-ai/substrate. This template naturally fits my existing (manual) workflow and setup, with the choice of using uv, ruff, commitizen etc.

What I have modified is mainly:

1. Replaced pre-commit with prek as prek is much faster, and backward compatible with pre-commit hooks.
2. Replaced MkDocs with Zensical as the later is the de facto successor and under more active development.
3. The option to create data and config directories, that is useful for data projects.
4. Added httpx2 and pydantic as a default installed dependency, as I use them in almost all my projects.

Other useful resources

I was researching about how Poe the Poet works, right before I decided to write this blog post. I discovered this really wonderful handbook, pydevtools.com, that is that kind of ‘oh I wish I knew this earlier’ kind of resource. The site has some really precise and concise style guides that summarize how should a workflow being configured. I also find the site cover topics that helps with understanding the Python eco-system, such as PEPs, core concepts (lockfiles, .python-version file, Conda, etc.). Highly recommended!