Managing dependencies

What is Poetry?

Poetry is “Python packaging and dependency management made easy.” It is an umbrella project whose components allow us to specify Python dependencies, recreate Python environments in a mostly deterministic way, build the project and publish it too.

What are the pyproject.toml and poetry.lock files?

The pyproject.toml file was standardised in several PEPs and is a centralised location to store configuration for various Python related tooling.

In particular, [tool.poetry.*] sections in pyproject.toml store Poetry related configuration, such as this project’s run time dependencies and more.

Poetry reads the project’s run time dependency constraints from pyproject.toml and stores one solution to these constraints in poetry.lock. We track known good pyproject.toml and poetry.lock versions in the repository.

Warning

After you make any changes to Poetry sections in pyproject.toml or poetry.lock, you and your coworkers need to run

poetry install --with dev,docs,tests,style,codequality

How do I reset my environment to a known good configuration?

Run

git restore pyproject.toml poetry.lock
poetry install --with dev,docs,tests,style,codequality

The above command ensures that your environment includes every package as recorded in poetry.lock. Depending on your previous commands, you may still have additional packages. To ensure that your environment includes exactly the packages recorded in poetry.lock, run

git restore pyproject.toml poetry.lock
# note the additional --sync
poetry install --sync --with dev,docs,tests,style,codequality

How do I add new dependencies?

To add run time dependencies, run

poetry add dependency-a dependency-b
git add pyproject.toml poetry.lock

To add development dependencies, figure out the appropriate Poetry group by reading pyproject.toml, then run

poetry add --group groupname dependency-a dependency-b
git add pyproject.toml poetry.lock

For example, the “docs” Poetry group looks approximately like this

[tool.poetry.group.docs]  # "docs" is the name of the group
optional = true

[tool.poetry.group.docs.dependencies]
sphinx-copybutton = {version = "^0.5.0"}
sphinxcontrib-fulltoc = {version = "^1.2.0"}
Sphinx = {version = "^4.0"}
numpydoc = {version = "^1.2"}

and to add new-dependency to this group, run

poetry add --group docs new-dependency
git add pyproject.toml poetry.lock

For more information, consult the poetry add documentation.

How do I update existing dependencies?

To update dependencies to versions within the constraints listed in pyproject.toml, run

poetry update
git add pyproject.toml poetry.lock

or selectively

poetry update dependency-a dependency-b
git add pyproject.toml poetry.lock

For more information, consult the poetry update documentation.

To update dependencies to the latest available version outside the constraints listed in pyproject.toml,

poetry add existingdependency@latest
git add pyproject.toml poetry.lock

For more information, consult the poetry add documentation.

How do I remove dependencies?

To remove a dependency from the run time dependencies, run

poetry remove dependency
git add pyproject.toml poetry.lock

and to remove a dependency from a Poetry group, run

poetry remove --group groupname dependency
git add pyproject.toml poetry.lock

See also the poetry remove documentation.

What are the differences between poetry lock, update, install and add?

The poetry lock command solves the constraints listed in pyproject.toml and writes a solution to the poetry.lock file.

The poetry install command installs the resolved dependencies as listed in poetry.lock into a virtual environment. If poetry.lock doesn’t exist, poetry install will run poetry lock by itself to create poetry.lock.

The poetry update command is equivalent to deleting poetry.lock and running poetry install, see here.

The poetry lock, poetry update and poetry install commands will never change pyproject.toml, but they may change poetry.lock. The poetry add command may change pyproject.toml and poetry.lock.

For more details, it is recommended to read at least this section of the Poetry documentation.

Can I edit pyproject.toml manually? What about poetry.lock?

The pyproject.toml file can be (but doesn’t have to be) manually edited. Afterwards run

poetry lock
git add pyproject.toml poetry.lock

to ensure that the poetry.lock file is a solution to the dependency constraints in pyproject.toml. Don’t edit poetry.lock manually. The syntax for version ranges is specified here.

For more information, consult the poetry lock documentation.

What is the difference between Poetry groups and extras?

Poetry groups are specific to Poetry and cannot be used by pip install. We take advantage of Poetry groups to selectively install developer dependencies in CI. Downstream users cannot install any of these Poetry groups directly.

Extras can be used downstream, for example pip install dependency[extra1,extra2]. We don’t currently use extras. Poetry’s syntax for extras is here.

Does pip install use the poetry.lock file?

No, it doesn’t. It does respect the constraints listed in the [tool.poetry.dependencies] in pyproject.toml though. The poetry.lock file is only relevant when running poetry install or other Poetry commands within the project. Its purpose is to ensure one consistent environment among developers, but it wouldn’t make sense to prescribe specific versions to downstream users.

What is the build-system section in pyproject.toml?

Specifically, this section

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

This section is defined in PEP 517 and PEP 518. It tells other tools trying to build the project (e.g. pip install pyddg) that the only build time dependency is Poetry core and that the build process should be handed off to Poetry core. Note that build time and run time dependencies are completely independent, so you likely don’t want to touch this section at all.