.. include:: ../../substitutions .. _dependencies: Managing dependencies ===================== .. contents:: :local: :backlinks: none What is Poetry? --------------- Poetry is "Python packaging and dependency management made easy." It is an umbrella project whose components allow us to specify Python dependencies, recreate Python environments in a mostly deterministic way, build the project and publish it too. What are the pyproject.toml and poetry.lock files? ------------------------------------------------------ The `pyproject.toml` file was standardised in several PEPs and is a centralised location to store configuration for various Python related tooling. In particular, `[tool.poetry.*]` sections in `pyproject.toml` store Poetry related configuration, such as this project's run time dependencies and more. Poetry reads the project's run time dependency constraints from `pyproject.toml` and stores *one* solution to these constraints in `poetry.lock`. We track known good `pyproject.toml` and `poetry.lock` versions in the repository. .. warning:: After you make any changes to Poetry sections in `pyproject.toml` or `poetry.lock`, you and your coworkers need to run .. parsed-literal:: poetry install \ |poetry_groups| How do I reset my environment to a known good configuration? ------------------------------------------------------------ Run .. parsed-literal:: git restore pyproject.toml poetry.lock poetry install \ |poetry_groups| The above command ensures that your environment includes every package as recorded in `poetry.lock`. Depending on your previous commands, you may still have additional packages. To ensure that your environment includes *exactly* the packages recorded in `poetry.lock`, run .. parsed-literal:: git restore pyproject.toml poetry.lock # note the additional --sync poetry install --sync \ |poetry_groups| How do I add new dependencies? ------------------------------ To add run time dependencies, run .. code-block:: bash poetry add dependency-a dependency-b git add pyproject.toml poetry.lock To add development dependencies, figure out the appropriate Poetry group by reading `pyproject.toml`, then run .. code-block:: bash poetry add --group groupname dependency-a dependency-b git add pyproject.toml poetry.lock For example, the "docs" Poetry group looks approximately like this .. code-block:: toml [tool.poetry.group.docs] # "docs" is the name of the group optional = true [tool.poetry.group.docs.dependencies] sphinx-copybutton = {version = "^0.5.0"} sphinxcontrib-fulltoc = {version = "^1.2.0"} Sphinx = {version = "^4.0"} numpydoc = {version = "^1.2"} and to add `new-dependency` to this group, run .. code-block:: bash poetry add --group docs new-dependency git add pyproject.toml poetry.lock For more information, consult the `poetry add documentation `__. How do I update existing dependencies? -------------------------------------- To update dependencies to versions *within* the constraints listed in `pyproject.toml`, run .. code-block:: bash poetry update git add pyproject.toml poetry.lock or selectively .. code-block:: bash poetry update dependency-a dependency-b git add pyproject.toml poetry.lock For more information, consult the `poetry update documentation `__. To update dependencies to the latest available version *outside* the constraints listed in `pyproject.toml`, .. code-block:: bash poetry add existingdependency@latest git add pyproject.toml poetry.lock For more information, consult the `poetry add documentation `__. How do I remove dependencies? ----------------------------- To remove a dependency from the run time dependencies, run .. code-block:: bash poetry remove dependency git add pyproject.toml poetry.lock and to remove a dependency from a Poetry group, run .. code-block:: bash poetry remove --group groupname dependency git add pyproject.toml poetry.lock See also the `poetry remove documentation `__. What are the differences between poetry lock, update, install and add? ---------------------------------------------------------------------- The `poetry lock `__ command *solves* the constraints listed in `pyproject.toml` and writes a solution to the `poetry.lock` file. The `poetry install `__ command installs the *resolved* dependencies as listed in `poetry.lock` into a virtual environment. If `poetry.lock` doesn't exist, `poetry install` will run `poetry lock` by itself to create `poetry.lock`. The `poetry update `__ command is equivalent to deleting `poetry.lock` and running `poetry install`, see `here `__. The `poetry lock`, `poetry update` and `poetry install` commands will never change `pyproject.toml`, but they may change `poetry.lock`. The `poetry add` command may change `pyproject.toml` and `poetry.lock`. For more details, it is recommended to read at least `this section of the Poetry documentation `__. Can I edit pyproject.toml manually? What about poetry.lock? ----------------------------------------------------------- The `pyproject.toml` file can be (but doesn't have to be) manually edited. Afterwards run .. code-block:: bash poetry lock git add pyproject.toml poetry.lock to ensure that the `poetry.lock` file is a solution to the dependency constraints in `pyproject.toml`. Don't edit `poetry.lock` manually. The syntax for version ranges is specified `here `__. For more information, consult the `poetry lock documentation `__. What is the difference between Poetry groups and extras? -------------------------------------------------------- `Poetry groups `__ are specific to Poetry and cannot be used by `pip install`. We take advantage of Poetry groups to selectively install developer dependencies in CI. Downstream users cannot install any of these Poetry groups directly. `Extras `__ can be used downstream, for example `pip install dependency[extra1,extra2]`. We don't currently use extras. Poetry's syntax for extras is `here `__. Does pip install use the poetry.lock file? ------------------------------------------ No, it doesn't. It does respect the constraints listed in the `[tool.poetry.dependencies]` in `pyproject.toml` though. The `poetry.lock` file is only relevant when running `poetry install` or other Poetry commands within the project. Its purpose is to ensure *one consistent environment among developers*, but it wouldn't make sense to prescribe specific versions to downstream users. What is the build-system section in pyproject.toml? --------------------------------------------------- Specifically, this section .. code-block:: toml [build-system] requires = ["poetry-core>=1.0.0"] build-backend = "poetry.core.masonry.api" This section is defined in `PEP 517 `__ and `PEP 518 `__. It tells other tools trying to build the project (e.g. `pip install pyddg`) that the only *build time* dependency is `Poetry core `__ and that the build process should be handed off to Poetry core. Note that *build time* and *run time* dependencies are completely independent, so you likely don't want to touch this section at all.