Newer
Older
Title: How to develop software
Author: Tamas
Topics:
- using the python template
- software development recommendations
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
# How to Develop Software
KM3NeT defines guidelines and recommendations for software development which
help to maintain a consistent project structure and development process.
## Continuous Integration
Each project on Git has to implement at least a basic continuous integration
(CI) routine based on GitLab CI, which is stored in a single YAML file in the
root of the project's repository. This file contains workflows to automate a set
of jobs whenever the project is updated, some of them are mandatory, others are
optional. The mandatory CI jobs to automate are:
- *Compilation**: the most basic requirement is that the software
actually compiles on at least one of the target systems. In case of KM3NeT,
the main target system as of writing this document is
Scientific Linux 7, which is used on most of the HPC environments.
- Running the *test suite*: the test suite consists of testing routines which
target specific parts of the software and make sure that they are working as
expected. Tests are categorised into different levels: unit-tests
(function-level), integration tests (interface between components), system
tests (complete integration of the software) and acceptance tests (define the
readiness of a product)
- *Installation*: the actual installation routine as described in the user guide
- *Documentation*: generating the documentation including the API description
Optional but highly recommended CI jobs to automate are:
- *Benchmarks*: to measure the actual performance and potential regressions
- Dynamically created *Tutorials*: comprehensive guidelines which describe how
perform specific tasks with the software. The code examples must not be
static but dynamically executed. This makes sure that the tutorial is
up-to-date and can be executed flawlessly by the user.
- *Publishing*: this includes the creation of Docker and Singularity images, as
well as other publishing routines like e.g. uploading a release to PyPI
## Development Process
The main development branch is the `master` branch on Git and is meant to refer
to the latest stable version of the software. New features, experimental
implementations and bug fixing is done on separate branches and eventually
merged back to `master` after a code review and approvals of at least two
additional developers. The CI makes sure that the software is working as
expected and also indicates when the code coverage -- the fraction of untested
parts of the software -- is decreased upon a merge.
A separate job in the pipeline is set up to check the coding style automatically
and ensures a common style among the whole project.
To keep track of important additions, changes, bug fixes and deprecations, a
`CHANGELOG` is available and updated accordingly. This files serves as an
overview for the users to keep track of the development process without spending
too much time to browse through closed issues and merge requests.
The versioning follow the Semantic Versioning 2.0.0 (https://semver.org)
conventions which indicates possible complications in up- or downgrading. The
version number consists of three parts in form of `MAJOR.MINOR.PATCH`, where
`MAJOR` is increased on incompatible API changes, `MINOR` when new functionality
is added in a backwards compatible manner and `PATCH` for anything else, like
bug fixes or cosmetics etc.
## Python Project Template
The most commonly used programming language in KM3NeT is Python. A Python
project template has been defined by KM3NeT and is based on the `cookiecutter`
(https://cookiecutter.readthedocs.io/) template framework. The template is
publicly available under https://git.km3net.de/templates/python-project and is
specifically designed to fit the KM3NeT GitLab CI environment. It includes a
skeleton Python project which will be populated with the meta information
obtained during the template creation process (project name, description,
authors, Git project URL, etc.) and features out-of-the-box
- a clear project folder structure
- a testing suite
- automatic documentation generation
- API documentation
- PyPI compliant setup
- a `Makefile` with commonly used routines (running the test suite, checking the
code-style, creating a Python virtual environment...).