diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index ad3750cabbfe7e525439368e83b60fd6d1996fbd..f7241f842fcf12b95e7b2de4d18b858066e13ad1 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -32,6 +32,7 @@ cache: source venv/bin/activate pip install -U pip setuptools make install + make dependencies test: diff --git a/Makefile b/Makefile index 7fa617367dc4c586634e165f55fdc8404c19e3f6..b914a01c7069b92cb727e87a55fb8c6d4cd35d5c 100644 --- a/Makefile +++ b/Makefile @@ -2,17 +2,11 @@ PKGNAME=orcasong ALLNAMES = $(PKGNAME) ALLNAMES += orcasong_contrib -default: build - -all: install - -build: - @echo "No need to build anymore :)" install: pip install . -install-dev: +install-dev: dependencies pip install -e . clean: @@ -38,6 +32,7 @@ lint: dependencies: pip install -Ur requirements.txt + pip install -Ur requirements_dev.txt .PHONY: yapf yapf: diff --git a/docs/orcasong.rst b/docs/orcasong.rst index 79d0dd402e901537736928f9b9855033d4816f6d..e52cbd43ebee5a9b7cd96db32cb2a0f495aa015c 100644 --- a/docs/orcasong.rst +++ b/docs/orcasong.rst @@ -1,15 +1,25 @@ .. _orcasong_page: +Producing DL files from h5 +========================== + +Describes how to use OrcaSong to produce files for Deep Learning +from h5 files. These files can contain either images (for convolutional +networks), or graphs (for Graph networks). + +.. contents:: :local: + + Mode 1: Producing images -======================== +------------------------ -Generate multidimensional images out of ORCA data. +Generate multidimensional images out of km3net data. .. image:: imgs/orcasong_function.PNG :height: 400px Basic Use ---------- +^^^^^^^^^ Import the main class, the FileBinner (see :py:class:`orcasong.core.FileBinner`), @@ -55,7 +65,7 @@ Or convert multiple files, which will all be saved in the given folder: Plotting binning statistics ---------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^ After the binning has succeeded, you can generate a plot which shows the distribution of hits among the bins you defined. For this, call the following @@ -67,7 +77,7 @@ This will plot the statistics for the files file_1_binned.h5, file_2_binned.h5, into the file my_plotname.pdf. Using existing binnings ------------------------ +^^^^^^^^^^^^^^^^^^^^^^^ You can use existing bin edges and mc info extractors from ``orcasong.bin_edges`` and ``orcasong.mc_info_extr``. These were designed for specific detector layouts @@ -75,12 +85,12 @@ and productions, and might not work properly when used on other data. Mode 2: Producing Graphs -======================== +------------------------ -Generate the nodes of graphs from ORCA data. +Generate the nodes of graphs from km3net data. Basic Use ---------- +^^^^^^^^^ Import the main class, the FileGraph (see :py:class:`orcasong.core.FileGraph`), @@ -102,12 +112,12 @@ of FileGraph determines this fixed length: General usage -============= +------------- Functionality that both modes have in common. Calibration ------------ +^^^^^^^^^^^ You can supply a detx file to the file binner, in order to calibrate the data on the fly: @@ -118,11 +128,12 @@ calibrate the data on the fly: Adding mc_info --------------- +^^^^^^^^^^^^^^ -To add info from the mc_tracks (or from anywhere in the blob), you can define some -function ``my_mcinfo_extractor`` which takes as an input a km3pipe blob, +Define a function ``my_mcinfo_extractor``, which takes as an input a km3pipe blob, and outputs a dict mapping str to float. +It should contain everything you need later down the pipeline, e.g. labels, +event identifiers, ... This will be saved as a numpy structured array "y" in the output file, with the str being the dtype names. Set up like follows: diff --git a/requirements.txt b/requirements.txt index 3b6f7c039aef284af9e823cd08fd484b8e1a09e5..33a42788a3157edc011cd5e8e0f104f8a4789c6f 100644 --- a/requirements.txt +++ b/requirements.txt @@ -2,11 +2,4 @@ numpy h5py matplotlib km3pipe -docopt -toml setuptools_scm -sphinx -sphinx-rtd-theme -sphinx-autoapi -twine -numpydoc==0.9.2 diff --git a/requirements_dev.txt b/requirements_dev.txt new file mode 100644 index 0000000000000000000000000000000000000000..6fb2818ff100cb027f96d098efeb34af86032a57 --- /dev/null +++ b/requirements_dev.txt @@ -0,0 +1,5 @@ +sphinx +sphinx-rtd-theme +sphinx-autoapi +twine +numpydoc==0.9.2