From 65029d15f9268d493be17956434f7240dc89ebc7 Mon Sep 17 00:00:00 2001 From: Stefan Reck <stefan.reck@fau.de> Date: Tue, 13 Oct 2020 15:33:49 +0200 Subject: [PATCH] minor change to setup, reqs and doc. --- .gitlab-ci.yml | 1 + Makefile | 9 ++------- docs/orcasong.rst | 37 ++++++++++++++++++++++++------------- requirements.txt | 7 ------- requirements_dev.txt | 5 +++++ 5 files changed, 32 insertions(+), 27 deletions(-) create mode 100644 requirements_dev.txt diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index ad3750c..f7241f8 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -32,6 +32,7 @@ cache: source venv/bin/activate pip install -U pip setuptools make install + make dependencies test: diff --git a/Makefile b/Makefile index 7fa6173..b914a01 100644 --- a/Makefile +++ b/Makefile @@ -2,17 +2,11 @@ PKGNAME=orcasong ALLNAMES = $(PKGNAME) ALLNAMES += orcasong_contrib -default: build - -all: install - -build: - @echo "No need to build anymore :)" install: pip install . -install-dev: +install-dev: dependencies pip install -e . clean: @@ -38,6 +32,7 @@ lint: dependencies: pip install -Ur requirements.txt + pip install -Ur requirements_dev.txt .PHONY: yapf yapf: diff --git a/docs/orcasong.rst b/docs/orcasong.rst index 79d0dd4..e52cbd4 100644 --- a/docs/orcasong.rst +++ b/docs/orcasong.rst @@ -1,15 +1,25 @@ .. _orcasong_page: +Producing DL files from h5 +========================== + +Describes how to use OrcaSong to produce files for Deep Learning +from h5 files. These files can contain either images (for convolutional +networks), or graphs (for Graph networks). + +.. contents:: :local: + + Mode 1: Producing images -======================== +------------------------ -Generate multidimensional images out of ORCA data. +Generate multidimensional images out of km3net data. .. image:: imgs/orcasong_function.PNG :height: 400px Basic Use ---------- +^^^^^^^^^ Import the main class, the FileBinner (see :py:class:`orcasong.core.FileBinner`), @@ -55,7 +65,7 @@ Or convert multiple files, which will all be saved in the given folder: Plotting binning statistics ---------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^ After the binning has succeeded, you can generate a plot which shows the distribution of hits among the bins you defined. For this, call the following @@ -67,7 +77,7 @@ This will plot the statistics for the files file_1_binned.h5, file_2_binned.h5, into the file my_plotname.pdf. Using existing binnings ------------------------ +^^^^^^^^^^^^^^^^^^^^^^^ You can use existing bin edges and mc info extractors from ``orcasong.bin_edges`` and ``orcasong.mc_info_extr``. These were designed for specific detector layouts @@ -75,12 +85,12 @@ and productions, and might not work properly when used on other data. Mode 2: Producing Graphs -======================== +------------------------ -Generate the nodes of graphs from ORCA data. +Generate the nodes of graphs from km3net data. Basic Use ---------- +^^^^^^^^^ Import the main class, the FileGraph (see :py:class:`orcasong.core.FileGraph`), @@ -102,12 +112,12 @@ of FileGraph determines this fixed length: General usage -============= +------------- Functionality that both modes have in common. Calibration ------------ +^^^^^^^^^^^ You can supply a detx file to the file binner, in order to calibrate the data on the fly: @@ -118,11 +128,12 @@ calibrate the data on the fly: Adding mc_info --------------- +^^^^^^^^^^^^^^ -To add info from the mc_tracks (or from anywhere in the blob), you can define some -function ``my_mcinfo_extractor`` which takes as an input a km3pipe blob, +Define a function ``my_mcinfo_extractor``, which takes as an input a km3pipe blob, and outputs a dict mapping str to float. +It should contain everything you need later down the pipeline, e.g. labels, +event identifiers, ... This will be saved as a numpy structured array "y" in the output file, with the str being the dtype names. Set up like follows: diff --git a/requirements.txt b/requirements.txt index 3b6f7c0..33a4278 100644 --- a/requirements.txt +++ b/requirements.txt @@ -2,11 +2,4 @@ numpy h5py matplotlib km3pipe -docopt -toml setuptools_scm -sphinx -sphinx-rtd-theme -sphinx-autoapi -twine -numpydoc==0.9.2 diff --git a/requirements_dev.txt b/requirements_dev.txt new file mode 100644 index 0000000..6fb2818 --- /dev/null +++ b/requirements_dev.txt @@ -0,0 +1,5 @@ +sphinx +sphinx-rtd-theme +sphinx-autoapi +twine +numpydoc==0.9.2 -- GitLab