more manuals

6cbd753f · Jutta Schnabel · 42664731 · 6cbd753f · 6cbd753f · 6cbd753f
Commit 6cbd753f authored 4 years ago by Jutta Schnabel
--- a/filelist_D4-9.txt
+++ b/filelist_D4-9.txt
 pages/Manuals.md
 pages/Softwaredevelopment.md
 pages/Using_templates.md
-pages/Using_vo.md
-pages/Using_courses.md
-pages/Manual_providing_data.md
+pages/Maintencance.md
+pages/Using_courses.md
\ No newline at end of file
--- a/index.rst
+++ b/index.rst
@@ -36,9 +36,8 @@ Welcome to Open Science @KM3NeT!
   pages/Manuals
   pages/Softwaredevelopment
   pages/Using_templates
-   pages/Using_vo
+   pages/Maintenance
   pages/Using_courses
-   pages/Manual_providing_data

   pages/statuspage


--- a/pages/Using_vo.md
+++ b/pages/Using_vo.md
--- a/pages/Manual_providing_data.md
+++ b/pages/Manual_providing_data.md
---
-Title: How to read hdf5 data
-Author: Steffen, Jutta
-Topics:
-  - describe download
-  - notebook
-status: unedited
-Comment: |
-  To create such a page from the notebook, 
-  export the jupyter notebook as markdown 
-  and add to this text file and store the notebook 
-  in the notebooks/ folder
---
-
-
-## Data from the ODC
--- a/pages/Manuals.md
+++ b/pages/Manuals.md
@@ -5,14 +5,16 @@ status: dump
 ---

 # Introduction
-There are two sides to the manuals for the KM3NeT open science system. On the one hand manuals are needed for the external user to understand the use of the data and the interfaces, on the other hand manuals are needed for KM3NeT members to actually contribute their software and data to the open science system. 
+There are two sides to the manuals for the KM3NeT open science system. On the one hand manuals are needed for the external user to understand the use of the data and the interfaces, on the other hand manuals have to be provided for KM3NeT members to actually contribute their software and data to the open science system. 

-However, as documentation to an evolving system as required for this deliverable is valid only for a short time, this document aims to give an overview over the different sources of documentation and point to the actual currently valid version. Only where documentation is not made available as versioned web resource or is in its content central to the architecture, the actual manual is copied or appended here.
+However, as documentation to an evolving system as required for this deliverable is valid only for a short time, this document aims to give an overview over the different sources of documentation and point to the actual currently valid version, not provide the full content of the references. Only where documentation is not made available as versioned web resource or is in its content central to the architecture, the actual manual is copied or appended here.

-Note that most documentation for software is built automatically for source documentation and human-created documenation content is for most part hosted on the KM3NeT gitlab instance, as described in Deliverable 4.8. As the KM3NeT solution builds heavily on third-party software, documentation of this software is available elsewhere and pointed out here.
+Note that most documentation for software is built automatically for source documentation and human-created documentation content is for most part hosted on the KM3NeT Gitlab instance, as described in Deliverable 4.8. As the KM3NeT solution builds heavily on third-party software, documentation of this software is available elsewhere and pointed to here.

 # Manuals for users of the KM3NeT open science system

+For further description of the individual components, see Deliverable 4.8 or the [Open Science Portal (OSP)](https://open-data.pages.km3net.de/openserver/).
+
 ## Manuals for the servers and platforms

 ### Summary
@@ -49,7 +51,7 @@ For the KM3NeT Open Data Center, all documentation (apart from the software fram

 Software documentation follows the publication guidelines for software as outlined in Deliverable 4.8. There, core documentation is required from autogenerated API documentation tools and an installation guide. Beyond this, the following additional documenation is provided by the listed packages, which are included in this list as they have been used in the open data example analysis. Additional software is available from the KM3NeT gitlab instance.

-### Summary
+### KM3NeT software packages

 | Software | documentation URL | description | content |
 | -------- | ----------------- | ----------- | ------- |
@@ -58,13 +60,47 @@ Software documentation follows the publication guidelines for software as outlin
 | km3pipe | https://km3py.pages.km3net.de/km3pipe/ | data processing framework | full docs |
 | kmeta | https://open-data.pages.km3net.de/kmeta/ | metadata management | user guide |

+### External software
+
+#### Using VO clients
+Information on how to download and install Aladin is found at https://aladin.u-strasbg.fr/java/nph-aladin.pl?frame=downloading.
+
+An introduction on how to use ADQL is found [here](http://docs.g-vo.org/adql/html/)
+
+  * Aladin: https://aladin.u-strasbg.fr/java/nph-aladin.pl?frame=downloading .
+  * TOPCAT: http://www.star.bris.ac.uk/~mbt/topcat/#install .
+
+#### Using the VO python interface
+You can find information on how to install Astropy at https://www.astropy.org/ and for the pyVO at https://pyvo.readthedocs.io/en/latest/
+
+Accessing data with Python:
+  * Astropy: https://www.astropy.org/ .
+  * PyVO: https://pyvo.readthedocs.io/en/latest/ .
+  * Matplotlib: https://matplotlib.org/3.1.1/users/installing.html .
+
+
 ## Integration examples and workflows

-Workflows are made accessible as Jupyter notebooks, which serve to show the integration between the data and software components. All example notebooks can be downloaded from the Open Science Portal as package.
+### Notebooks
+Workflows are made accessible as Jupyter notebooks, which serve to show the integration between the data and software components. All example notebooks can be downloaded from the Open Science Portal as package. The notebooks contain in-file documentation as one of their core features is the combination of code blocks and comment blocks which allow rich annotation of the executed code. Also, they demonstrate the interaction between the various platforms and the application of the software for a real-life example.

 | Name | File name | description | components |
 | -------- | ----------------- | ----------- | ------- |
-| 
+| 1 Week of ORCA-4 | A01_recorded_rate.ipynb | data taking stability, local coordinates | KM3NeT demo dataset (ODC), openkm3 |
+| Quality parameters and event selection | A02_quality_parameters_event_selection.ipynb | parameter displays, selecting events of interest | KM3NeT demo dataset (ODC), openkm3 |
+| Coordinate transformations | A03_events_on_the_sky.ipynb | sky coordinates of the events | KM3NeT demo dataset (ODC), openkm3, km3astro |
+| Gravitational wave follow-up | A04_gravitational_wave_followup.ipynb | search for coincidences between GW alert and ORCA neutrinos | KM3NeT demo dataset (ODC), openkm3, km3astro, ligo-gracedb |
+| ANTARES point source search | ANTARES_PointSource.ipynb | Simple point source search from ANTARES 2007-2017 data | ANTARES 2007-17 dataset (VO), pyvo, sensitivity & background-pdf (ODC), openkm3, gammapy |
+
+### Courses and tutorials
+
+Courses at the Education portal introduce the user to complete workflows in step-by-step guides. The following courses have at this point been provided in the portal:
+
+  * Analysis and visualization with Aladin: [https://edu.km3net.de/lesson/aladin/](https://edu.km3net.de/lesson/aladin/)
+  * Analysis and visualization with TOPCAT: [https://edu.km3net.de/lesson/topcat-2/](https://edu.km3net.de/lesson/topcat-2/)
+  * A simple analysis in python: [https://edu.km3net.de/lesson/a-simple-analysis-in-python/](https://edu.km3net.de/lesson/a-simple-analysis-in-python/)
+  * Use case 1 - Correlate neutrino data with Gamma Ray Bursts: [https://edu.km3net.de/lesson/correlating-with/](https://edu.km3net.de/lesson/correlating-with/)
+  * Use case 2 - Correlate neutrino data from different experiments: [https://edu.km3net.de/lesson/correlate-neutrino-data-from-different-experiments/](https://edu.km3net.de/lesson/correlate-neutrino-data-from-different-experiments/)



--- a/pages/Softwaredevelopment.md
+++ b/pages/Softwaredevelopment.md
@@ -4,10 +4,14 @@ Author: Tamas
 Topics:
  - using the python template
  - software development recommendations
-status: review please
+status: reviewed
+Reviewer: Jutta
 ---

 # Manuals for KM3NeT members
+
+Manuals for KM3NeT members concern mostly two parts: On the one hand how to provide the data and to publish them, on the other hand how to operate, maintain and contribute to the development of the open science servers and pipelines.
+
 ## How to Develope Software

 KM3NeT defines guidelines and recommendations for software development which

--- a/pages/Using_courses.md
+++ b/pages/Using_courses.md
 ---
-Title: How to use analysis data
-Author: Rebecca, Jutta
+Title: How to use courses
+Author: Dimitris, Jutta
 Topics:
  - ANTARES notebook example
  - openkm3 example
-status: unedited
-Comment: |
-  To create such a page from the notebook, 
-  export the jupyter notebook as markdown 
-  and add to this text file and store the notebook 
-  in the notebooks/ folder
+status: review
 ---

-## Accessing acoustic data
+## How to use the webinar materials

-## Science services
+KM3NeT members are encourged to use the open science materials to promote the KM3NeT work with young researchers and showcase collaborative work with the open science tools. To this end, the materials and concept for the online course described in Deliverable 4.10 are made available and offered for reuse.
+
+### Webinar setup
+The webinar was organized using Indico, a web-based management system for meetings. An [event](https://indico.cern.ch/event/959379/) was created on Indico with open access, that means no credentials were required in order to participate. For the webinar, a video meeting room was used linked in the description of the event's page.
+
+Time slots were created with added material to the different sections. In the case of the webinar, a time slot for each educational video, with the link of the lesson's page which contain the video in the Virtual Education Center, was added in the materials. 
+
+The participants were asked to follow the tutorials, with additional time for questions and feedback added after each lesson. The available time for each time slot was calculated from the video duration added to the time given to the participants to hands-on practise with the lesson's task. With the experience from this test webinar, the reccomendation to the coordinator is to give the participants 10 minutes to practise with the tasks of the Aladin & TOPCAT lessons, and 15 minutes with the tasks of the python lessons.       
--- a/pages/Using_templates.md
+++ b/pages/Using_templates.md
@@ -20,26 +20,23 @@ The Public plot template is a template to automatically create and populate a KM

 #### Motivation

-  * Standardize the archiving of public plots.
-  * Easily track the history of a public plot.
-  * A ready-to-use CI to create public plot(s).
-  * Ensure reproducibility and flexibility (easy to annotate, ability to hide graphs, change look and aspect ratio, colours, fonts etc).
-  * Get high-level data (with the corresponding units and explanation) to reuse the plot and easily modify it for conference presentations, publications, outreach events, posters…).
-  * Get a quick explanation of the plot and people involved in the study.
-  * Get access to the corresponding analysis repository on git.
-  * Easily share results with the outside word for collaborative work or benchmarks with other experiments.
-  * Ensure consistency in public plots archiving.
-
-
-#### The public plot template
-
-  * A public plot template has been defined by KM3NeT and is based on the cookiecutter (https://cookiecutter.readthedocs.io/) template framework. The template is publicly available under https://git.km3net.de/templates/km3net-public-plots . The template is
-specifically designed to fit the KM3NeT GitLab CI environment. The template is automatically populated with the meta information obtained during the template creation process (plot version, description, analysis repository, authors, etc).
-
-  * A demonstration on how a public plot repository looks like on git is available in https://git.km3net.de/templates/km3net-public-plots-demo/ .
-  * A step by step guide to produce this demonstration is here: https://git.km3net.de/templates/km3net-public-plots/-/blob/master/README.md .
-  * The official public plots which have been approved and produced with the public plot template are made available to the public in https://git.km3net.de/publications/plots .
-  * The public plot template contains the following files/directories:
+* Standardize the archiving of public plots
+* Easily track the history of a public plot
+* A ready-to-use CI to create public plot(s)
+* Ensure reproducibility and flexibility (easy to annotate, ability to hide graphs, change look and aspect ratio, colours, fonts etc)
+* Store and provide high-level data (with the corresponding units and explanation) to reuse the plot and easily modify the plot generation for customized use in conference presentations, publications, outreach events, posters etc.)
+* Easily share results for collaborative work and easy integration the data in common workflows, also for sharing with other experiments
+* Ensure consistency in public plots archiving
+
+
+#### Template project structure and use
+
+A public plot template has been defined by KM3NeT and is based on the cookiecutter [https://cookiecutter.readthedocs.io/](https://cookiecutter.readthedocs.io/) template framework. The template is publicly available under [https://git.km3net.de/templates/km3net-public-plots](https://git.km3net.de/templates/km3net-public-plots) . The template is specifically designed to fit the KM3NeT GitLab CI environment. The template is automatically populated with the meta information obtained during the template creation process (plot version, description, analysis repository, authors, etc).
+
+* A demonstration on how a public plot repository looks like on git is available in [https://git.km3net.de/templates/km3net-public-plots-demo/](https://git.km3net.de/templates/km3net-public-plots-demo/)
+* A step by step guide to produce this demonstration is here: [https://git.km3net.de/templates/km3net-public-plots/-/blob/master/README.md](https://git.km3net.de/templates/km3net-public-plots/-/blob/master/README.md)
+* The official public plots which have been approved and produced with the public plot template are made available to the public in a [designated namespace](https://git.km3net.de/publications/plots).
+* The public plot template contains the following files/directories:
    - LICENSE: a file containing the official KM3NeT license for public plots.
    - Makefile: a Makefile with commands like `make dependencies` and `make plot`.
    - README: a file pupulated with meta data (authors, contributors, analysis repository etc).
@@ -50,12 +47,10 @@ specifically designed to fit the KM3NeT GitLab CI environment. The template is a
    - requirements: a text file for software environment.
    - CHANGELOG: a file to keep track of the history (versions) of the public plot.

-
-**Note**: practically, from the user perspective, if one would like to reproduce a public plot, the steps are simplified to:
-  * cloning the public plot.
+This setup allows extremly easy rebuild of the plot. To reproduce the public plot, the steps are simplified to:
+  * cloning the public plot project.
  * typing `make dependencies`.
  * typing `make plot`.
-And the plot should be reproduced!

 ### Analysis template

@@ -63,20 +58,15 @@ The Analysis template is a template to automatically create and populate a KM3Ne

 #### Motivation

-  * Standardize the archiving of an analysis (steps, results, motivations ...).
-  * Easily track the history of an analysis.
-  * A ready-to-use CI to make an analysis.
-  * Ensure reproducibility.
-  * Track changes/operations performed on data (with the corresponding units and explanation) to understand and reproduce the analysis.
-  * Get a quick explanation of the analysis and people involved in it.
-  * Easily share details about an analysis within the collaboration but also with the outside word for collaborative work or benchmarks with other experiments, when necessary.
-  * Link an analysis repository to a KM3NeT publication (when required).
-  * Ensure consistency in analysis archiving.
+In addition to the motivation already described in the public plot template, analysis-specific aspects of the template generation are:
+  * Tracking changes/operations performed on data (with the corresponding units and explanation) to understand and reproduce the analysis and processing steps.
+  * Allow for a standardized review processed for the publication of analysis-related high level data through common structures in the storage of workflow-related information
+  * Link an analysis repository to a KM3NeT publication and facilitate the data publication process.
+  * Ensure consistency in analysis archiving

 #### The analysis template

-  * An analysis template has been defined and is based on the cookiecutter (https://cookiecutter.readthedocs.io/) template framework. The template is publicly available under https://git.km3net.de/templates/km3net-analysis-template . The template is
-specifically designed to fit the KM3NeT GitLab CI environment. The template is automatically populated with the meta information obtained during the template creation process (analysis version, description, authors, etc).
+  * An analysis template has been defined and is based on the [cookiecutter](https://cookiecutter.readthedocs.io/) template framework. The template is publicly available under [https://git.km3net.de/templates/km3net-analysis-template](https://git.km3net.de/templates/km3net-analysis-template).

  * The analysis template contains the following files/directories:
    - LICENSE: a file containing the official KM3NeT license for analysis.
@@ -91,8 +81,7 @@ specifically designed to fit the KM3NeT GitLab CI environment. The template is a
    - requirements: a text file for software environment.
    - CHANGELOG: a file to keep track of the history (versions) of the analysis.

-**Note**: practically, from the user perspective, if one would like to reproduce an analysis, the steps are simplified to:
-  * cloning the analysis repository.
-  * typing `make dependencies`.
-  * typing `make analysis`.
-And the analysis should be reproduced!
+Again, this simplifies (in a case where no external restricted dependencies and processing are involved) the execution of the main steps to
+  * cloning the analysis repository
+  * typing `make dependencies`
+  * typing `make analysis`