Interim commit before too much messing around

filelist_D4-9.txt

 pages/Manuals.md
-pages/Using_vo.md
-pages/Using_km3events.md
-pages/Using_plotdata.md
-pages/Dataserver_api.md
 pages/Softwaredevelopment.md
 pages/Using_templates.md
+pages/Using_vo.md
+pages/Using_courses.md
+pages/Manual_providing_data.md

index.rst

@@ -34,12 +34,11 @@ Welcome to Open Science @KM3NeT!
    pages/Testing
 
    pages/Manuals
-   pages/Using_vo
-   pages/Using_km3events
-   pages/Using_plotdata
-   pages/Dataserver_api
    pages/Softwaredevelopment
    pages/Using_templates
+   pages/Using_vo
+   pages/Using_courses
+   pages/Manual_providing_data
 
    pages/statuspage
 

pages/Dataserver_api.md

----
-Title: KM3NeT server REST API
-Author: Jutta
-status: unedited
----
-
-## Accessing the ODC via REST API
\ No newline at end of file

pages/Manuals.md

@@ -4,19 +4,50 @@ Author: Jutta
 status: dump
 ---
 
-# Manuals
-* no copying but pointing to the documentation
-
-systematics:
-- external manuals
-    - per platform
-    - per software
-    - per workflow
-
-- internal manuals
-    - providing data and software: templates
-    - publishing data -> procedures
-    - operating the system
-    - change procedures
+# Introduction
+There are two sides to the manuals for the KM3NeT open science system. On the one hand manuals are needed for the external user to understand the use of the data and the interfaces, on the other hand manuals are needed for KM3NeT members to actually contribute their software and data to the open science system. 
+
+However, as documentation to an evolving system as required for this deliverable is valid only for a short time, this document aims to give an overview over the different sources of documentation and point to the actual currently valid version. Only where documentation is not made available as versioned web resource or is in its content central to the architecture, the actual manual is copied or appended here.
+
+Note that most documentation for software is built automatically for source documentation and human-created documenation content is for most part hosted on the KM3NeT gitlab instance, as described in Deliverable 4.8. As the KM3NeT solution builds heavily on third-party software, documentation of this software is available elsewhere and pointed out here.
+
+# Manuals for users of the KM3NeT open science system
+
+## Manuals for the servers and platforms
+
+### Summary
+
+| platform | documentation URL | content |
+| -------- | ----------------- | ------- |
+| VO server Help | http://vo.km3net.de/static/help.shtml | Start page for Help |
+
+### VO server
+The DaCHS server comes with auto-generated documentation to the end points and numerous auto-generated listings of metadata, in addition to links to interface software and tutorials on their use.
+
+| Topic | documentation URL | content |
+| -------- | ----------------- | ------- |
+| TAP service | http://vo.km3net.de/__system__/tap/run/info | How to use TAP service |
+| ADQL | http://vo.km3net.de/__system__/adql/query/info | How to use ADQL query |
+| Data set descriptions | http://vo.km3net.de/__system__/dc_tables/show/tableinfo/ant20_01.main | Description of data table for ANTARES data |
+
+### KM3NeT Open Data Center
+
+
+## Manuals for software
+
+### Summary
+
+| Software | documentation URL | description | content |
+| -------- | ----------------- | ----------- | ------- |
+| 
+
+## Integration examples and workflows
+
+Workflows are made accessible as Jupyter notebooks, which serve to show the integration between the data and software components. All example notebooks can be downloaded from the Open Science Portal as package.
+
+| Name | File name | description | components |
+| -------- | ----------------- | ----------- | ------- |
+| 
+
 
 

pages/Python.md

@@ -19,4 +19,6 @@ The output format is preferably CSV and JSON to maximise interoperability. For l
 
 #### Python interface to KM3NeT data
 
-In addition to offering services and data through the KM3NeT Open Data Center, the [openkm3](https://git.km3net.de/open-data/openkm3) python client was developed to directly use open data in python from local computer and within e.g. Jupyter notebooks. It interlinks with the ODC REST-API and allows to query metadata of the resources and collections. In additionn to that, it offers function to interpret the data according to its KM3NeT type description ([ktype](Datamodels.md#ktype)), e.g. returning tables in a required format. These interface options will be expanded according to the requirements of data integrated to the ODC.
+In addition to offering services and data through the KM3NeT Open Data Center, the [openkm3](https://open-data.pages.km3net.de/openkm3/) python client was developed to directly use open data in python from local computer and within e.g. Jupyter notebooks. It interlinks with the ODC REST-API and allows to query metadata of the resources and collections. In additionn to that, it offers function to interpret the data according to its KM3NeT type description ([ktype](Datamodels.md#ktype)), e.g. returning tables in a required format. These interface options will be expanded according to the requirements of data integrated to the ODC.
+
+In addition to that, basic functions relevant for astrophysics are offered in the [km3astro](https://km3py.pages.km3net.de/km3astro/) package. As development of the python environment is an ongoing process, the number of packages offered for KM3NeT data interpretation will surely grow in the future.

pages/Repositories.md

@@ -30,7 +30,7 @@ At the end of the operation of KM3NeT and, in case the server can no longer be o
 ### DataCite and Zenodo
 Universal citation of data and digital objects is generally handled through assignment of a [Digital Object Identifier (DOI)](https://www.doi.org/), a persistent identifier or handle used to identify objects uniquely which standardized by the International Organization for Standardization (ISO). The VO does support the integration of a DOI in their resource metadata, but does not provide an authority to assign DOIs, as the organization assigning a DOI generally also has to host the data to ensure the longevity of the data resource.
 
-This leads to the dilemma that either the KM3NeT collaboration would have to become a member organization to e.g. [DataCite](https://datacite.org/), a global non-profit organisation that allows the assignment of DOIs to its member organizations, and operate a repository, or copy or mirror data to another repository in order to get a DOI. While this issue is not yet resolved and will be investigated further also in the context of the [ESCAPE project](https://projectescape.eu/), currently the second option is chosen for the small amount of data that is currently provided in this demonstrator. As hosting repository, [Zenodo](https://zenodo.org/) was chosen as well-established data repository in the physics community.
+This leads to the dilemma that either the KM3NeT collaboration would have to become a member organization to e.g. [DataCite](https://datacite.org/), a global non-profit organisation that allows the assignment of DOIs to its member organizations, and operate a repository, or copy or mirror data to another repository in order to get a DOI. While this issue is not yet resolved and will be investigated further also in the context of the [ESCAPE project](https://projectescape.eu/), where basic considerations are taken with respect to the [EOSC](https://ec.europa.eu/info/publications/persistent-identifier-pid-policy-european-open-science-cloud_en), currently the second option is chosen for the small amount of data that is currently provided in this demonstrator. As hosting repository, [Zenodo](https://zenodo.org/) was chosen as well-established data repository in the physics community.
 
 #### Registry structure
 Zenodo was initiated by the [OpenAIRE project](https://www.openaire.eu/) and is hosted by CERN to offer a respository for research funded by the European Commission. It hosts publications, audio and visual media, software and datasets. It allows to create "communities" to group various resources, which has been used to create a [KM3NeT community](https://zenodo.org/communities/?p=KM3NeT). Upload can be managed through an API as well as through a web interface and standardized metadata is required.
@@ -53,8 +53,3 @@ Neither Github nor PyPI make software citable in the strict sense, as they do no
 
 #### Archiving
 Both PyPI and Zenodo copy the relevant software to their platforms, copies of the source code are stored at multiple sites and make archiving easy.
-
-* Software can be integrated directly from GitHub
-
-
--> will search for common approach in ESCAPE project.

pages/Softwaredevelopment.md

@@ -8,7 +8,7 @@ status: review please
 ---
 
 # Manuals for KM3NeT members
-## How to Develop Software
+## How to Develope Software
 
 KM3NeT defines guidelines and recommendations for software development which
 help to maintain a consistent project structure and development process.

pages/Using_templates.md

@@ -10,7 +10,7 @@ Reviewer: Jannik
 status: reviewed
 ---
 
-## Using templates
+## Providing data
 
 KM3NeT defines guidelines and recommendations for physics analysis and public plots requirements. Templates are used to reenforce KM3NeT guidlines for physics analysis and public plots production, to ensure analysis reproducibility, consistency in analysis/plots archiving and documentation. In the following, we describe in details KM3NeT public plots templates and KM3NeT analysis templates.
 

pages/Using_vo.md

@@ -8,7 +8,7 @@ Topics:
 status: draft
 ---
 
-## Accessing data through the VO server
+### Use of third-party tools
 
 A short [online course](https://edu.km3net.de/course/accessing-the-km3net-open-data/) has been provided at our Education Portal.
 
@@ -16,7 +16,7 @@ Complying with the standards set by IVOA, the KM3NeT open data are accessible wi
 
 The standard Table Access Protocol allows the existance of TAP clients. The most widely used are the Aladin client as well as the TOPCAT client. The TAP clients interoperate and communicate under the Simple Application Messaging Protocol (SAMP). There is also an option to access the data via python scripting. This is possible using the pyVO, a package affiliated with Astropy.
 
-### Using VO clients
+#### Using VO clients
 Information on how to download and install Aladin is found at https://aladin.u-strasbg.fr/java/nph-aladin.pl?frame=downloading.
 
 An introduction on how to use ADQL is found [here](http://docs.g-vo.org/adql/html/)
@@ -24,48 +24,10 @@ An introduction on how to use ADQL is found [here](http://docs.g-vo.org/adql/htm
   * Aladin: https://aladin.u-strasbg.fr/java/nph-aladin.pl?frame=downloading .
   * TOPCAT: http://www.star.bris.ac.uk/~mbt/topcat/#install .
 
-### Using the VO python interface
+#### Using the VO python interface
 You can find information on how to install Astropy at https://www.astropy.org/ and for the pyVO at https://pyvo.readthedocs.io/en/latest/
 
 Accessing data with Python:
   * Astropy: https://www.astropy.org/ .
   * PyVO: https://pyvo.readthedocs.io/en/latest/ .
   * Matplotlib: https://matplotlib.org/3.1.1/users/installing.html .
-
-#### Example use
-
-```python
-import pyvo as vo
-
-service = vo.dal.TAPService("http://vo.km3net.de/__system__/tap/run/tap")
-
-# resultset works like a numpy record array
-resultset = service.search("SELECT * FROM ant20_01.main")
-print(resultset.fieldnames)
-print(resultset)
-
-resultset = service.search("SELECT * FROM ant20_01.main WHERE ABS(decl)<10 AND ABS(ra)<20 AND nhit>90")
-print(resultset)
-
-resultset = service.search("SELECT * FROM ant20_01.main WHERE ABS(mjd-57474.8)<1.0")
-print(resultset)
-```
-
-### Courses on the education portal
-  * Analysis and visualization with Aladin: https://edu.km3net.de/lesson/aladin/ .
-  * Analysis and visualization with TOPCAT: https://edu.km3net.de/lesson/topcat-2/ .
-  * A simple analysis in python: https://edu.km3net.de/lesson/a-simple-analysis-in-python/ .
-  * Use case 1 - Correlate neutrino data with Gamma Ray Bursts: https://edu.km3net.de/lesson/correlating-with/ .
-  * Use case 2 - Correlate neutrino data from different experiments: https://edu.km3net.de/lesson/correlate-neutrino-data-from-different-experiments/ .
-
-
-### How to use the webinar materials
-The webinar held using the Indico, a web-based management system for meetings [3]. An event created on Indico with open access [4], that means no credentials were required in order to participate. The Indico supports the option for a video room accessible from the event page. For the webinar though, a google meeting room was used linked in the description of the event's page.
-
-There is an option for an Indico event to contain time slots in order to distribute the available time covering the different meeting topics. In these time slots, material can be assigned. In the case of the webinar, a time slot created for each educational video, with the link of the lesson's page which contain the video in the Virtual Education Center [5], added in the materials. The available time for each time slot was calculated from the video duration added to the time given to the participants to hands-on practise with the lesson's task. With the experience from this test webinar, the reccomendation to the coordinator is to give the participants 10 minutes to practise with the tasks of the Aladin & TOPCAT lessons, and 15 minutes with the tasks of the python lessons.       
-
-[3]: https://indico.cern.ch/
-[4]: https://indico.cern.ch/event/959379/  
-[5]: Deliverable 4.11  
- 
-