diff --git a/pages/Courses.md b/pages/Courses.md index dce194ea634162d82d5e5f0729f9228f2b7d60cf..5442d2fa8c3f0cbac7213709a4b45590fb021e54 100644 --- a/pages/Courses.md +++ b/pages/Courses.md @@ -1,3 +1,4 @@ +--- Title: Courses and Webinars Author: Dimitris Topics: diff --git a/pages/Dataformats.md b/pages/Dataformats.md index ff5c3834e957fca0f0423cb71b71bc58a05d1d92..ab7113ee6358090f29331828ee8d0b09cb439ce7 100644 --- a/pages/Dataformats.md +++ b/pages/Dataformats.md @@ -1,3 +1,45 @@ +--- Title: Open data formats Author: Jutta +status: dump --- + +# Open data formats + +## Neutrino sets in the VO + +Tabulated high-level neutrino event data can be provided through the VO +registry, utilizing access protocols like the Table Access Protocol +(TAP) and query languages like the Astronomical Data Query Language +(ADQL). To query these data sets related to astronomical sources, the +Simple Cone Search (SCS) protocol allows to pick specific events +according to particle source direction, using Unified Content +Descriptors (UCDs) to identify the relevant table columns. The +underlying data format is the VOTable which allows for metadata +annotation of data columns. As the DaCHS software provides input +capabilities for various formats like FITS\footnote{Flexible Image + Transport System, \url{https://fits.gsfc.nasa.gov/}.} or text-based +tables on the server side, a common KM3NeT open event table format +can be chosen quite independently and the interface adapted such that high-level neutrino data +sets can be both offered through the VO and alternative access +protocols, as long as the required metadata description is handled +adequately. + +VO standards are at the current stage not fully adapted +to the inclusion of neutrino data and require development of metadata +standards for easy interpretability of the data, a matter which is targeted within the ESCAPE project. Open questions in this regard are the linkage of observation probabilities to a given event selection, the inclusion of ``non-observation'' in a given field of view +and within a given time as relevant scientific information to retrieve +from the services, and the introduction of a dedicated vocabulary for the +description of neutrino data. This vocabulary will need to be developed within KM3NeT as a matter of internal standardization, however, the process will draw guidance from the VO expertise and framework. + +## Multimessenger alerts + +Single or stacked neutrino events of high astrophysical signal probability will be selected in KM3NeT to trigger an alert to other observatories indicating a +possible target for multimessenger observations \cite{MM}. The +VOEvent format, together with the VOEvent Transport +Protocol as implemented in the Comet +software\footnote{J. Swinbank, Comet, \url{https://comet.transientskp.org}.} will be used to distribute these events as outgoing alerts. As the format is specifically tailored to the use +in multimessenger alerts, indicating a quite restricted scientific target, the providing of context information for the events can be specifically adapted to this use case. However, harmonization of metadata standards like parameter descriptors and event identifiers in reference to the full neutrino event sets will also have to be implemented. + +## Providing simulation-driven services +Providing context information on a broader scale in the form of e.g. sensitivity services and instrument response functions alongside the VO-published data sets is still under investigation. On the one hand, VO access protocols like TAP facilitate the use of standardized queries on services. On the other hand, integrating those services with the data sets in a meaningful and user-transparent way, e.g. through VO Datalinks, still requires further deliberation. Therefore, the development of these services will be use-case driven and also include the application of similar services for studies in other fields of KM3NeT research beyond astrophysics. diff --git a/pages/Datamodels.md b/pages/Datamodels.md index faba15e65e96cd5e709725b560c8bed5dc36fd3a..ba139ed466b75f0f81ef843f9c639efdcd786698 100644 --- a/pages/Datamodels.md +++ b/pages/Datamodels.md @@ -1,6 +1,12 @@ +--- Title: Metadata generation and datamodels Author: Jutta Topics: * data models * configurations for software --- + +* Started collection of metadata definitions (from publication level), based on standards of e.g. W3C, IVOA, DataCite etc. +* Basic class for open science publications: km3resource +* To be extended to earlier processing stages to standardize data processing and handling +* Basic identifying data classes: ktype (pointing to a class definition) and kid (unique identifier of objects in KM3NeT) diff --git a/pages/Dataserver_api.md b/pages/Dataserver_api.md index 53b4bbe14103037c562f6ae7054397bbff0d427f..6835cb8e67737049aa400278914e3de4f7a15a9f 100644 --- a/pages/Dataserver_api.md +++ b/pages/Dataserver_api.md @@ -1,3 +1,4 @@ +--- Title: KM3NeT server REST API Author: Jutta --- diff --git a/pages/Detector.md b/pages/Detector.md index c22e0caff489c303d272605f0b837b5b2e5a91f6..9f25be696fe163067c1c6f6ed13ae81ac68719d0 100644 --- a/pages/Detector.md +++ b/pages/Detector.md @@ -1,9 +1,11 @@ +--- Title: Detector and Data Taking Author: Jannik (Mainly based on LoI) +--- -**Detector** +## Detector The KM3NeT Research Infrastructure will consist of a network of deep-sea neutrino detectors in the Mediterranean Sea with user ports for Earth and Sea sciences. @@ -17,7 +19,7 @@ The ARCA (Astroparticle Research with Cosmics in the Abyss) detector is being in Technical details on the detector design are given in [1]. -**Data Acquisition** +## Data Acquisition The readout of the KM3NeT detector is based on the 'all-data-to-shore' concept, in which all analogue signals from the PMTs that pass a reference threshold are digitised. This data contain the time at which the analogue pulse crosses the threshold level, the time that the pulse remains above the threshold level (known as time-over-threshold, or ToT), and the PMT address. This is typically called a *hit*. All digital data (about 25 Gb/s per building block) are sent to a computing farm onshore where they are processed in real time. The recorded data is dominated by optical background noise from Cherenkov light from K40 decays in the seawater as well as bioluminescence from luminescent organisms in the deep sea. Events of scientific interest are filtered from the background using designated software, which exploit the time-position correlations following from causality. To maintain all available information for the offline analyses, each event contains a snapshot of all the data in the detector during the event. @@ -29,7 +31,7 @@ In parallel to the optical data, acoustic data and instrument data are recorded. During operation the continuous data stream sent by the detector is split into small time intervals, called *runs*, with typical durations of a few hours. This is done for practical reasons of the data acquisition. In addition, this procedure allows to selected a set of run periods with high-quality data based on the monitored detector status, environmental conditions and data quality. The calibration for timing, positioning and photon detection efficiency is done offline using the calibration data. -**Simulations** +## Simulations To assess the detector efficiency and systematics, dedicated Monte Carlo simulations are processed. Due to the changing data-taking conditions of the detector in the deep-sea environment, time-dependent simulation data-sets are required. These are implemented in a run-by-run simulation strategy, where runs are sufficiently small time intervals of data taking with stable conditions. The detector response is simulated individually for these periods. The simulation data are generated at the raw-data level and are subjected to the same filter and reconstruction processing as the real data. Since large statistics are required for precise analyses, the simulation data will significantly exceed the real data in volume. diff --git a/pages/Docker.md b/pages/Docker.md index 5262ba71a71be8acbc0e5d4ff44c39f52b7451c0..1466244faa23097fbc5a18cf902e149411b99b15 100644 --- a/pages/Docker.md +++ b/pages/Docker.md @@ -1,3 +1,4 @@ +--- Title: Docker Author: Tamas Title: diff --git a/pages/ESCAPE.md b/pages/ESCAPE.md index ec2551b16db69bb3cb9cefb0c4665af64007e802..cab9dadc0908309f28b29ed117e211a7dd141512 100644 --- a/pages/ESCAPE.md +++ b/pages/ESCAPE.md @@ -1,3 +1,4 @@ +--- Title: ESAP & ESCAPE Author: Jutta --- diff --git a/pages/FAIR.md b/pages/FAIR.md index fabac80b906fe8cebd6c1182621520cd94417d3f..d439d79abea4fbc61db6ec9d72e02543ac63a588 100644 --- a/pages/FAIR.md +++ b/pages/FAIR.md @@ -1,3 +1,4 @@ +--- Title: The FAIR principles Author: Jutta Topics: @@ -5,3 +6,21 @@ Topics: * dedication to open science --- +## Publishing FAIR data +The widely-accepted paradigm for open science data publication requires the implementation of the FAIR principles \cite{FAIR} for research data. This involves the definition of descriptive and standardized metadata and application of persistent identifiers to create a transparent and self-descriptive data regime. Interlinking this data to common science platforms and registries to increase findability and the possibility to harvest from the data through commonly implemented interfaces is as mandatory as is the definition of a policy standard including licensing and access rights management. In all these fields, the standards of KM3NeT are currently developing, including the implementation of a data management plan, the installation of a data provenance model including the application of workflow management, and the setting of data quality standards. In this development process, the application of existing standards especially from the astrophysics community, the development of dedicated KM3NeT software solutions and the integration of the KM3NeT efforts developed during the KM3NeT-INFRADEV project\footnote{see \url{https://www.km3net.org/km3net-infradev/}} are integrated into the ESCAPE project\footnote{European Science Cluster of Astronomy \& Particle physics ESFRI research Infrastructures, \url{https://projectescape.eu}.}, which forms the main development environments for open data publication in KM3NeT. + +## Compliance with the FAIR principles +Although the FAIR data principles define a series of criteria that data and metadata should meet in order to enhance their public usage [4], the KM3NeT collaboration is working to ensure that the data for internal usage are also FAIR compliant. In the following sections, the internal compliance with the FAIR principles is detailed, and the strategy for open access data compliance with FAIR is described. + +### Metadata for findability and accessibility +A database has been implemented which houses data and metadata related to different aspects of the KM3NeT research infrastructure. Amongst others, this database hosts metadata related to the data taking runs, and calibrations, as well as detector identifiers needed to find the existing data. Data storage at high-performance computing clusters is tracked and files are identifiable through a unique numbering system, where filenames contain the detector identifier and the run number. The metadata contain all the necessary information to track the data in each file down to the original raw data from which they were produced. Additionally, the information about the trigger parameters used in each run is also contained in the metadata. Metadata for software contain complete information about the software versions used to produce each data file as well as information about the computing environment. +Metadata are currently stored within the processed file, although future options for external storage of metadata are investigated to comply with high-performance data management systems like Dirac [10] within the EOSC. External metadata storage will also secure the future provenance if outdated data sets are deleted. + +### Standardization for access, interoperability and reusability +Currently two different frameworks are maintained, documented and developed for official use within the KM3NeT Collaboration that allow to use the data: the KM3pipe framework, which is developed in python language [11]; and the Jpp framework which is a C++ based software design. Complementing file storage in a ROOT-based format, an HDF5 format definition for both low and high-level data is envisaged, so that the data can be accessed by open source libraries without additional dependencies. All KM3NeT processing software is available in portable environments for use in docker [9] or singularity [14] to ensure portability, and partly available under MIT license. +Introduction of semantic metadata according to established conventions by the World Wide Web Consortium and extensions by the IVOA will further enhance the interoperability of the data processing chain and products. + +### Compliance of open access data with FAIR principles +The properties which make the KM3NeT data compliant with the FAIR principles will be propagated in their transformation to the open-access datasets. During the last few months, contacts have been made with the German Astrophysical Virtual Observatory (GAVO) [12], which is the German contribution to the IVOA. The first conversations between KM3NeT and GAVO members have been focused on the required standards for the publication of datasets corresponding to searches of cosmic neutrinos in the Virtual Observatory. By respecting and developing these standards it is ensured that the provided data will comply with the FAIR principles. + + diff --git a/pages/Git.md b/pages/Git.md index dbbf7415d9e82fbb953557f5b2c5483734982ae2..17586deb036da85827233fa0c5049491bfb22f1c 100644 --- a/pages/Git.md +++ b/pages/Git.md @@ -1,3 +1,4 @@ +--- Title: Gitlab und Github Author: Tamas Title: diff --git a/pages/Infrastucture.md b/pages/Infrastucture.md index eb3fb2ea51d9ae5a8e6a07f2758313bbaea39fe7..0d5fe3da5a190cddf683de4c1f751bb885bc1c09 100644 --- a/pages/Infrastucture.md +++ b/pages/Infrastucture.md @@ -1,5 +1,7 @@ +--- Title: Architecture overview Author: Jutta Topics: * servers, repositories, webpages --- + diff --git a/pages/KM3NeTserver.md b/pages/KM3NeTserver.md index 0adf401a456f4598d45099d2d794457fad08ea97..de731f65a6a6387545cc6a26d0ae966aa46e5367 100644 --- a/pages/KM3NeTserver.md +++ b/pages/KM3NeTserver.md @@ -1,3 +1,10 @@ +--- Title: KM3NeT server Author: Jutta --- + +* For all data not publishable through the IVOA, serving as interface and/or server to the data +* Based on Django REST API +* Usable for event data sets (hdf5-files with standardized metadata), plots or services, environmental data ... +* Data accessible through webpage, through REST-API or python based package (openkm3) +* Similar structure to e.g. Gravitational Wave Open Science Center diff --git a/pages/Multimessengers.md b/pages/Multimessengers.md index 242031b874b05370f2f6b1aa2c57c2a120a4abc9..8ec39236ac9796ee3d3bea5a272ccd3cc5938410 100644 --- a/pages/Multimessengers.md +++ b/pages/Multimessengers.md @@ -1,3 +1,4 @@ +--- Title: MM alerts Author: Feifei Topics: diff --git a/pages/OSP.md b/pages/OSP.md index 1d8d2962d5b1d10c1ca99dff625c2791ba3b4d07..79e413d70a3fdbc202e90ebe64a7d61535a8b016 100644 --- a/pages/OSP.md +++ b/pages/OSP.md @@ -1,6 +1,13 @@ -Title: python interface -Author: Tamas, Jutta +--- +Title: Open Science Portal +Author: Jutta Topics: * installation * interface through software --- + +* Git project serving webpage (Gitbook) +* Knowledge base for open data +* Pointing to all open science products +* Including introduction to KM3NeT data and linking to manuals and projects (how to handle the data, tutorials and manuals for KM3NeT members) +* Should be central access point for external users diff --git a/pages/Procedures.md b/pages/Procedures.md index 72dfe35581db10f5ec021e9142f84cb0261fa93a..527080e0fe415c5a77012d7892a22955e4688326 100644 --- a/pages/Procedures.md +++ b/pages/Procedures.md @@ -1,6 +1,88 @@ +--- Title: Publication procedures Author: Kay, Jutts Topics: * data/software releases * publication procedures --- + +## Establishing the Open Science Committee + +## Software publication + +### Software quality standards + +Core requirements (Code) +Storage: git / CI/CD -> SFTP +Installation: Containers +Documentation: wiki -> link to code documentation in git (doxygen or similar: reference guide/API, Getting started & Concepts) +Change procedure: git workflow +Coding standards: +C++ Style Guide / ROOT, +Python Style Guide PEP-8 +Java Style Guide +Julia? Fortran? + +Recommendations +Tutorial: Getting started / Guides / Concepts / API +Examples +Installation guide + + + +### Publication procedures + + +#### Roles - who is what? +Author: substantial contribution / idea +Maintainer: responding on issues +Copyright holders: KM3NeT (author list, DOI) +Contributors +WG Coordinator +Referees: nominated by PC and/or OSC +Last step -> actor should be maintainer, author, contributor +Add collaborative feedback as new step +Relating to +OSC: software & data “experts†+PC + +### Stages + + +#### Definition of roles, software candidates and (specific) standards + +#### Internal development to meet standards (after pre-review) + +#### Reviewing process + +#### Publication +Certified by KM3NeT: +License of KM3NeT +Software repository choice + + +#### Maintenance + +### Implementation + +Setting up procedures and standards (Who/what?) +* Update of the note, circulation through collaboration + * Proposal on OSC to IB (from ECAP) + * Transition procedures + +Current candidates +* Without full reviewing process +* Interim referees: + * Unused: either two referees (extensive) + * Published results: one referee (shortened) +* Improve to meet standards + +Software published, but not reviewed +(Software used, but not published) -> encourage publication + + +## Data publication + +### Data quality standards + +### Publication procedures diff --git a/pages/Processing.md b/pages/Processing.md index ace3128f7eb72c041d3824430885d341af23101f..c01bbc20ca60dc396df2293eaaaead6075f4ef3b 100644 --- a/pages/Processing.md +++ b/pages/Processing.md @@ -1,3 +1,106 @@ +--- Title: Processing overview Author: Jutta --- + +Data processing therefore follows a tier-based approach \cite{km3comp}, where initial +filtering for particle interaction-related photon patterns (triggering of photon +``hits'') serves to create data at a first event-based data level. +In a second step, processing of the events, applying calibration, +particle reconstruction and data analysis methods leads to enhanced data sets, +requiring a high-performance computing infrastructure for flexible +application of modern data processing and data mining techniques. + +For physics analyses, derivatives of these enriched data sets are +generated and their information is reduced to low-volume high-level data which can be +analysed and integrated locally into the analysis workflow of the +scientist, see Figure \ref{levels}. For interpretability of the data, a full Monte Carlo +simulation of the data generation and processing chain, starting at the +primary data level, is run to generate reference simulated data for +cross-checks at all processing stages and for statistic +interpretation of the particle measurements. + +\begin{figure} +\includegraphics[width=\textwidth]{figs/Data_levels.pdf} +\caption{KM3NeT data levels related to open data publication, including data format description, user access rights and open data publication layer.} \label{levels} +\end{figure} + +\subsection{Event-based data generation} + +Data processing at the DAQ level follows paradigms of particle physics +and utilizes computing and software methodological approaches of this community. At +the shore stations, event triggering in the Data Acquisition (DAQ) +system leads to a significant reduction of the data stream. The data stream also includes relevant +instrumentation readouts for a comprehensive understanding of data taking +conditions. Photon-related information is written to +ROOT-based \cite{root} tree-like data structures and +accumulated during a predefined data taking time range of usually several +hours (so-called data runs) before being transferred to high-performance +computing (HPC) clusters. Instrumentation and environmental data +collected at the detector site are stored separately in a central database. Acoustic and other environmental data serve as basis for Earth and Sea-science initiatives. Access to this information following an Open Science +approach is under development, however, it will not be covered in the +scope of this report. + +Both the complex process of neutrino detection in a natural environment and the low expected +count rate of the cosmic neutrino signal in comparison to atmospheric background events +necessitates the full modelling +of particle generation, detector response and data processing. To this +end, a dedicated simulation chain, starting form cosmic air-shower +particle generation or astrophysical neutrino flux assumptions, +replicates the complete data-processing pipeline. At the event generation level, photon +distributions induced by these particles within the detection volume are +generated, masked by a simulation of the detector response and treated to +the same processing as measurements starting from the second data +level of the offline event format. + + +\subsection{Event data processing} + +Processed event data sets at the second level represent input to physics analyses, e.g.~regarding neutrino oscillation and particle properties, and studies of +atmospheric and cosmic neutrino generation. Enriching the data to this +end involves probabilistic interpretation of temporal and spatial photon distributions for the +reconstruction of event properties in both measured and simulated +data, and requires high-performance computing capabilities. Due to the +distributed infrastructure of the KM3NeT building blocks and the +contribution of computing resources from various partners, data +processing will, in the final detector configuration, necessitate a +federated computing approach, the implementation of which is prepared through containerization +of the required software and testing of distributed resource management +approaches. In this context, the use of a middleware like e.g.~ DIRAC\footnote{Distributed Infrastructure with Remote Agent Control Interware, \url{http://diracgrid.org/}} is explored, again linking +closely to the particle physics community. + +Access to data at this level is restricted to collaboration members due +to the intense use of computing resources, the large volume and complexity of the data and +the members' primary exploitation right of KM3NeT data. However, data at this stage is already converted to +HDF5\footnote{The HDF5 file format, \url{https://www.hdfgroup.org/}} format as a less customized hierarchical format. This format choice increases interoperability and facilitates the application of data analysis software packages used e.g.~in machine learning and helps to pave the way to wider +collaborations within the scientific community utilizing KM3NeT data. + + +\subsection{High level data and data derivatives} + +\subsubsection{Summary formats and high-level data} + +As mostly information on particle type, properties and direction is relevant for +the majority of physics analyses, a high-level summary format has been designed to +reduce the complex event information to simplified arrays +which allow for easy representation of an event data set as a table-like data structure. +Although this already leads to a reduced data volume, these neutrino +data sets are still dominated by atmospheric muon events at a ratio of about +$10^{6}:1$. Since for many analyses, atmospheric muons are considered background events to +both astrophysics and oscillation studies, publication of low-volume +general-purpose neutrino data sets requires further event filtering. Here, the choice of optimal filter criteria is usually dependent on the properties of the +expected flux of the signal neutrinos and performed using the simulated event sets. + + +\subsubsection{Event simulation derivatives as service} + +To correctly judge the statistical significance of a measured neutrino event rate, +the full high-level simulation data sets are used in KM3NeT internal +studies to ensure a high accuracy of the count rate estimate. As handling +these large data sets is impractical for inter-experimental studies, but +the information is crucial for the interpretability of the data, +parameterized distributions of relevant observables need to be derived +from the simulation data sets and offered as services. Even in absence +of significant neutrino measurements in the construction phase of KM3NeT, +offering sensitivity estimates as in \cite{sensitivity} for given +models is beneficial for the development of common research goals and the development of a corresponding open service is currently under investigation. diff --git a/pages/Python.md b/pages/Python.md index 0c2f25f41abfbbd5e77f68b3e49e31795a3be4b8..1d89c658be9909efe94debd07ded4e32aaccc424 100644 --- a/pages/Python.md +++ b/pages/Python.md @@ -1,5 +1,23 @@ +--- Title: Python interface Authors: Tamas, Jutta Topics: * installation * interface through software +--- + + + + + + + +## Python interface to KM3NeT data +### openkm3 +* Small python package to directly use open data in python from local computer +* Interfaces with data center API, allows to query datasets (km3resources) +* Provides functions to download & interpret data products + +# Current capability +* Loading hdf-File as pandas DataFrame, reading additional parameter infos & provenance +* Reading histogram data as plain table, pandas Dataframe or automatically build plot diff --git a/pages/Quality.md b/pages/Quality.md index 206bdddbd0e4b44e79288cbdf816021bbe334415..8f1ab485e8f127b163c80f32c66dc57f8aa4e3bd 100644 --- a/pages/Quality.md +++ b/pages/Quality.md @@ -1,3 +1,4 @@ +--- Title: Quality management Author: Rodri Topics: diff --git a/pages/Repositories_VOserver.md b/pages/Repositories_VOserver.md index 9d7c6b63007470c569c9067b775529a51c29f159..c4e9839bdbde2ba2573de51351fb08d32bf649f9 100644 --- a/pages/Repositories_VOserver.md +++ b/pages/Repositories_VOserver.md @@ -1,3 +1,4 @@ +--- Title: VO repository Author: Jutta --- diff --git a/pages/Repositories_Zenodo.md b/pages/Repositories_Zenodo.md index 1a95d9f8be37b086a29c72c092c2f58e86146f53..1a2a2ff7bb512c862df1bb55e3a76c6395ca3739 100644 --- a/pages/Repositories_Zenodo.md +++ b/pages/Repositories_Zenodo.md @@ -1,3 +1,10 @@ +--- Title: Integrating to Zenodo Author: Jutta --- + +* For findability data needs to be citable and registered within a large repository (e.g. DataCite) +* Zenodo as platform widely used in the community, assigning persistent identifiers (DOI) to datasets, images, publications & software +* Software can be integrated directly from GitHub +* KM3NeT “Community†created to connect different KM3NeT contributions +* Can register smaller data samples (planned for KM3NeT example data) as well as public plots, posters etc. diff --git a/pages/ScienceTargets.md b/pages/ScienceTargets.md index 4978ac438f4dec60474f3fdd5bb6013021b445c3..a3312255c0f988713676416c43cf421e2f020b91 100644 --- a/pages/ScienceTargets.md +++ b/pages/ScienceTargets.md @@ -1,14 +1,16 @@ +--- Title: Scientific targets Author: Jannik +--- -**Scientific Targets** +# Scientific Targets The KM3NeT neutrino detectors will continuously register neutrinos from the whole sky. The neutrinos of astrophysical interest, i.e. those from extra-terrestrial origin, need to be identified in the background of atmospheric neutrinos, i.e. those created in Earth’s atmosphere by interactions of cosmic-ray particles. Access to cosmic neutrino data is of high importance for a wide astrophysics community beyond the KM3NeT Collaboration to relate cosmic neutrino fluxes to observations by other neutrino observatories or using other messengers [REFERENZ to Multimessenger], and to compare them with theoretical predictions. The atmospheric neutrinos carry information on the particle physics processes in which they are created, and – in particular those registered with KM3NeT/ORCA – on the neutrinos themselves. These data are relevant for a wide astroparticle and particle physics community. Finally, KM3NeT will monitor marine parameters, such as bioluminescence, currents, water properties and transient acoustic signals and provides user ports for Earth and Sea sciences. (Taken from Grant Agreement) -**Astro Physics** +## Astro Physics The main science objective of KM3NeT/ARCA is the detection of high-energy neutrinos of cosmic origin. Neutrinos represent an alternative to photons and cosmic rays to explore the high-energy Universe. Neutrinos can emerge from dense objects and travel large distances, without being deflected by magnetic fields or interacting with radiation and matter. Thus, even modest numbers of detected neutrinos can be of utmost scientific relevance, by indicating the astrophysical objects in which cosmic rays are accelerated, or pointing to places where dark matter particles annihilate or decay. @@ -21,7 +23,7 @@ The ARCA detector allows to reconstruct the arrival direction of TeV-PeV neutrin Further details on the detector performance can be found in [1]. -**Neutrino Physics** +## Neutrino Physics Neutrinos have the peculiar feature that they can change from one flavour to another when propagating over macroscopic distances. This phenomenon of neutrino flavour change is known as 'neutrino oscillation'. The Nobel Prize in Physics of the year 2015 was awarded to T. Kajita and A. B. McDonald for the discovery of neutrino oscillations, which shows that neutrinos have mass [1]. One open question is the so-called 'neutrino mass ordering'. It refers to the sign of one of the two independent neutrino mass differences, the absolute value of which has already been known for more than two decades. diff --git a/pages/SeaScience.md b/pages/SeaScience.md index 33f5ae99658c253d7c3471172da74ba294539291..50c6a88c94411bc9311a4f347f4fac34fa11ae3c 100644 --- a/pages/SeaScience.md +++ b/pages/SeaScience.md @@ -1,13 +1,13 @@ +--- Title: Environmental data -Author: Jannik, Daniele, Giorgio +Author: Jannik Topics: * options for sea science * types of environmental data +Status: draft --- -**Proposed text by Jannik** - -**Environmental data** +## Environmental data The KM3NeT research infrastructure will also house instrumentation for Earth and Sea sciences for long-term and on-line monitoring of the deep-sea environment. Until now, measurements in the deep sea are typically performed by deploying and recovering autonomous devices that record data over periods of months to years. This method is severely constrained by bandwidth limitations, by the absence of real-time interaction with the measurement devices and by the delayed access to the data. A cabled deep-sea marine observatory, like KM3NeT, remedies these disadvantages by essentially providing a power socket and high bandwidth Ethernet connection at the bottom of the sea. This is an important and unique opportunity for performing deep-sea research in the fields of marine biology, oceanography, environmental sciences and geosciences. To this end, both the French and Italian KM3NeT sites are nodes of the European Multidisciplinary Seafloor and water column Observatory (EMSO) [1]. diff --git a/pages/Simulation.md b/pages/Simulation.md index c783bb92382f03a00daa7616bb204716753f130f..1b6d3163474cca519bafda24c2bc5d8b5ef66b91 100644 --- a/pages/Simulation.md +++ b/pages/Simulation.md @@ -1,3 +1,4 @@ +--- Title: Event simulation Author: Jutta, Tamas --- diff --git a/pages/Softwaredevelopment.md b/pages/Softwaredevelopment.md index 6062d22b37ec63753843dea0cc6c035e68816932..123d9940c13124bc6ae7a7080409e630c77fa244 100644 --- a/pages/Softwaredevelopment.md +++ b/pages/Softwaredevelopment.md @@ -1,3 +1,4 @@ +--- Title: How to develop software Author: Tamas Topics: diff --git a/pages/Usecase_ANTARES.md b/pages/Usecase_ANTARES.md index 1f1574da9b986ca69c338b50a1cadfba707304b0..49dcfd73b7f29234c380172149f3081e4e18f4f8 100644 --- a/pages/Usecase_ANTARES.md +++ b/pages/Usecase_ANTARES.md @@ -1,3 +1,4 @@ +--- Title: ANTARES use case Author: Rebecca Topics: diff --git a/pages/Usecase_Environmental.md b/pages/Usecase_Environmental.md index 1eb376313bf8a84b21631056c4238e4a6f757e7b..8b332cb41d98b749e8311919e98eb86c17bec752 100644 --- a/pages/Usecase_Environmental.md +++ b/pages/Usecase_Environmental.md @@ -1,6 +1,13 @@ +--- Title: Environmental data usecase Topics: * short description use * data description * interfaces --- + +* Acoustic data will be provided through separate REST-API +* Envisioned data including psd tables and audio as wave files for a few minutes per day/week +* Data sets integrated as km3resources to Open Data Center +* Jupyter notebook demonstrating data use + diff --git a/pages/Usecase_KM3NeT.md b/pages/Usecase_KM3NeT.md index 14fa3ec1d098b98f16a3768cedf3e8cee8011dd8..8e951a695abff96cd93f116a2856155d1c09c584 100644 --- a/pages/Usecase_KM3NeT.md +++ b/pages/Usecase_KM3NeT.md @@ -1,3 +1,4 @@ +--- Title: KM3NeT use case Author: Steffen Topics: @@ -5,3 +6,9 @@ Topics: * data description * linking to notebooks & background --- + + +* Dataset to show use of KM3NeT data in a non-astrophyics context (or using a muon dominated data samples) +* Runs selected from ORCA4 line data according to quality criteria +* Processed to metadata enriched hdf5 +* Analysis examples provided as jupyter notebook diff --git a/pages/Using_alerts.md b/pages/Using_alerts.md index 3b2cfa393c9c6226788d1383e42a35630ca760b3..a4d3b31f7d53ba9b2cc63fe21cb5d08d51b58522 100644 --- a/pages/Using_alerts.md +++ b/pages/Using_alerts.md @@ -1,3 +1,4 @@ +--- Title: How alert data can be used Author: Feifei, Jutta Topics: diff --git a/pages/Using_km3events.md b/pages/Using_km3events.md index e5e5fd6c45f69ee8cc66e98ec6d6991c936da9c5..832c820546e2641167339fe512b60677f0ba7a43 100644 --- a/pages/Using_km3events.md +++ b/pages/Using_km3events.md @@ -1,3 +1,4 @@ +--- Title: How to read hdf5 data Author: Steffen, Jutta Topics: diff --git a/pages/Using_plotdata.md b/pages/Using_plotdata.md index 0efa026a94dcca401e449f1a8bc6c5d8b5783d41..945bfd68eb215b5f5e8dcf4b34d509c16e657769 100644 --- a/pages/Using_plotdata.md +++ b/pages/Using_plotdata.md @@ -1,3 +1,4 @@ +--- Title: How to use analysis data Author: Rebecca, Jutta Topics: diff --git a/pages/Using_templates.md b/pages/Using_templates.md index abbad74c83b62cd07eced92874e7531c38f5b920..7a784a0bcc57ffb61c58ce03f48b570b995361c8 100644 --- a/pages/Using_templates.md +++ b/pages/Using_templates.md @@ -1,3 +1,4 @@ +--- Title: Using public plot and analysis templates Author: Zineb, Jutta Topics: diff --git a/pages/Using_vo.md b/pages/Using_vo.md index 4dc1e58fbf09cd2cb13d88f89f00bfe6e5558b4e..a4bd895055071d58d32dfaaec66660b9b8e54d89 100644 --- a/pages/Using_vo.md +++ b/pages/Using_vo.md @@ -1,6 +1,44 @@ +--- Title: How to use VO SCS data Author: Jutta Topics: * summary from open course * https://edu.km3net.de/course/accessing-the-km3net-open-data/ + +Status: draft --- + +# Accessing data through the VO server + +A short [online course](https://edu.km3net.de/course/accessing-the-km3net-open-data/) has been provided at our Education Portal. + +Complying with the standards set by IVOA, the KM3NeT open data are accessible with services defined by the Table Access Protocol (TAP). The services operating under this protocol allow the execution of queries in order to retrieve the data from the database tables as well as to inspect various metadata. The TAP services support SQL commands. It is highly recommended the queries to be written in Astronomical Data Query Language (ADQL). A nice introduction of ADQL used in a TAP service is found here. Another way to retrieve the data is the Simple Cone Search (SCS). This defines queries for sky search in a cone with given position and radius. + +The standard Table Access Protocol allows the existance of TAP clients. The most widely used are the Aladin client as well as the TOPCAT client. The TAP clients interoperate and communicate under the Simple Application Messaging Protocol (SAMP). There is also an option to access the data via python scripting. This is possible using the pyVO, a package affiliated with Astropy. + +## Using VO clients +Information on how to download and install Aladin is found at https://aladin.u-strasbg.fr/java/nph-aladin.pl?frame=downloading. + +An introduction on how to use ADQL is found here. + +## Using the VO python interface +You can find information on how to install Astropy at https://www.astropy.org/ and for the pyVO at https://pyvo.readthedocs.io/en/latest/ + +```python +import pyvo as vo + +service = vo.dal.TAPService("http://vo.km3net.de/__system__/tap/run/tap") + +# resultset works like a numpy record array +resultset = service.search("SELECT * FROM ant20_01.main") +print(resultset.fieldnames) +print(resultset) + +resultset = service.search("SELECT * FROM ant20_01.main WHERE ABS(decl)<10 AND ABS(ra)<20 AND nhit>90") +print(resultset) + +resultset = service.search("SELECT * FROM ant20_01.main WHERE ABS(mjd-57474.8)<1.0") +print(resultset) +``` + +An introduction on how to use ADQL is found [here](http://docs.g-vo.org/adql/html/) diff --git a/pages/VOserver.md b/pages/VOserver.md index c9499015a6806cf18e3d5c78c24308c658ac703e..5cf2df8f3ed06a5c6a688ab9832cb29c9ff23a02 100644 --- a/pages/VOserver.md +++ b/pages/VOserver.md @@ -1,3 +1,29 @@ +--- Title: VOserver Author: Jutta --- + +## Approach to the Virtual Observatory standard + +The Virtual Observatory (VO) standards\cite{VO} +serve to create an interface between astronomy-related data resources +from astrophysics experiments which act as data providers. The focus is on the scientific end user to easily interface from their personal +computer with the provided data sets. The KM3NeT collaboration is a data +provider to the VO and operates a data server\footnote{at \url{http://vo.km3net.de/}} running the DaCHS software\cite{dachs}. +The well-developed data sharing regime of the VO serves well as a +guideline for the implementation of astrophysical data sharing in the +KM3NeT collaboration. However, considering the role of neutrino physics +just at the edge of use for astronomical studies, KM3NeT data +integration also meets some limitations considering the scientific +usability of the provided data sets. In addition to that, publication of +data through VO standards is clearly limited to astronomy-related data +in a celestial reference frame. + +* Running server with DaCHS software +* Registered as data provider to the VO (ivo://km3net.org) +* Can publish data sets to the VO registry - done with Antares 2007-2017 data +* Data accessible through widely used tools in astrophysics (Aladin, Topcat …) + + + + diff --git a/pages/Workflows.md b/pages/Workflows.md index 48bda382d0f076abfaa62af3e10149e07d663040..8fdd493f99ef2d50cbf2ab370c8ff86a3bdd9720 100644 --- a/pages/Workflows.md +++ b/pages/Workflows.md @@ -1,3 +1,4 @@ +--- Title: Workflow management Author: Jutta, Tamas --- diff --git a/pages/grafik.png b/pages/grafik.png new file mode 100644 index 0000000000000000000000000000000000000000..3588a2b47c18e1eec03e92c120f6e6594f07db90 Binary files /dev/null and b/pages/grafik.png differ diff --git a/pages/grafik_1.png b/pages/grafik_1.png new file mode 100644 index 0000000000000000000000000000000000000000..a4ac301b5eb57718d35111ed003e65eb7ccb847f Binary files /dev/null and b/pages/grafik_1.png differ