Tamas Gal (0667c6b3) at 23 Jan 10:29
Add Git access token to example.env
Tamas Gal (e18f0e0f) at 17 Dec 23:42
Improve logging and channel retrieval
OK, I replied to the mail thread but I would really like to have such discussions in a Git issue.
I am aware that there are two different purposes. For some of the online monitoring process I need the exact calibration which is used for data taking, to be able to track problems which might affect the triggering etc.
For online reconstruction and alerts, we indeed need something else which includes offline calibration and they will inevitably want to use something like best_detx_for_run(det_id, run)
to get something which is as best as it can be, regardlich of any definitions or bikeshed talks about when a run starts or ends. If the solution is to poll for the run - 1
OK, I don't like it but I can implement it.
By the way, I am just caught in a mail thread that escalated to @ddornic .
Tamas, you should be aware this is not a coding problem but this needs a decision. The online reconstruction is deliberately using offline calibrations. That seems weird to me. It is the typical situation for which we will later get blamed later.
OK, what is the current suggestion to get the calibration used for datataking for a given detector ID and run (no MCP access).
var cmd = new Oracle.ManagedDataAccess.Client.OracleCommand (
"select detectorid, run, runstarttime, runendtime, calibid, validfrom, validthrough, qtag, calibtype, calibrationtype, calibstart, calibend, tagtime, ranking from " +
"( " +
" select detectorid, run, runstarttime, runendtime, calibid, validfrom, validthrough, qtag, calibtype, o3.operation as calibrationtype, calibstart, calibend, tagtime, row_number () over (partition by detectorid, run, calibtype order by priority desc, tagtime desc) as ranking from " +
" ( " +
" select detectorid, run, runstarttime, runendtime, calibid, validfrom, validthrough, tagid as qtag, calibtype, calibstart, calibend, tagtime from " +
" ( " +
" select detectorid, run, runstarttime, runendtime, calibid, validfrom, validthrough, tagid, calibtype, calibstart, calibend, endtime as tagtime, row_number () over (partition by detectorid, run, calibtype, calibid order by endtime desc) as rnum from " +
" ( " +
" select r3.detectorid, run, r3.starttime as runstarttime, r3.endtime as runendtime, c1.operationid as calibid, validfrom, validthrough, o2.typeid as calibtype, o2.starttime as calibstart, o2.endtime as calibend from " +
" ( " +
" select r1.detectorid, r1.run, r1.starttime, min (r2.starttime) as endtime from km3sys.tb_detector_run r1 " +
" inner join km3sys.tb_operations o4 on r1.detectorid = :i_detectorid and r1.run = :i_run and o4.oid = r1.detectorid " +
" inner join km3sys.tb_operations o5 on o5.locationid = o4.locationid and o5.oid like 'D%' " +
" inner join km3sys.tb_detector_run r2 on r2.detectorid = o5.oid and r2.starttime > r1.starttime " +
" group by r1.detectorid, r1.run, r1.starttime " +
" ) r3 " +
" inner join km3sys.tb_calibrations c1 on c1.detectorid = r3.detectorid and validfrom <= r3.starttime and (validthrough is null or validthrough >= r3.endtime) " +
" inner join km3sys.tb_operations o2 on o2.oid = c1.operationid " +
" where exists (select* from km3sys.tb_calibration_tagging t1 where t1.calibrationid = c1.operationid and t1.typeid = 'T' and t1.tagid = 'OFFLINE') " +
" ) " +
" inner join km3sys.tb_calibration_tagging t2 on t2.calibrationid = calibid and t2.typeid = 'Q' and t2.tagid in ('OK', 'DEPRECATED') " +
" inner join km3sys.tb_operations o1 on o1.oid = t2.operationid " +
" ) " +
" where rnum = 1 " +
" ) " +
" inner join km3sys.tb_calibration_tags on typeid = 'Q' and qtag = tagid " +
" inner join km3sys.tb_operationtypes o3 on o3.oid = calibtype " +
") " +
wherestr, dbconn);
If you check the query code, these are all OFFLINE calibrations. It is very consistent with the fact that the run must be closed. I can provide a different entrypoint or selector to get the online ones. In that case, the selection would be
"ONLINE calibrations that are valid at the beginning of the run"
but then the Calibration group should act accordingly.
Well, I am simply seeking for a solution: get the best calibration we can think of for a run using a function call. This has to be the hardest problem of 2023.
So the only way to get that is by looking at the runtable?
Your choice...
The point is that the online group is probably mixing ONLINE and OFFLINE calibrations for their purposes.
Not even the one used for data taking?
So there is absolutely no valid latest calibration for such a run?
Indeed. So you have to change detx_for_run to issue an error.
For the sake of completeness:
def detx_for_run(det_id, run, version=5):
"""Retrieve the latest valid calibrated detector file for given run"""
api = APIv2()
cals = api.RunCalibration(DetOId=todetoid(det_id), Run=run, Ranking=1)
calibration_ids = dict()
# type is e.g. "COMPASS_CALIBRATION" or "STATUS_CALIBRATION", corresponding to "ccal" or "scal"
for cal in cals:
calibration_ids[cal["CalibrationType"]] = cal["CalibrationId"]
return detx(
det_id,
pcal=calibration_ids.get("DOM_POSITION_CALIBRATION", 0),
rcal=calibration_ids.get("DOM_ROTATION_CALIBRATION", 0),
tcal=calibration_ids.get("PMT_T0_CALIBRATION", 0),
acal=calibration_ids.get("ACOUSTIC_T0_CALIBRATION", 0),
ccal=calibration_ids.get("COMPASS_CALIBRATION", 0),
scal=calibration_ids.get("STATUS_CALIBRATION", 0),
version=version,
)
Then the default is in km3db. So km3db should issue an error.
The query asks for the calibset IDs, if that is empty for any of them, it defaults the corresponding calibset to 0. The calibset IDs are then passed to the detx
endpoint. What is the reasonable default value for those sets?
That's the point. Who/what is putting 0's as default values?
apiv2.0.0 retrieves the same (notice I changed the run number between queries to show how it appears).
{
"Data": [],
"DataType": "RunCalibration",
"Encoding": "NativeJSON",
"ProvenanceInfo": {
"SystemInfo": "dbwebserver: KM3WebService, Version=2.0.21348.46227, Culture=neutral, PublicKeyToken=null 20231204-1 57dd4d111cb6e91c19083c105e82c3ab58119dc9",
"Configuration": "",
"UUID": "1ba04963-72a1-4632-81c1-8a718d0130ee"
},
"Error": {
"Message": "Warning: no data found. If this is unexpected, please check your inputs.",
"Code": "OK",
"Arguments": null
},
"Start": "2023-12-14T09:50:24.027644Z",
"Stop": "2023-12-14T09:50:24.047465Z",
"Inputs": [
{
"UUID": "d371706b-91ea-2100-5214-f14e18169c51",
"URL": "arg:DetectorId=D0ORCA018"
},
{
"UUID": "e5fff318-a4e1-5ab2-e26c-7ac912f4ecba",
"URL": "arg:Ranking=1"
},
{
"UUID": "62f99c42-e83a-ca16-50ff-49358a030861",
"URL": "arg:Run=19616"
},
{
"UUID": "a4a2bfae-3ee3-1cac-8c6e-92fad330a1b9",
"URL": "db:RunCalibration"
}
],
"Outputs": [],
"IsEmpty": true,
"APIVersion": "2.0.0"
}
I am trying to retrieve the calibset IDs from https://km3netdbweb.in2p3.fr/apiv2.1.0/RunCalibration/s?DetOId=D0ORCA018&Run=19616&Ranking=1
and the output is empty while Code: "OK"
.
The 0s are then the default values. Why is apiv2.0.0
returning a non-empty output? Is that change according to the new(?) definition of validity?
You are using APIv2.1.0, and that's correct. If you can reproduce the 0's, please trace where they come from. Apparently not from the API.