Integrate Online Acoustic Monitoring script for ORCA site

mentioned in merge request !3 (merged)

created merge request !3 (merged) to address this issue

It would be nice if you'd bring the script into shape, which should not be that much work. If you look at this example: https://git.km3net.de/km3py/km3mon/blob/master/scripts/ahrs_calibration.py

You see that the script has a module doc string which defines the usage including all the options:

"""
Runs the AHRS calibration online.

Usage:
    ahrs_calibration.py [options]
    ahrs_calibration.py (-h | --help)

Options:
    -l LIGIER_IP    The IP of the ligier [default: 127.0.0.1].
    -p LIGIER_PORT  The port of the ligier [default: 5553].
    -d DET_ID       Detector ID [default: 29].
    -o PLOT_DIR     The directory to save the plot [default: plots].
    -h --help       Show this screen.

"""

It should at least accept an option for the plot directory -o PLOT_DIR and a detector ID -d DET_ID as this is provided by the monitoring system.

Can you please push the script to the branch I just created instead of attaching it, so we can continue with the code review?

It's fairly straightforward:

git clone git@git.km3net.de:km3py/km3mon
cd km3net
git checkout 36-integrate-online-acoustic-monitoring-script-for-orca-site
cp /path/to/Online_Acoustic_Monitoring.py scripts/acoustics.py
git add scripts.acoustics.py
git commit "Add acoustics monitoring"

Thanks in advance.

added awaiting feedback enhancement labels

I ran the command git clone git@git.km3net.de: km3py / km3mon without problems.

Then I guess you wanted to write cd km3mon.

The next two commands (git checkout 36-integrated-online-acoustic-monitoring-script-for-orca-site and cp /home/carlo/Online_Acoustic_Monitoring.py scripts / acoustics.py) did not give problems.

Instead, I get an error when I run:

git add scripts.acoustics.py -> fatal: pathspec 'scripts.acoustics.py' did not match any files

git commit" Add acoustics monitoring " -> error: pathspec 'Add acoustic monitoring' did not match any file (s) known to git.

I checked and the file acoustics.py appears correctly pasted on my computer's local folder

Thank you,

I apologize in advance for my inexperience!

I think I managed to do what you asked me.

Carlo

Sorry for the late answer!

OK well done! You have done the git add ... and git commit -m ..., but you still need to synchronise your repository to this server (I don't see any new commits on the server yet). Currently everything is still present on your local machine

mentioned in merge request !5 (closed)

created merge request !5 (closed) to address this issue

@cguidi I tried to run the script but I get this error on the ORCA site:

[off1user@antorcaoff1 v2]$ python scripts/acoustics.py -n 1 -d 44 -o plots/
1579176786.1433704
2020-01-16 12:13:08 WARNING ++ DBManager:
2020-01-16 12:13:08 CRITICAL ++ km3pipe.db: No DOM found for OMKey '(1, 0)' and DetOID '44'.
Traceback (most recent call last):
  File "scripts/acoustics.py", line 72, in <module>
    domID = db.doms.via_omkey((du, dom), detid).dom_id
AttributeError: 'NoneType' object has no attribute 'dom_id'

I'll try to figure it out and push the changes to !5 (closed)

I fixed all errors but now it complains that no data is available:

2020-01-16 12:27:37 ERROR ++ km3pipe.db: No data found at URL 'streamds/toashort.txt?detid=44&minrun=7171&maxrun=7171&domid=808432849&emitterid
=14'.
2020-01-16 12:27:37 ERROR ++ km3pipe.db: No data found at URL 'streamds/toashort.txt?detid=44&minrun=7171&maxrun=7171&domid=808476701&emitterid
=16'.
2020-01-16 12:27:37 ERROR ++ km3pipe.db: No data found at URL 'streamds/toashort.txt?detid=44&minrun=7171&maxrun=7171&domid=808432836&emitterid
=16'.
2020-01-16 12:27:37 ERROR ++ km3pipe.db: No data found at URL 'streamds/toashort.txt?detid=44&minrun=7171&maxrun=7171&domid=808949646&emitterid
=16'.
2020-01-16 12:27:38 ERROR ++ km3pipe.db: No data found at URL 'streamds/toashort.txt?detid=44&minrun=7171&maxrun=7171&domid=808483678&emitterid
=16'.
2020-01-16 12:27:38 ERROR ++ km3pipe.db: No data found at URL 'streamds/toashort.txt?detid=44&minrun=7171&maxrun=7171&domid=808476737&emitterid
=16'.

Is this what you meant lately, or is something else not working correctly?

In this case it does not find data because it is looking for data relating to DU 1.

Currently it is possible that it gives similar errors also for the other DUs and it has due to what I was telling you about. Simply the little square corresponding to the piezo in question will be black in the plot.

Now it crashed with:

Traceback (most recent call last):
  File "scripts/acoustics.py", line 191, in <module>
    ind = np.where(DUs[1] < 1000)
IndexError: list index out of range

Btw. I called the script with python scripts/acoustics.py -n 1 -d 44 -o plots/

Here are the changes I needed to make to get it running: !5 (diffs)

I think you need to change

ind = np.where(DUs[1] < 1000)

with:

ind = np.where(DUs[0] < 1000).

Now it should work.

As detid I used "D_ORCA005". I don't know if it's the same to use "44".

You have to obtain a plot like this:

https://elog.km3net.de/Operations+FR/3652

44 is safe because I changed your code to accept both From the monitoring system you'll get the det_id passed, which is the number (the det_oid is the string representation of the detector, D_ORCA005)

OK, I'll try it later, I have to drive home now... Thanks for checking.

@tgal Is there any news? Does the program work properly?

Thank you,

Cheers,

Carlo

Yep I tried it but got some other issues

I will report at latest on Monday!

changed due date to January 27, 2020

I actually launched the script with the modification and it now hangs here:

2020-01-23 18:47:11 ERROR ++ km3pipe.db: No data found at URL 'streamds/toashort.txt?detid=44&minrun=7210&maxrun=7210&domid=808945629&emitterid=16'.
2020-01-23 18:47:11 ERROR ++ km3pipe.db: No data found at URL 'streamds/toashort.txt?detid=44&minrun=7210&maxrun=7210&domid=808483667&emitterid=16'.
2020-01-23 18:47:11 ERROR ++ km3pipe.db: No data found at URL 'streamds/toashort.txt?detid=44&minrun=7210&maxrun=7210&domid=808432848&emitterid=16'.
2020-01-23 18:47:12 ERROR ++ km3pipe.db: No data found at URL 'streamds/toashort.txt?detid=44&minrun=7210&maxrun=7210&domid=808432849&emitterid=16'.
1579801632.5239863
9.18619418144226

Is this ok?

It is not actually blocked. It is waiting for 10 minutes with respect to when it begun to run the first time. After 10 minutes it does again the whole calculation and generates a new plot.

This is the plot after running it for an hour:

Yes, the plot is how it should be. In fact, all the piezos of DU1 did not receive any impulse. This is the reason why they are black. Probably, if you run the program with 5 DUs you will have the same result for all the lines, or at most very few red / yellow / green squares. Unfortunately, the current situation is really this. For some reason not all data is saved in the database. These days I check in the raw acoustic data whether the pulses are actually emitted by the beacons or not. In any case, the program gives the correct output.

The program should generate a plot like that every 10 minutes.

@tgal I have seen in the elog https://elog.km3net.de/Operations+FR/3821 that the convention for the numbering of the DUs has been changed. So I should make some changes to the program. I thought I would no longer ask the user for the number of DUs but directly show a plot with all the DUs that are at sea at that time. I have already made changes to the version I have of the program on my PC and it works. Where can I modify the latest version you edited?

If I try to launch the latest version of the program (taken from "WIP: Resolve "Integrate Online Acoustic Monitoring script for ORCA site") I get this error:

AttributeError: 'CLBMap' object has no attribute 'omkeys'

Doesn't it happen to you?

Thank you,

cheers,

Carlo

Which KM3Pipe version are you using? CLBMap does have omkeys https://km3py.pages.km3net.de/km3pipe/autoapi/km3pipe/db/index.html?highlight=clbmap#km3pipe.db.CLBMap

OK thanks. I didn't have the latest version of KM3pipe. Now it works properly.

Btw. the latest version of you script is in master. However I am now thinking about removing it and merge it when it's working until it meets the requirements.

Yes, I should make the necessary changes to work with the new DUs names. I also found a small bug that I need to fix.

Hello @cguidi , are there news concerning the implementation?

Hello @aenzenhoefer

Yes, the program should now work properly for any number of strings. Has the problem with data that was saved late in the database been resolved? I'm still doing some tests and then I'll send you the most updated version of the program. Where should I upload it?

The DB access seems to work more reliably lately.

Currently only the latest two or three minutes of data remains on the server.

I think it would be nice to include it in the general monitoring repository. Please @tgal provide information where to put it

Hello @tgal,

the program is ready. It works with any number of lines. At this moment it provides an image of the situation every 20 minutes. This is because it takes a little less than 20 minutes on my computer to complete the analysis. I don't know how long it will take when it runs on the server. In any case we can leave this value in my opinion. Or we can make sure that this value is settable by the user. Where should I upload the program? Here? !5 (closed)

Thank you,

cheers,

Carlo

Thanks Carlo, just commit it to whatever branch you like and create a merge request. We can then review it and integrate it.

created merge request !8 (merged) to address this issue

mentioned in merge request !8 (merged)

closed via merge request !8 (merged)

mentioned in commit f18abb6b

mentioned in merge request !9 (merged)

mentioned in commit 480f8f17

mentioned in merge request !10 (closed)

mentioned in issue #42 (closed)

mentioned in commit 9a7dd6f4

Integrate Online Acoustic Monitoring script for ORCA site

Designs

Child items ...

Activity