Post mortem

 PM Data Collection and Storage c/o  Nikolai Trofimov         PM Home Page

• Post Mortem Meeting 4/10/2005 & 24/3/2006
• PM & Alarms c/o Niall
• Client API results of discussion on 6th and 21st of June
• Progress of prototyping
• Post Mortem meeting 15/03/05
• Meeting 25/1/2005
• Progress on BinX prototype

Example PM SDDS files FGC QPS1 QPS2

Recent talks

• Greg: 25th June 2005 - status of prototyping to AB CO TC
• Roberto 5th June 2005

Outline of proposed solution

• Client sends data to central PM server via client API
• Client data sent in binary, data structure described by BinX
• Data transfer to PM server via CMW
• Server collects and saves binaries.
• Another process collates and performs conversion to SDDS using BinX description to decode. (binary data kept)
• SDDS files are browsable and are input into analysis routines

Requirements

Key Use Cases:

1. Immediate diagnostics after beam loss - what happened.
2. Off-line analysis of PM event
3. MB quench
4. Decision support analysis: is it safe to continue (key example: Quench)

Scope: all relevant beam measurements and all equipment information.

• SCALABILITY
• RELIABILITY

Gateway and push

• Data collection - all systems: beam instrumentation, analogue, machine protection, equipment.
• Conversion to the format for transmission up should be simple and not load the real-time environment too much
• Should try and minimise  the size of data pushed [binary, zip...?]
• Naming conventions should be used.
• How long is the buffer - configurable, sample duration, sample frequency. Amount. lengths before and after should be variable.
• Triggers: external or internal, manual, test. External trigger by GM timing system [latency 1 ms.]
• Common time stamping format. Variable resolution 1 ms. to 1 ps.
• Streaming or pseudo-streaming might be required.

Collection

• Configuration: sectorizable
• Need to cater for mobile systems
• Parallel PMs - what happens if two PM happen almost synchronously?.

Processing

• Verification that every that should have reported back has reported back. Check that data is correct (or at least reasonable).
• Conversion to final file format
• Selective upload to database
• Monitoring, notification, status
• Initiate analysis

Analysis

• Automatic and customizable on-line analysis. Scan for faults. What happened first? Summary.
• Browser: select PM event, select any signals from different systems and plot, export to file (various format), print.  Pre-defined fixed signal selection sets. Appropriate timestamp handling, meshing different data sets - single time axis. Display of timestamp events (from PIC, BIC etc.). Usual dataviewer functionality: zoom, multiple graphs, rudimentary fitting.
• Easily accessible raw and result data
• Pull in logging data
• Pull in setting data
• Pull in alarm data

• Engineers will want to use the analysis tools from their offices - web based.

• Operation: with beam, without beam, test. Beam1 v. beam 2?

• Sub-system PM: beam dump - after every dump trigger. Not strictly PM.

Data Providers

• BIC
• PIC
• Power Converters - self triggering (turn off, interlock) - circular buffer in gateway, extra stuff in the FGC if required.
• RF power - PLC Lynx/SO front end level 3-4 Hz not too much data
• RF low level , VME, circulate  80 MSamples - 6 ms worth of data 11 Mb per cavity plus damper  100 Msamples 1000 turns 0.5 Gbytes  (PM of function generators - reference functions only). Plus slow logging - voltage amplitude. 
• RF - BIC trigger post-mortem data sent only when there's a beam dump
• HWC: Power converters plus QPS, PIC,
• LBDS

PIC

What triggers to push PM data? When BIC is not present? Woah.... What triggering  the PM?

QPS - Herve

• 21 Gigabytes for a sector quench
• Front-ends (DQAMG, DQAMS, DQAMC)
• Push data to gateway (DQGTW) - takes about 4 minutes
• Have test mode: 1 set of 3 front-ends or whole sector
• Event driven: quench, switch opening, faulty trigger.

MB Quench analysis   Quench Signal Analysis [Adriaan]

QPS data description   QPS PM buffer example

QPS - Main Bend Quench Use Case

Send buffer in bits - server has to pick up the pieces.

Here is the size for the Post-Mortem buffer:
For DQAMC type A: This is 2500 times:
. LHC Time stamp (long)
. Number of block (int)
. 7 doubles
. 1 int

For DQAMC type B: This is 2500 times:
. LHC Time stamp (long)
. Number of block (int)
. 8 doubles
. 1 int

For DQAMG: This is 2500 times:
. LHC Time stamp (long)
. Number of block (int)
. 40 doubles
. 10 int

Please not that currently we don't know if
we have to split the ints in bits, this will depend on the tool to
analyse the Post-Mortem data.

LBDS - Bren

The only particularity of the LBDS might be the max ~10s time we need for getting a green light from the PM system after a dump, for compatibility with the "Inject and Dump" mode....

Power converters c/o Steve

What data is required and at what frequency?

My intention at the moment is to buffer the status data that the gateways receive at 50Hz from all FGCs for an appropriate period in a circular
buffer and to send that out in the case of a post-mortem event.  If I can get away with this, then post-mortem can be handled at the gateway level
without needing to involve the FGCs.  If more data is required, or a higher resolution, then the gateways will have to send commands to the
FGCs to extract additional data.

Issues

1. What format do the clients send the data?    Need FESA, PVSS, other API      Andy, Steve, Herve, Franck, Sergio, Lionel
2. Write binary - do the number crunching at top level?
3. Mechanism - secure [CMW...?]  
4. Destination - where is it put. nfs/directory structure
5. Conversion to SDDS or whatever.  Browsable.
6. PM server - executive management - checks
7. Database upload - archival - possibly generic browser based on database.
8. System specific analysis tools - automatic - on-request
9. Pull in logging, alarms, settings

1. What bloody format do the clients put their data in?  XML v native v SDDS   Data tagging being key. Data items have to be named

2. Compressing and CMW seem OK.       Franck, Sergio

3. Put it on a file sever - set up hierarchy etc.

4. Final file data format. Conversion. WHAT???? XML  SDDS

Steve's comments

What format should the data be sent in?

I don't have a particularly strong opinion on this subject.  The candidates are SDDS and XML plus numerous other potentially suitable languages.  Unless a language requires a lot of work on the front-end side  to format the data, I think the decision should be made by the people who  will be responsible for the receiving side, since they will have to extract and manipulate the received data.

Ideally, something simple and easy to construct.

How should the data be sent?

The basic options are whether to write directly to the NFS file system from the front-ends or to use some other communications protocol.

With NFS we need to be careful as to its behaviour in the event of a network outage or the server being down.  On the plus side, it is a very simple solution.

Otherwise, a robust and reliable communications protocol should be found.  This means either implementing a library to be used on the front-ends or CMW in some form.

 

First suggestion... Mike (based on what they've put in place at RHIC)

Essentially:

• Freeze rolling buffers on receipt of post mortem event
• Collect buffers in the gateways
• Gateways write data to file system in the form of SDDS files in a generic directory (symbolic link, clock directory name etc.)
• Post-process by PM server to make ready for final use and free directory for subsequent re-use. Bang in some database info to let everyone know that event has been recognized and processing is underway. Editable data (operator comments etc)
• Other dependent processes notified
• Pull in logging data
• Pull in setting data
• Pull in alarm data
• Post-processing analysis preformed by dedicated servers (beam loss analysis)

Worry about:

• responsibility for making sure data is properly written
• failures during write process
• data correctly written & makes sense - sanity checks

Discussion [29/3/2004] of above including a look at power converters, QPS, and SDDS (Notes by Robin)

Second suggestion - Sergio

Here some details on my tests.

1 - The client creates a document XML using the library libxml2 and some basic functions (C/C++) which I developed. The document is validated using XSchema (Optional). 2 - The document is compressed using the library zlib. 3 - The document is then sent to the PM server by using the CMW. The functions are written in C/C++. The maximum size is 500 Kbytes in compressed mode, which corresponds to approximately > 3 MBytes of document XML. 4 - The document is uncompressed at the time of its arrival in the PM server. The PM, server is written in Java. 5 - The document is validated using XSchema (optional) 6 - The document is saved in the local data base

The following step is to integrate the few basic functions (C/C++) in the QPS and to make tests.

I also made some research on the tools for analysis. I found that the CERN develops with other laboratories a product which is called JAS3 (Java Analysis Studio). I installed it and make some positives tests.

Third suggestion - Sergio

After our last meeting on postmortem, the use of the CMW and the data compression, to transmit the data of postmortem were reasonable.

The only point still outstanding was the use of the SDDS format! As I explained during the various meetings, it is an old format which does not have the potential of the XML. Moreover, it is a "format propri�taire". For me, it is also essential to separate the format of the postmortem data and the tools of visualization and analysis.

I thus made some search and I found that the transmission and the use of data in binary format are a problem also for others such as the "datagrid", and a proposal was submitted to the W3C, the "Binary XML description language". The University of Edinburgh supports and develops a C/C++ library BinX and Java JNI interface for the use of "Binary XML". See http://www.edikt.org/binx/

Tests: I carried out tests of transfer of data between platforms Intel and PowerPC and its works very well. I read the binary data with BinX library, and then I transformed the binary data into XML also with the BinX library. Then, with the data in format XML, I transformed them in the XML AIDA format with a XSLT and then visualized in JAS3 for analysis.