CERN Accelerating science

Requirement: persist measurements for later analysis, retrieval into application, post-mortem etc. Want to be able to get into MAD, Excel, Mathematica, ROOT etc, Standard mechanisms required.

Have to bear in mind the need to cross-correlate with logging data etc.

BDIFF aka ROSALI:

BI have come up with a format to service SPS requirements. Design to be read into Excel. XML light. Tab separated variables for this reason. BI would like to store more than one table type in the same file. Example here.

The main purpose of this project is to provide an Machine Development tool application to:

* define and standardize BDI instrument outputs (formats profiles, multi-profiles, images, multi images) compatible with excel, Mathematica, MAD...
* provide the C++ library to save directly in the FE and Java libraries to get these data from the FE and save them under these formats
* based on this, make the ROSALI Java tool (VMET like) that will:
o allow to build a sequence of actions
o allow to execute, save and load these sequence of actions
o provide basic standard actions based on these types of files as input/output
o provide an easy way to add standard actions
o provide an easy to insert user defined Java actions
o provide standard dataview at each step of the loop.
o eventually provide an easy way to have standard & user defined Mathematica actions
o eventually provide an easy way to have standard & user defined MAD actions
o eventually provide an easy way to launch excel

Lionel nearly choked when he saw it, and has a number of well founded objections to the format.

SDDS 

Good candidate, a lot of the issues addressed. See above link for a good introduction.

Comments by Stephen Page:

1. Can we define everything we need? XML will allow anything we like, but we would need to define some standards. SDDS has many things thought out already, but do we need anything on top of that?

2. How simply can the files be generated by the gateways? Either XML or SDDS should be trivial.

3. How simply can the data be extracted from the files for analysis /database storage?

4. What tools exist for data extraction / analysis from the files. XML will be strong from a parsing / editing point of view, but SDDS will perhaps offer better analysis tools and it is a fairly simple format anyway.

5. Flexibility of the languages. XML is almost completely open to us, which would imply some work to define a standard but we would not be
limited in what we could do. If SDDS fulfils our needs then it should be fine, but can we extend it or contribute to its development if we need more?

Personally I think SDDS looks most attractive for the moment as we would get a lot for free.

XML

[NO!!!] - Stephen Jackson's comment, likely to be too heavy if used rigourously.

Database

Metakit

Suggestion c/o Johannes. In his opinion more mature, better maintained, more options, better API, easily embeddable, etc, than SDDS.

ROOT

Scary. Could imagine converting to files that are ROOT readable though.