MICADO DFS dossier

To recap, my baseline idea is the following:

1.Write our data reduction software (DRS) deliverables to ESO to theirspecs: something like attached vlt-spe-eso*.pdf is expected. So theseare the pipelines.

2.Take these pipeline products and embed it into a prototype datahandling system (DHS) to support our work in the development and testingphase of the pipelines. This way the pipelines get connected to anarchive from which they can take input data (simulated or real) andstore their results. The DHS can be accessed via web services andcommand-line for archiving and processing. The webservice does notrequire alocal installation of software, the command-line does require alocal installation.

So the system yields an environment in which we (Austria and NL) cancollaborate during developing, testing, verifying and validating thepipelines and have a common archive of the simulated data and lab testdata.

Having such an prototype DHS turned out to be very handy for the DRS development phasefor MUSE, as explained in the presentation by Rees Williams.

Expertise to build and run the DHS is available in Groningen. My idea isthat as a start we get such a prototype DHS running at Groningen whichAustria and Groningen can both use. This relieves each Austrianinstitute from having the expertise and meeting the technicalspecifications to run it.

Austria can access the Groningen installation via webservices andcommand-line (via remote login on our Groningen network). So having theDHS only installed in one place is perfectly doable to collaborate. Itonly means all hardware connected (a database, storage and computeresources) have to be physically there. If Austria wants to pool insuch resources for MICADO we can install it in Austria as well. And ifAustria wants to build up DHS expertise as well that is also an option.So do not get me wrong: I am very positive towards the idea of Austriaalso installing the DHS in general.

If we go for the only-at-Groningen scenario as a start what is left arethe technical specifiations to embed the DRS into the DHSprototype. What is needed is an xml definition of the DRS design andknowledge of python.

The xml modelling is needed to map the DRS onto the DHS system design ina straightforward way re-using mapping tools we built for MUSE. The DHSis written in python (i.e.as a 'glue' language) using its object oriented programming approach.Lastly, the DHS takes its DRS software from a code repository. Currentlywe use CVS.

If Austria wants to build-up DHS expertise rather sooner than later I amhappy to go more into the technical specifications of the DHS in

So now to the technical specifiations for the DHS.

The DHS consists of a software repository, a database, a dataserver and Distributed Processing UnitThe DHS stores the pipeline software in a software repositories that can be accessed across institutes (currently we work with CVS for the repository).

The database is an Oracle database (but can become a different one, e.g., MySQL if we would want to).

(software that allows to connect the DHS to storage hardware, irrespective of the type of hardware and the physical location of the hardware)

(software that allows to connect the DHS to compute clusters, irrespective of the type of hardware and the physical location of the hardware). These components are used by webservices that create a weThe dataserver is the core component of the archive which we have My idea would be that we start with the database, dataserver and Distributed Processing Unit in Groningen only and that Austria connects to this via webservices

Once pipeline development has frozen we can turn the prototype into a real data handling system for commissioning and GTO data handling. Again, MUSE has done this.

Now on time-lines.