projects:micado:dfsdossier [OmegaCEN]

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
{{NEWPAGE}}
/********************** DON'T EDIT ABOVE THIS LINE **********************/

MICADO DFS dossier


To recap, my baseline idea is the following:

1.Write our data reduction software (DRS) deliverables to ESO to their
specs: something like attached vlt-spe-eso*.pdf is expected. So these
are the pipelines.



2.Take these pipeline products and embed it into a prototype data
handling system (DHS) to support our work in the development and testing
phase of the pipelines.  This way the pipelines get connected to an
archive from which they can take input data (simulated or real) and
store their results. The DHS can be accessed via web services and
command-line for archiving and processing. The webservice does not
require a
local installation of software, the command-line does require a
local installation.

So the system yields an environment in which we (Austria and NL) can
collaborate during developing, testing, verifying and validating the
pipelines and have a common archive of the simulated data and lab test
data.

Having such an prototype DHS turned out to be very handy for the DRS development phase
for MUSE, as explained in the presentation by Rees Williams.



Expertise to build and run the DHS is available in Groningen. My idea is
that as a start we get such a prototype DHS running at Groningen which
Austria and Groningen can both use. This relieves each Austrian
institute from having the expertise and meeting the technical
specifications to run it.

Austria can access the Groningen installation via webservices and
command-line (via remote login on our Groningen network). So having the
DHS only installed in one place is perfectly doable to collaborate. It
only means all hardware connected (a database, storage and compute
resources) have to be physically there.  If Austria wants to pool in
such resources for MICADO we can install it in Austria as well. And if
Austria wants to build up DHS expertise as well that is also an option.
So do not get me wrong: I am very positive towards the idea of Austria
also installing the DHS in general.



If we go for the only-at-Groningen scenario as a start what is left are
the technical specifiations to embed the DRS into the DHS
prototype. What is needed is an xml definition of the DRS design and
knowledge of python. 

The xml modelling is needed to map the DRS onto the DHS system design in
a straightforward way re-using mapping tools we built for MUSE. The DHS
is written in python (i.e.
as a 'glue' language) using its object oriented programming approach.
Lastly, the DHS takes its DRS software from a code repository. Currently
we use CVS.



If Austria wants to build-up DHS expertise rather sooner than later I am
happy to go more into the technical specifications of the DHS in


So now to the technical specifiations for the DHS.


The DHS consists of a software repository, a database, a dataserver and Distributed Processing Unit
The DHS stores the pipeline software in a software repositories that can be accessed across institutes (currently we work with CVS for the repository).

The database is an Oracle database (but can become a different one, e.g., MySQL if we would want to).

(software that allows to connect the DHS to storage hardware, irrespective of the type of hardware and the physical location  of the hardware)

 (software that allows to connect the DHS to compute clusters,  irrespective of the type of hardware and the physical location  of the hardware). These components are used by webservices that create a weThe dataserver is the core component of the archive which we have My idea would be that we start with the database, dataserver and Distributed Processing Unit in Groningen only and that Austria connects to this via webservices





Once pipeline development has frozen we can turn the prototype into a real data handling system for commissioning and GTO data handling. Again, MUSE has done this.


Now on time-lines.