projects:kids:atlas [OmegaCEN]

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
{{NEWPAGE}}
/********************** DON'T EDIT ABOVE THIS LINE **********************/

= ATLAS Survey

This is the ATLAS portal for the KiDS Survey Prodcution team. We want to use the ATLAS data to improve the KiDS survey photometric homogeneity (using overlaps) and (2) support KiDS science cases which are also applicable to ATLAS (for now just the KiDS Strong-Lensing project). 

== Status (last update 14 Apr 2015)

See [[https://docs.google.com/presentation/d/1_LWxm17k6fAFb2w3nbjMmksYNXCNXp3Ed-yVlmD3cF8/edit?usp=sharing | ATLAS slides]]

We are producing u,g,r,i (not z!) ATLAS coadds re-using/adapting the KiDS pipeline.  

== Gaps in coadded data

Gaps between CCDs; in the release notes of ATLAS {{:projects:kids:atlas_dr1.pdf|}}: is written that in data observed before November 2011 the vertical gaps between the CCDs are not always filled in well because of the way the observations were executed. I suppose this is what you are seeing.

== South-West secondary star diagnosis

Matt Owers notes:
- The issue I noted earlier with the small secondary star ~2" to the
southwest of brighter stars only appears in one g-band image. It is odd
that the secondary star is not the same brightness as the standard and I
can't really think what might cause this.

[[http://quality.astro-wise.org/QualityWISE?object_str=CoaddedRegriddedFrame&object_id=1360ba421d1f42f1e053c016a9c3851e&project=ATLAS | example case, g-band ATLAS data with  rawfitsdata with DATE-OBS=set([datetime.datetime(2011, 9, 25, 1, 9, 10), datetime.datetime(2011, 9, 25, 1, 7, 35)])
]]:


**Diagnosis**: poor image quality in raw science exposures. 3 Figures below show a zoom in on an area in the example case:
 
{{:projects:kids:coaddatlas.png?800|Coadded calibrated image}}
{{:projects:kids:rawfits1.png?800|RawFitsData exposure 1}}
{{:projects:kids:rawfits2.png?800|RawFitsData, exposure 2}}

The RawFitsData with the "secondary star" has DATE-OBS=2011-09-25T01:09:10.066.

**DETECTION**: most direct on RawScience. Probably using PSF anisotropy (high ellipticity with fixed PA) information. In absence of PSF anisotropy information of RawScienceFrames on can look at it for a [[http://quality.astro-wise.org/QualityWISE?object_str=ReducedScienceFrame&object_id=12D50F3E66484BB3E053C016A9C3FF8A&project=ATLAS&progenitors=1#progenitors | ReducedScienceFrame with 
DATE-OBS=2011-09-25T01:09:10.066]]. 

{{http://ds.astro.rug.astro-wise.org:8000/QC-JMCFARLAND-OMEGACAM-------OCAM_g_SDSS-ESO_CCD_%2396---PSF_Anisotropy-ReducedScienceFrame-57115.6735075-04b68968a6ed53483c0909d1c30538bcb0d3753e.png?800}}

**ASTROMETRY NOTE**: this appears as one reason for bad astrometric residuals. See the correlated global astrometric parameters in the [[http://quality.astro-wise.org/QualityWISE?object_str=CoaddedRegriddedFrame&object_id=1360ba421d1f42f1e053c016a9c3851e&project=ATLAS | example case, g-band ATLAS data with  rawfitsdata with DATE-OBS=set([datetime.datetime(2011, 9, 25, 1, 9, 10), datetime.datetime(2011, 9, 25, 1, 7, 35)])]].

== ATLAS tiles mapping onto Astro-WISE classes

(work in progress by Gijs)

Goal: define units of data for ATLAS in the same way as for KiDS. This ensures that whatever script/approach we develop that works on one will work for the other. 

First the ATLAS observing strategy: 
 * each of the ugriz filters in each tile is observed with 2 dithers
 * see link to ATLAS homepage below for more details.

Now the mapping of it on Astro-WISE classes
* Observations after 2011-10-31: each tile is observed in ugriz. Observations for each of the 5 filters maps onto a single ObservingBlock.
* Observations before 2011-10-31: more a mess. We are considering to clean it by creating manually same mapping as >2011-10-31

KiDS survey operations handle on units of data:
   * survey tile: data from all bands with a tile as defined by the survey. Example is the handling of multi-band productions (mult-band catalogs) and analysis (stellar locus, QSO/galaxy selection in color-color space)
   * ObservingBlock: all exposures that go into data from a single band. Example is the creation of coadded images and source extraction of those.
   * Template?
   * Exposure: a single exposure. 

For KiDS the mapping from survey tile to observing blocks to templates and exposures is straightforward:
   * survey tile->observing blocks: the tilename can be inferred from the observingblock name
   * observingblock->exposures: all exposures have as observingblock dependency the observing block. NB: there exist no additional exposures with that observingblock as dependency.

For ATLAS>2011-10-31
   * survey tile -> observing blocks: the tilename can be inferred from the observingblock name
   * observingblock->exposures: all exposures have as observingblock dependency the observing block. NB: there exist no additional exposures with that observingblock as dependency.  

== Validation set: tile overlapping with SDSS

ATLAS data (1 pointing position) overlapping with SDSS (SEGUE):

The DATE_OBS for the RawScienceFrames is:
2013-06-02 00:45:26
2013-06-02 00:43:42
2013-02-11 07:10:16
2013-02-11 07:11:53
2012-05-20 02:03:29
2012-05-20 02:04:54
2013-02-03 07:02:43
2013-02-03 07:04:18 

== First production set: Cluster: APMCC917

* Cluster information: [[http://simbad.u-strasbg.fr/simbad/sim-id?Ident=APMCC+917 | APMCC 917]]
* Wget export script: 
The DATE_OBS for the RawScienceFrames is:
2011-09-29 05:58:21
2011-09-29 05:56:49
2011-09-25 01:06:04
2011-09-25 01:04:31
2011-09-29 04:57:25
2011-09-29 04:55:54
2011-09-24 05:45:24
2011-09-24 05:46:54
2011-09-29 05:59:54
2011-09-29 06:01:24
2011-09-25 01:07:35
2011-09-25 01:09:10
2011-09-29 04:58:55
2011-09-29 05:00:29
2011-09-29 06:04:26
2011-09-29 06:02:56
2011-09-25 01:12:12
2011-09-25 01:10:41
2011-11-29 02:39:18
2011-11-29 02:40:43
2011-12-05 01:45:05
2011-12-05 01:46:30
2011-08-22 07:39:28
2011-08-22 07:38:01
2011-11-28 01:40:30
2011-11-28 01:41:56
2011-11-29 02:42:31
2011-11-29 02:43:55
2011-12-05 01:50:15
2011-12-05 01:51:40
2011-08-22 07:40:54
2011-08-22 07:42:21
2011-11-29 02:45:41
2011-11-29 02:47:06
2011-12-05 01:54:03
2011-12-05 01:55:28
2011-11-19 01:47:03
2011-11-19 01:48:31
2011-09-25 05:46:51
2011-09-25 05:45:25
2011-12-13 01:41:51
2011-12-13 01:43:18
2011-09-27 06:20:29
2011-09-27 06:21:54
2011-11-19 01:50:18
2011-11-19 01:51:47
2011-09-25 05:48:19
2011-09-25 05:49:44
2011-12-13 01:45:26
2011-12-13 01:46:57
2011-11-19 01:53:37
2011-11-19 01:55:03
2011-09-25 05:52:38
2011-09-25 05:51:13
2012-10-14 01:30:26
2012-10-14 01:28:44
2012-09-13 03:10:12
2012-09-13 03:08:29
2012-06-24 08:25:34
2012-06-24 08:23:54
2012-10-16 00:59:05
2012-10-16 00:57:21
2012-10-14 01:34:23
2012-10-14 01:32:39
2012-09-13 03:14:14
2012-09-13 03:12:32
2012-06-24 08:29:40
2012-06-24 08:27:56
2012-10-16 01:03:02
2012-10-14 01:38:16
2012-10-14 01:36:37
2012-09-13 03:18:07
2012-09-13 03:16:27

== Coadd QC

Report 2015-04-16: QC of 207 ATLAS coadds in SAMI cluster regions

^  QC check  ^  Nr OK  ^  Nr NOK  ^
| Cross-talk  |  207  |  0  |
| Bg subtr.  |  206  |  1  |
| Calibrations  |  207  |  0  |
| Astrom  |  162  |  45  |
| Photom  |  207  |  0  |
| PSF difference  |  196  |  11  |
^ Overal verdict  ^  156  ^  51  ^

**Astrometry statistics**

{{:projects:kids:astrom.png?500}}

Data points: blue = u; green = g; red = r; magenta = i
Dashed lines: QC limits adopted in qcCoadd_ATLAS.py

**PSF sizes and size variations**

{{:projects:kids:psfsizes.png?500}}

Data points: blue = u; green = g; red = r; magenta = i


== Data in hand

[[http://dbview.astro-wise.org/DbView?project=ATLAS&set_cookie=&QSort=&QDesc=descending&2ndQSort=&2ndQDesc=descending&numrows=rownum&fcontext_project_only=on&show_expanded=no&Exportselect=NoDownload&class_str=CoaddedRegriddedFrame&mode=table_res&depth=1&python_dict=%2528dp0%250AS%2527query%2527%250Ap1%250A%2528dp2%250AsS%2527expanded%2527%250Ap3%250A%2528dp4%250AS%2527CoaddedRegriddedFrame%2527%250Ap5%250A%2528lp6%250AS%2527open%2527%250Ap7%250AaS%2527root%2527%250Ap8%250Aag5%250AasS%2527CoaddedRegriddedFrame.instrument%2527%250Ap9%250A%2528lp10%250AS%2527prefilled%2527%250Ap11%250AaS%2527link_property%2527%250Ap12%250AaS%2527Instrument%2527%250Ap13%250AasS%2527CoaddedRegriddedFrame.filter%2527%250Ap14%250A%2528lp15%250Ag11%250AaS%2527link_property%2527%250Ap16%250AaS%2527Filter%2527%250Ap17%250AassS%2527derived%2527%250Ap18%250A%2528dp19%250As.&expanded_leaf=&expanded_leaf_type=&expanded_leaf_class=&derived_leaf=&node_state=&CoaddedRegriddedFrame.expanded=open&CoaddedRegriddedFrame.instrument.expanded=prefilled&CoaddedRegriddedFrame.filter.expanded=prefilled&CoaddedRegriddedFrame.is_valid.op=%3E&CoaddedRegriddedFrame.is_valid=0&CoaddedRegriddedFrame.instrument.name=OMEGACAM&CoaddedRegriddedFrame.instrument.name.op=%3D&CoaddedRegriddedFrame.regridded_frames.base=RegriddedFrame | coadds]]

== VST-ATLAS pipeline description

We have a first draft pipeline. The VST-ATLAS pipeline is derived from the KiDS production pipeline. 
A quick summary of the salient steps:

^step^description ^
| QC upon ingest| not done, we simply take the most recent exposures of a field to be the right ones (not using ESO's QC (grade A,B,C) |
| Readnoise | nightly OmegaCAM calibrations |
| Gainlinearity | regularly (~ 3 per month) OmegaCAM calibrations |
| Flat-fielding and illumination correction | re-using the calibrations made for KiDS | 
| Fringe correction]] | using same fringe maps as KiDS (not doing z) |
| Cross-talk | done, using our general OmegaCAM configuration|
| Background subtraction | row-by-row subtraction like KiDS | 
| Hot pixels | nightly OmegaCAM hotpixels (todo: check all are public) |
| Cold pixels | regular (~2-3 per week) OmegaCAM calibrations |
| Satellite maps | we do not know yet what is used (either single-frame based or 2-frame based |
| Photometric calibration | done, nightly ZPTS as much as possible, except for u: forever |
| Chip-to-chip photometric corrections | use fixed set (link to values will come)  |
| Exposure-to-exposure photometric homogenization | applied |
| Coadd-to-coadd photometric homogenization | no applied | |
| Astrometric calibration |done, global |
| Automatic Coadd masks | not done |
| Sourcelists | not done |
| Manual Coadd Masks | not done |

== References / handy links.

* [[ http://astro.dur.ac.uk/Cosmology/vstatlas/ | VST-ATLAS homepage]]
* [[http://dbview.astro-wise.org/DbView?project=KIDS&set_cookie=&QSort=&QDesc=descending&numrows=100&fcontext_project_only=off&show_expanded=no&Exportselect=NoDownload&class_str=ObservingBlock&mode=table_res&depth=1&python_dict=&expanded_leaf=&expanded_leaf_type=&expanded_leaf_class=&derived_leaf=&node_state=&ObservingBlock.expanded=open&ObservingBlock.name.op=like&ObservingBlock.name=%25vst%25atlas%25 | Observing Blocks]]
* [[http://dbview.astro-wise.org/DbView?project=OMEGACAM%40VST&set_cookie=&QSort=creation_date&QDesc=descending&numrows=100&fcontext_project_only=on&show_expanded=no&Exportselect=NoDownload&class_str=RawScienceFrame&mode=table_qry&depth=1&python_dict=%2528dp0%250AS%2527query%2527%250Ap1%250A%2528dp2%250AsS%2527expanded%2527%250Ap3%250A%2528dp4%250AS%2527RawScienceFrame.chip%2527%250Ap5%250A%2528lp6%250AS%2527prefilled%2527%250Ap7%250AaS%2527link_property%2527%250Ap8%250AaS%2527Chip%2527%250Ap9%250AasS%2527RawScienceFrame.instrument%2527%250Ap10%250A%2528lp11%250Ag7%250AaS%2527link_property%2527%250Ap12%250AaS%2527Instrument%2527%250Ap13%250AasS%2527RawScienceFrame.filter%2527%250Ap14%250A%2528lp15%250Ag7%250AaS%2527link_property%2527%250Ap16%250AaS%2527Filter%2527%250Ap17%250AasS%2527RawScienceFrame%2527%250Ap18%250A%2528lp19%250AS%2527open%2527%250Ap20%250AaS%2527root%2527%250Ap21%250Aag18%250AassS%2527derived%2527%250Ap22%250A%2528dp23%250As.&expanded_leaf=RawScienceFrame.astrom&expanded_leaf_type=inline_property&expanded_leaf_class=Astrom&derived_leaf=&node_state=&RawScienceFrame.chip.expanded=prefilled&RawScienceFrame.instrument.expanded=prefilled&RawScienceFrame.filter.expanded=prefilled&RawScienceFrame.expanded=open&RawScienceFrame.OBJECT.op=like&RawScienceFrame.OBJECT=%25ATLAS%25&RawScienceFrame.chip.name=ESO_CCD_%2365&RawScienceFrame.chip.name.op=%3D&RawScienceFrame.instrument.name=OMEGACAM&RawScienceFrame.instrument.name.op=%3D#RawScienceFrame.astrom | RawScienceFrames (CCD #65)]]
* {{:projects:kids:fieldcentres_sgp.dat|}}
* 

== Scratch notes

* Check calibration pool settings are applicable for GTO projects
* Check GTO calibration pools are not compromised