Recent Changes - Search:

Running HITS

The information of this page applies only for the Astro HPC Software Stack integrated by the SciCom Group and the AstroLab at the Center for Mathematical Modeling. There is no guarranty of any of this procedure will work in other systems different than the HPC systems described within this page.

Nowadays (Jun, 2014) the Astro-HPC Stack is loaded as share resource over a lustre (1.8.4) filesystem in the HPC system levque.dim.uchile.cl in the path /home/apps/astro. Further details, check Astro HPC Software Stack.

General processing Scheme


"HITS processing scheme"
  1. Data Acquisition: images are captured at the telescope

Data Organization

PathRoleDescription
/home/apps/astro/INBOXIncoming raw filesMosaic input files to be processed (fits.fz)
/home/apps/astro/ARCHIVEArchiving of processed filesimages per CCD classified per object, MJD, CCD and process
/home/apps/astro/DATAMapped DataCCDs mapped by Field/Epoch/CCD/image
/home/apps/astro/SHAREDShared DataRandom Forest Classifier, catalogs, stamps, PSFs among other shared files required for the image subtraction algorithm
/home/apps/astro/tmpLocal DataReference images (raw, crblaster)
/home/apps/astro/WEBWeb resultsCandidates per CCD for visual inspection
/home/apps/astro/logsExecution Logsjobs execution logs: doastro, crblaster, doastro-revisit

Mapping

The semantics of the mosaic images obtained from the telescope are a bit different from the semantics used by the Image Subtraction Procedure (ISP). There are three big differences among them:

  1. The observed area of the sky observed by the telescope is called Object according to the mosaic fits header. The ISP calls it Field.
  2. The time when the telescope captured the mosaic image is called MJD-OBS, a Julian date formatted time-stamp. The ISP calls Epoch to the enumeration of the MJD-OBS series ordered in a non-decreasing order.
  3. The CCD name of the mosaic image used by the ISP is called EXTNAME (Extension name) in the mosaic fits image.

As the epoch field is non-deterministic (depends on the order of arrival of files), individual images are archived by OBJECT, MJD-OBS and EXTNAME in the ARCHIVE directory, and mapped to FIELD,EPOCH,CCD Name within the Data directory.

The mapping functions to translate Field<->Object, Epoch<->MJD-OBS and CCD Name<->EXTNAME are implemented in the file mapping-functions file located at the /home/apps/astro/sbin directory

The files are indexed by OBJECT, MJD-OBS and EXTNAME in the ARCHIVING, and by FIELD, EPOCH, CCD in the DATA directory.

There is an Object mapping file for specifying a custom mapping between Object and Field names. This file is located at /home/apps/astro/etc and consists of two columns indicating the former the Object name and the later the Field name.

If no mapping between Object<->Field name is provided, we consider the Object name as the Field name.

Scripts

doArchive

Archive all the mosaic images located at the INBOX directory.
./doArchive {list of mosaic images}

doMapping

Create the mappings for all the CCDs archived in the ARCHIVE directory.
./doMapping [-m mapping file] -o {object} [-e epoch|MJD]

checkData

Check the existence of result files in the ARCHIVE directory. The noraw options avoids to report missing raw images. ./checkData -o {field} -e {epoch|MJD} -c {ccd} -O {noraw}

checkMapping

Check the correctness of archived data mappings. No arguments required. ./checkMapping

clearMapping

Clear the existing mapping done in the DATA directory (Mapped data) ./clearMapping

pushData.sh

Script used to "push" data from the telescope to the dataReceiver daemon running in the HPC system where HITS is installed. The script defines some execution parameters such as the source_dir scanning frequency or the user and remote host used to transfer the mosaic images. One important thing is the public ssh key of the observer account (in the DECAM) shall be authorized in the remote HPC system (in the .ssh/authorized_keys file) in order to allow a secure transfer of data without prompting passwords.
./pushData.sh {source_dir} {remote_dir}

removeObject

Remove an Object from the system, meaning remove ARCHIVE, MAPPED, SHARED, results and temporal data related to the given object or field.
./removeObject {field}

renameField

Rename a field name from the ARCHIVE, Mapping and results directory.
./renameField {mapped_field_name} {mapped_new_field_name}

renameFitsObject

Rename an object name in a mosaic image, modifying the header of each HDU within the fits. ./renameFitsObject {fits_file} {old_obj_name} {new_obj_name}

startReceiver

Start the data receiver daemon. This scripts works as a deamon, detaching a process which archive and index a new mosaic image file each time a new file is detected in the INBOX directory. Also this daemon submit the parallel calibration and processing jobs to process the incoming mosaic image.
./startReceiver

stopReceiver

Stops the data receiver daemon.
./stopReceiver

simulateUpload

Simulate the upload process from the telescope of mosaic images. In other words, take a set of fits files and copy them into the INBOX directory at a regular time interval. So, the data receiver daemon detects the new file and process it.
./simulateUpload {object} {source_directory}

Job Scripts

mkCalibrations.sge

Create Bias master file, dome flats and masks with calibartion images from DECam. qsub mkCalibrations.sge {bias_obj_name} {dome_obj_name} {ccd_list}

crblaster.sge

Remove cosmic rays from an image (a single CCD) by using CRBlaster
qsub crblaster.sge {object_name} {epoch|MJD} [ {CCD list} ]

doastro.sge

Image Substraction and transient object detection Procedures.
qsub doastro.sge {object_name} {epoch|MJD} {reference epoch} [ {CCD list} ]

doastro-revisit.sge

Computes the Photometric solution for all images by each detected candidate.
qsub -t {start:end} doastro-revisit.sge {obj_name} {ref.epoch} [ {CCD list} ]
The parameters start and end denotes the CCD index to revisit.
E.g. qsub -t 1:10 doastro-revisit.sge MyObject 2 N1 N2 N3 N4 N5 N6 N7 N8 N9 N10
submits 10 jobs, each one revisiting one CCD from the given CCD list.

Utilitary Scripts

doRevisit

Submit doastro-revisits jobs in batchs of batch_n+1 - batch_n jobs.
./doRevisit {Object} {batch_1 batch_2 ... batch_n}

recomputeObject

Submit the jobs to recompute a Field/Object. Epoch and CCD parameters are optional, meaning if no Epoch or CCD are given, all Epochs or CCDs are recomputed respectively.
./recomputeObject -o {field} [ -m {epoch/MJD} ] [ -c {CCD}]

Configuration

The configuration of a HITS execution is done in the file /home/apps/astro/sbin/astro.cfg, where several parameters are set.

ParameterDefault ValueDescription
ARCHIVE/home/apps/astro/ARCHIVEArchiving directory.
MAPPING/home/apps/astro/etc/DECAM-2012.mappingObject<->Field mapping file.
MAPPER/home/apps/astro/DATAMapped Data.
INCOMING_DIR/home/apps/astro/INBOXRaw mosaic images directory.
SHARED_DIR/home/apps/astro/SHAREDShared Data directory.
MEM_DIR/dev/shmFast memory filesystem (local for each compute node.
ARRIVAL_RECORD/dev/shm/incoming_donetemporal FIFO list file indicating processing order of mosaic files.
WORKER_QUEUE/dev/shm/worker_queueClassification and Archiver worker queue.
ERROR_CODES/home/apps/astro/etc/error_codes.txtISP Error codes description.
INOTIFYWAIT/home/apps/astro/bin/inotifywaitFilesystem monitoring application (from gnome inotify library)
PIMCOPY/home/apps/astro/bin/pimcopyParallel imcopy binary for CCD extraction.
IGNORE_CCD"FN1 FN2 FN3 FN4 FS1 FS2 FS3 FS4"List of CCDs to be ignored by the ISP.
REFERENCE_EPOCH2Epoch of the image to be used as reference to perform the image subtraction. 2 means the second mosaic image captured for each field.
BIAS_OBJBiasName of the Bias Object used in the telescope calibration process.
DOME_OBJdomeflatName of the dome flat object used in the telescope calibration process.

Pimcopy options:

  1. -i input.fits.fz : input file
  2. -o output_dir: output directory
  3. -d : create directories with CCD names when decompressing
  4. -t tmp_dir : create files in temporary directory and then move them into ouput directory
  5. If options -d and -t option are used, directories are created only in temporary directory and individual files are moved into output directory without any directories

Non real-time execution example:

Shortcut for 2013A data:

  • /home/apps/astro/sbin/runpipe Blind13A Blind_1 all or /home/apps/astro/sbin/runpipe Blind13A Blind_1 N5, where 'all' indicates that all CCDs will be processed and 'N5' indicates that only CCD N5 will be processed.

Shortcut for 2014A data:

  • /home/apps/astro/sbin/runpipe Blind14A Blind14A_01 all or /home/apps/astro/sbin/runpipe Blind14A Blind14A_01 N5, where 'all' indicates that all CCDs will be processed and 'N5' indicates that only CCD N5 will be processed.

Note the different arguments because of the different field naming convention between 2013A and the other semesters.

Detailed steps executed by the previous script can be done as follows:

  1. copy some raw or processed files from /share/astro-raw/DECAM-14A_process or /share/astro-raw/DECAM/DECAM-13A_process_xtalk to astro/INBOX. For 13A you can use, e.g.qsub -t 1:1 /home/apps/astro/sbin/copydata.sge Blind13A Blind_1 to copy, archive and map the data.
  2. check that you copied the correct files with describeFits * to check that their OBJECT field in the header is, e.g. Blind13A_10
  3. In astro/INBOX archive all files with doArchive *, this should create a directory Blind14A_10 in astro/ARCHIVE. You can also give a list of files.
  4. Map the archived files with doMapping -o Blind14A_10, this should create a directory Blind14A_10 in astro/DATA, with symbolic links that relate the archived files with MJD names with names that have the epoch naming convention.
  5. Remove cosmic rays with qsub -t 1:njobs crblaster.sge Blind14A_10 (epoch) [CCD list], where njobs is the number of jobs you want to submit. This should create cosmic ray removed files in astro/ARCHIVE/(MJD)/(CCD)/Blind14A_10_(CCD)_(MJD)_image_crblaster.fits, where (MJD) is related to the epoch given and ncores is the number of cores requested.
  6. Run the image subtraction pipeline with qsub -t 1:njobs doastro.sge Blind14A_10 (epoch) (ref_epoch) [CCD], where njobs is the number of cores requested. This will create projected, difference, inverse variance, flux and flux signal to noise ratio files in astro/ARCHIVE/(MJD)/(CCD) as well as many other auxiliary files in astro/SHARED and web outputs in astro/ARCHIVE/WEB. You can also use the script doastro_ordered in astro/sbin to run the differences sequentially and in chronological order for a more realistic result, e.g. doastro_ordered Blind14A_10 N6 2 1 3 4 5 6 7 8 9 10 11 12 13 14 15 15 16 17 19 22 25 27 0, where the three first options are the field, CCD and reference field, while the rest are the epochs to subtract and where 0 indicates a revisit of the entire field using the positive repeated candidates.
  7. Revisit all image differences for new candidates with qsub -t 1:njobs doastro-revisit.sge Blind14A_10 (ref_epoch) [CCD list], where njobs is the number of cores requested. This will mainly add files to astro/SHARED and to astro/ARCHIVE/WEB.

Real-Time execution

...

Generate transient catalogues

To generate a catalogue of transient candidates with fluxes and absolute celestial coordinates run, e.g.

python dodifferencecatalogue.py --field Blind14A_01 --CCD N1 --sharedir /share/astro-arc/DECAM-14A/SHARED --archivedir /share/astro-arc/DECAM-14A/ARCHIVE [--filter]

This generates a catalogue of all transient candidates in field Blind14A_01, CCD N1, based on the HITS image subtraction pipeline whose results were stored in /share/astro-arc/DECAM-14A/SHARED and /share/astro-arc/DECAM-14A/ARCHIVE. With the option --filter the script only outputs candidates which were classified as true candidates by a random forest classifier.

The catalogue is generated in file /share/astro-arc/DECAM-14A/SHARED/(field)/(CCD)/(field)_(CCD)_catalogue.dat

The file contains the following columns:

  1. ipix: pixel i del candidato en el archivo de referencia
  2. jpix: pixel j del candidato en el archivo de referencia
  3. RA: RA del candidato
  4. DEC: DEC del candidato
  5. rms: root mean square de la transformación (en grados), estimación del error en RA DEC
  6. MJDproc: fecha de archivo a procesar (diferencia = MJDproc - MJDref)
  7. MJDref: fecha de archivo de referencia (diferencia = MJDproc - MJDref)
  8. flux: flujo integrado del candidato en ADU del archivo de referencia
  9. fluxSNR: señal a ruido del flujo integrado del candidato
  10. probeven: probabilidad de que el candidato sea real (usando random forest en base a CCD pares)
  11. probodd: probabilidad de que el candidato sea real (usando random forest en base a CCD impares)
  12. conv1st: 1 si la imagen de referencia (im1) fue convolucionada, 0 si la imagen nueva (im2) fue convolucionada para hacer la diferencia.
  13. icand: el indice del candidato en archivos de candidatos
  14. candfile: archivo binario con información acerca de los candidatos, contiene también estampillas del candidato (ver nota abajo)
  15. reffits: fits reference file w/o cosmic rays
  16. procfits: fits new file w/o cosmic rays
  17. procprojfit: fits of projected new file
  18. difffits: difference file
  19. invVARfits: inverse variance of difference file (useful for doing accurate photometry)

Para recuperar imágenes a partir de archivos binarios de candidatos se debe ejecutar lo siguiente:

  1. cand = np.load(<archivo .npy de candidatos>)
  2. nstamp = 21
  3. imSNR = cand[icand, 10: 10 + nstamp * nstamp]
  4. im1 = cand[icand, 10 + nstamp * nstamp: 10 + 2 * nstamp * nstamp]
  5. im2 = cand[icand, 10 + 2 * nstamp * nstamp: 10 + 3 * nstamp * nstamp]
  6. imt = cand[icand, 10 + 3 * nstamp * nstamp: 10 + 4 * nstamp * nstamp]

La imagen de diferencia se obtiene como im2 - imt si conv1st = 1 o imt - im1 si conv1st = 0

Los candidatos de un campo se grafican, e.g. http://levque.dim.uchile.cl/astro/Blind14A_01/N1/Blind14A_01_N1_final_good_all.html

Running in astro-1 (10.50.2.40)

  1. Raw files are in /astro/scratch/hits/TUCSON
  2. toolchain is in /opt/astro ($TOOLCHAIN variable in astro.cfg)
  3. development python code and bash scripts are in /astro/scratch/devel/sbin, final python code and bash scripts are in /astro/scratch/hits/sbin

Running in leftraru (146.83.2.103)

  1. Raw files are in /home/astrouser/TUCSON
  2. toolchain is in /home/apps/astro ($TOOLCHAIN variable in astro.cfg)
  3. development python code and bash scripts are in $HOME, final python code and bash scripts are in /home/apps/astro/sbin