Data Processing and Preprocessing Summary
In this document, Aymeric Sauvageon from CEA/DRF/Irfu/DAp presents a detailed overview of the preprocessing steps involved in data processing from L0 to L1. It covers the definition of L0/L1 and coding, utilization of the database for processing, input file specifications from China, packet content details, and L0c processing algorithm. The document provides insights into creating data structures, generating fits files, and processing ground test binary files. For further details, refer to the provided links and resource descriptions.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
PreProcessing summary Aymeric SAUVAGEON CEA/DRF/Irfu/DAp PP summary 1 26/02/2019
Dfinition du L0/L1 (et codage ) Documents: Note on data processing from L0c to L1 (SV-GS-NT-xxx 3.2 du 18/01/2019) UTILISATION de la BdD pour traitement L0c en L0d/L1 (SV-GS-NT-xxx 2.0 du xx/01/2019) Page Wiki sous Redmine, SGS IT https://forge.in2p3.fr/projects/sgs- it/wiki#Sp cification-des-entr es-des-pipelines-bande-X-L0c-processing donne les liens: Template fits (sous-projet) https://forge.in2p3.fr/projects/templates-fits/wiki/Wiki Cas de test (avec fichiers en entr e-sortie) https://forge.in2p3.fr/projects/sgs-it/wiki/L0bandeXtests Codage traitements L0c pour avoir des fichiers fits: Algorithme test en C r crire en Python comme module du FSC PP summary 2 A. Sauvageon 26/02/2019
Input file from China In doc. SSDC-FSC IRD (SV-GS-IR-457-JPO) it is specified: that L0c tar file (available at SSDC) contains an XML file describing the meta data of the following DAT file (also included in tar file): SVOM_L0C_<Type>_<APID>_<StartTime>_<EndTime>_<Orbit>_<Station>_<Ver>_<Index>. dat shall contain an integer number of binary TM packets. Where: Type = data type acronyms, the value should be ED or SD, ED represents Engine Data, SD represents Science and Science HK Data. APID = identifier of the filtered APID, 4 HEX representation, extracted from packet header forcing first 5 bits to 00000 StartTime= first TM packet date with the format YYYYMMDDThhmmss, UTC time EndTime= last TM packet date with same format Orbit = satellite pass number acquired from X-pass-plan, computed from the descending node with the format XXXXXX, starting from 1 Station = station acronyms , with the format KR000, HK000, SY4G1(station acronyms||antenna num||record system||channel num) Ver = the number production and the algo version, value range from YYX=000 to 99z. Index = the file number, with the format X, value range from 1 to 9. It will be only 1 if the file is less than 2 Gbytes NB: If file exceeds 2 Gbytes, first file (with 2*1024*1024 packets) has a filename ending with _1.dat and another L0c file (with remaining packets) with _2.dat PP summary 3 A. Sauvageon 26/02/2019
Packet content In doc. SVOM French Payload Monitor and Control Requirements (SV-SCPL- SP-455-CNES) it is specified: For same APID, the packet structure (primary and secondary header) is the same but content can differ PP summary 4 A. Sauvageon 26/02/2019
L0c processing In doc. Note on data processing from L0c to L0d/L1 : I propose a 2 steps algorithm: First, it creates data structures as similar as possible to source data (no conversion, no use of other files, initial DAT file can be re-created from these structures) L0D Then, it creates fits files (data format is changed to be user-friendly) as inputs of FSC pipelines. This algorithm can be easily improved to process also ground test binary files of CCSDS packets: To read different APID in same file; To read file with packets having corrupted headers (this can be useful also for Chinese file since it can be altered by transmission or disk storage) or ending with a partial packet. TBC: structure data (L0D) are stored in FSC database (only headers, full packet as fits files). TBC: pre-processing (step 2) creates L0E and L1 fits (cutting science data according to ObsId): Checks if pointing is stable within same Target Id; L0E fits just split L0D data in columns, keeping raw format (UTC time) L1 fits decode L0E columns into engineering values, using AUXiliary files (conversion laws for engineering and leap seconds to go from UTC to TT). The volume at each X-band pass is determined by its duration and TM rate. SYS-R-6.3-40 gives the nominal number of passes per day (6), the minimum total duration per day (30 minutes, TBC) and the max duration between 2 passes (12 hours and 10 hours for 90% of time). Biggest file is for ECLAIRs (18/21=85.7%) photons (24/25=96%, other packets less than 1/s against mean 24/s for photons, see start of 3.3): around 1.3 Gbyte for 1.4 million of packets (nominal case is 2700 Mbits for 0.33 million of packets). PP summary 5 A. Sauvageon 26/02/2019
Directory structure (1) In doc. Note on data processing from L0c to L0d/L1 : Similarly to Swift, we can have 4 main directories: AUX, ANC, OBS, VHF. To limit the number of sub-directories, we group data by week (week is an operational duration since workplan for GP is approved every week at OCG meeting). We try to have less than one thousand sub- directories (or files) per directory for 10 years mission: it is the case if we group by week. VHF is A. Formica domain A first proposal for repository organization is (Type is first digit, directories are naturally sorted by category: GRM, GRB, CAT, NOM-GRB,NOM-CAL,NOM-AT, ToO-EX,ToO-MM, GP-PPT,GP-CAL): Repository/OBS/T_obsIDnum_NNN.0/ (for L1 photon files, obsId number, snapshot NNN automatically calculated by my program) Repository/ANC/ORB/T_obsIDnum.NNN/ (for L1 orbital files, obsId number, snapshot NNN) Repository/ANC/ORB/www/ (for L0E orbital files during week www) Repository/OBS/www/ (for L0E files, week number www after a reference date) Repository/ANC/L0D/www/yyyymmdd_xxxxxx.0/ (week www based on date in L0c filename, which corresponds to first PPS time in data, orbit number on 6 digits). We need to group files per week, because there are 6 passes per day (and 7 files per pass, around 50 files per day) more than 10 000 (6*365*5) directories for five years! PP summary 6 A. Sauvageon 26/02/2019
Directory structure (2) Repository/AUX/PLA/WkPlan/ (for workplans, one per week for GP, one per day for ToO) Repository/AUX/PLA/TcPlan/ (for TC plans, TBD) Repository/ANC/HKS/www/ (for HK science files during week www) Repository/ANC/HKE/www/ (for HK engineering received from FPOC) What is the format of time in fits files ? We can keep the initial format (4 bytes for seconds and 3 bytes for microseconds) or use only one field (double precision real on 8 bytes is sufficient: 1e-14 is ok for 10 s on 21 years). CHOICE: one real field in MJD (elapse seconds with s precision, from T_PPS_REF = 20170101T00:00:00) in TT for L1 CNV (so use of auxiliary file for leap seconds). MJDREF 57754.0 / Modified Julian Date of time origin TIMESYS TT / Time frame system TIMEUNIT s / Time unit TIMEREF LOCAL / Time reference frame While for L0D/L0E RAW, PPS integer values or one real field in MJD UTC time: TIMESYS UTC / Time frame system How do we cut orbital/attitude fits files ? Since AAV packets contain ObsID field we can cut SVO-ATT-CNV as photon files. As I correlate PVT packets with AAV with time, we can do same cut for SVO-ORB-CNV. Choose origin at L1 level: FPL by default (PDPU if missing) ? PP summary 7 A. Sauvageon 26/02/2019
Proposal for L0D Parameters to put file SVO-DAT-L0D-yyyymmddThhmnss_HH.fits (from directory Repository/ANC/L0D/www/yyyymmdd_xxxxxx.0) From user: type = L0d (FSCLEVEL = L0D RAW) To build directory and filename: o Week (www, 0 to 999, WEEK = Integer) o UTC time (yyyymmddThhmnss, 20170101 to , ERTFIRST = UTC_format) o Orbit (xxxxxx, 1 to around 116800 for 20 years, REVOL = Integer) o value of the code packet ID (HH, _VAL = Integer) Only in fits keywords: o APID donne PID et PCAT (_APID = Integer) o Number of CCSDS packets processed (_NBPROC = Integer) o Instrument name (from APID, by default: INSTRUME = ALL) o Start time of data in UTC (TSTART = Real) o End time of data in UTC (TSTOP = Real) Idem for file SVO-ERR-L0D-yyyymmddThhmnss.fits o code packet ID is 0xFF for unknown (_VAL = 255) INSTRUME = ALL (default) changed according to PID PP summary 8 A. Sauvageon 26/02/2019
Proposal for L0E Parameters to put file SVO-AAV-RAW-yyyymmddThhmnss_Instr.fits (from directory Repository/ANC/ORB/www/) From user: type = L0e (FSCLEVEL = L0E RAW) To build directory and filename: o Week (www, 0 to 999, WEEK = Integer) o UTC time (yyyymmddThhmnss, 20170101 to , ERTFIRST = UTC_format) o Instrument name (Instr from APID, by default: INSTRUME = ALL) Only in fits keywords: o APID donne PID et PCAT (_APID = Integer) o Number of CCSDS packets processed (_NBPROC = Integer) o value of the code packet ID (HH, _VAL = Integer) o Start time of data in UTC (TSTART = Real) o End time of data in UTC (TSTOP = Real) Idem for file MXT-EVT-RAW-yyyymmddThhmnss_III.fits (III is SKY, CAL, BKG translated from HH, file in directory Repository/ANC/OBS/www/) Keywords specific for MXT : o Camera configuration (CONFMIN CONFMAX = Integer) from column CAM_CONF, not in ECLAIRs PP summary 9 A. Sauvageon 26/02/2019
Proposal for L1 Parameters to put file SVO-ATT-CNV-yyyymmddThhmnss.fits (TBC: from directory Repository/ANC/ORB/www/ObsID/) Do we cut data by ObsId ? From user: type = L1 (FSCLEVEL = L1 CNV) To build directory and filename: o Week (www, 0 to 999, WEEK = Integer) o UTC time (yyyymmddThhmnss, 20170101 to , ERTFIRST = UTC_format) Only in fits keywords: o Number of AAV bulletins processed (_NBPROC = Integer) o Start time of data in TT (TSTART = Real) o End time of data in TT (TSTOP = Real) Parameters to put file MXT-EVT-CNV-yyyymmddThhmnss_III.fits (III is SKY, CAL, BKG translated from HH, file in directory Repository/OBS/www/ObsID/) Idem, plus: o APID donne PID et PCAT (_APID = Integer) o Number of raw events processed (_NBPROC = Integer) o value of the code packet ID (HH, _VAL = Integer) o Camera configuration (from column CAM_CONF) only for MXT, not ECLAIRs o Camera temperature (from column CAM_TEMP) only for MXT, not ECLAIRs 4 keywords OBSID (d compos en OBS_NUM OBS_TYPE) + OBS_SEQ (sequence=snapshot) PP summary 10 A. Sauvageon 26/02/2019
MERCI PP summary 11 A. Sauvageon 26/02/2019
