Understanding Quasi-Definitive Data in Geomagnetic Research

Quasi-Definitive Data 2015

Quasi-Definitive Data http://www.intermagnet.org/faqs-eng.php#quasi-definitive INTERMAGNET has defined a standard for a new data type called quasi-definitive. As the name implies the data should be close to the expected definitive value, but is to be delivered more rapidly than an observatories annual definitive data. This initiative will be useful for a number of scientific activities, but with the Swarm satellite mission near its start date, the data will be particularly useful in providing ground truth data for the satellite s magnetometers. Quasi-definitive data are (H, D, Z) or (X, Y, Z) 1-minute data: Corrected using temporary baselines Made available less than 3 months after their acquisition Such that the difference between the quasi-definitive and definitive (X, Y, Z) monthly means is less than 5 nT for every month of the year Point c is checked a posteriori by comparing quasi-definitive and definitive data from the previous year. Tech. Ref. Manual V4.6 Appendix E-3 and http://www.intermagnet.org/faqs-eng.php#data-types (Quasi-definitive) Data are defined as data that have been corrected using provisional baselines. Produced soon after their acquisition, their accuracy is intended to be very close to that of an observatory's definitive data product. 98% of the differences between quasi-definitive and definitive data (X, Y, Z) monthly mean values should be less than 5nT.

Definitive and Quasi-definitive data downloaded October/November 2017 from (old) INTERMAGNET FTP site ftp.intermagnet.org 70 IMOs submitted at least some 2015 QD data ABK, API, ASC, ASP, BEL, BOU, BOX, BRW, CKI, CLF, CMO, CNB, CSY, CTA, CYG, DLT, DOU, EBR, ESK, EYR, FRD, FUR, GAN, GDH, GNG, GUA, HAD, HER, HON, HRN, HUA, HYB, IPM, JCO, KAK, KDU, KEP, KHB, KNY, KOU, LER, LRM, LYC, LZH, MAB, MAW, MBO, MCQ, MGD, MMB, NAQ, NEW, NGK, NVS, PET, PHU, PST, SBA, SBL, SFS, SJG, SPT, SUA, TAM, TDC, THL, THY, TUC, UPS, WNG 17 of these 70 IMOs had no 2015 definitive data available BOX, CLF, CYG, DLT, EYR, GAN, GDH, HUA, HYB, IPM, KOU, LZH, MBO, NAQ, PHU, THL, WNG As of early June 2018 definitive data for CYG, EYR, HUA, HYB have become available, but these data are not included in this analysis Data from 53 IMOs available for this comparison Differences calculated using QDC written by Peter Crosthwaite. Same software used for 2012 data comparison presented at the 2014 Hyderabad meeting.

2015 QD Availability: 53 IMOs Data available for yearly comparison % (X) 100.0 80.0 60.0 40.0 20.0 0.0 cnb sfs sbl bel cta mab sba sua kdu khb mmb spt kny nvs pst lyc maw mgd tam api cki csy cmo ebr had her ler new pet abk gng ngk brw gua lrm tdc tuc asc asp frd hrn kak thy esk jco kep mcq bou dou fur hon sjg ups Data available for yearly comparison % (Y) 100.0 80.0 60.0 40.0 20.0 0.0 mmb sbl cta sba sfs sua cnb kdu khb spt bel kny mab nvs pst maw mgd csy cmo had ler new pet tam abk api cki gng lyc ngk brw ebr gua her lrm asc asp frd hrn thy esk jco kep mcq bou dou fur hon kak sjg tdc tuc ups Data available for yearly comparison % (Z) 100.0 80.0 60.0 40.0 20.0 0.0 cta sbl sfs cnb pst spt kny mab nvs sba sua bel kdu khb maw mgd mmb csy ler tam abk cki lyc ngk pet api cmo ebr gua her lrm new asc frd had hrn esk fur gng kep mcq tdc tuc asp brw kak sjg thy ups jco bou dou hon

Annual Mean Difference: 53 IMOs (Def QD) Yearly mean difference (X) 3 2 1 0 cta sba sbl sua cnb kdu khb mmb sfs spt nvs pst kny maw lyc mab mgd bel cmo csy ebr had her ler new pet tam abk api asc cki gng ngk asp brw frd gua hrn lrm ups bou esk mcq tdc thy fur jco kep sjg tuc dou hon kak -1 -2 Yearly mean difference (Y) 2 1 0 sfs cnb spt cta kny pst sba sbl sua kdu khb mab maw mgd mmb nvs ler tam abk cki lyc api bel cmo csy lrm ebr frd had her new pet ups asc esk gng kep mcq ngk tdc asp brw fur gua hrn kak sjg tuc bou thy jco dou hon -1 -2 -3 Yearly mean difference (Z) 2 1.5 1 0.5 0 cta sfs cnb spt her kny maw pst sba sbl sua cki kdu khb mab mgd mmb ler nvs tam pet abk lyc api bel cmo csy ebr gua lrm new frd gng had ups esk fur kep mcq ngk tdc tuc asc asp brw hrn kak sjg thy jco bou dou hon -0.5 -1

Monthly Mean Difference: 53 IMOs (Def QD) Monthly mean difference (X) 7 5 3 1 nT -1 cta sba sbl cnb sfs spt nvs pst khb kny maw kdu mab mgd mmb cmo csy had ler lrm lyc new pet tam abk api bel cki gng ngk asp brw ebr frd gua her hrn asc esk mcq thy fur jco kep sjg tdc ups bou dou hon kak -3 -5 -7 Monthly mean difference (Y) 6 4 2 0 -2 sfs cnb spt pst sbl cki cta kdu khb kny mab maw mgd mmb sba ler nvs tam lyc abk api bel csy gua lrm cmo ebr frd had her new pet esk gng kep mcq ngk sjg brw fur kak tdc ups asc asp hrn thy jco bou dou hon -4 -6 nT -8 -10 -12 -14 -16 -18 -20 Monthly mean differences (Z) 7 6 5 4 3 2 1 nT 0 -1 cta sbl cnb sfs nvs spt kny maw pst sba kdu khb mab mgd mmb csy ler lrm tam cki had lyc new pet abk api bel cmo esk gng gua ngk ebr her asc frd hrn thy brw fur kep mcq sjg tdc asp bou dou hon kak ups jco -2 -3 -4 -5

Min. and Max. Minute Difference: 53 IMOs X (North) Min and max (X) 400 300 200 100 nT 0 cnb cta sbl sfs spt tam kny nvs pst kdu khb mab maw mgd mmb sba csy ler new cki gng had lyc pet abk api bel gua lrm ngk cmo ebr frd her hrn asc esk mcq thy brw fur kak kep sjg tdc ups asp bou dou hon jco -100 -200 -300 Min and max (X) - zoomed 5 4 3 2 1 nT 0 cnb sfs kny spt tam cta pst sba sbl kdu khb mab maw mgd mmb nvs ler abk cki lyc api bel cmo csy lrm ebr frd had her new pet esk gng kep mcq ngk asp brw fur gua hrn kak sjg tdc ups asc thy jco bou dou hon -1 -2 -3 -4 -5

Min. and Max. Minute Difference: 53 IMOs Y (East) Min and max (Y) 400 300 200 100 nT 0 sbl cnb cta sba sfs spt csy pst kny maw nvs kdu khb mab mgd mmb had ler lrm new pet tam abk api bel cki esk gng hrn lyc ngk brw cmo ebr gua her asc asp frd kep thy mcq tdc ups bou dou fur hon kak sjg jco -100 -200 -300 -400 Min and max (Y) - zoomed 5 4 3 2 1 nT 0 cnb sfs cta sbl khb sba kdu mab mmb spt kny nvs pst lyc maw mgd tam api bel cki csy had cmo ebr her ler new abk gng ngk pet gua lrm tdc asc asp brw frd hrn kak esk jco kep mcq thy bou dou fur hon sjg ups -1 -2 -3 -4 -5

Min. and Max. Minute Difference: 53 IMOs Z (Vertical) Min and max (Z) 200 150 100 50 0 nT cnb sfs spt pst cta maw sba sbl csy kdu khb mab mgd mmb kny nvs ler lyc tam api bel cki cmo ebr frd had her lrm new pet abk esk gng mcq ngk brw fur gua kep sjg tdc ups asc asp hrn kak thy jco bou dou hon -50 -100 -150 -200 -250 Min and max (Z) - zoomed 5 4 3 2 1 nT 0 sfs cnb spt cta kdu khb kny pst sba sbl mab maw mgd mmb nvs ler tam abk cki lyc api bel cmo csy lrm ebr frd had her kep new pet esk gng mcq ngk asp brw fur gua hrn kak sjg tdc ups asc thy jco bou dou hon -1 -2 -3 -4 -5

Monthly Means (X) 53 IMOs n=603 150 100 Count 50 0 -1 -4 -3 -2 20 0 1 2 3 4 -4.8 -4.6 -4.4 -4.2 -3.6 -3.4 -3.2 -2.8 -2.6 -2.2 -1.8 -1.6 -1.4 -1.2 -0.8 -0.6 -0.4 -0.2 -3.8 -2.4 -20 0.2 0.4 0.6 0.8 1.2 1.4 1.6 1.8 2.2 2.4 2.6 2.8 3.2 3.4 3.6 3.8 4.2 4.4 4.6 4.8 Def - QD Difference Monthly Means (Y) 180 160 140 120 Count 100 80 60 40 20 0 -4 -3 -2 -1 20 0 1 2 3 4 -4.6 -4.4 -4.2 -3.8 -3.6 -3.4 -3.2 -2.8 -2.6 -2.4 -2.2 -1.8 -1.6 -1.4 -1.2 -0.8 -0.6 -0.4 -0.2 -4.8 -20 0.2 0.4 0.6 0.8 1.2 1.4 1.6 1.8 2.2 2.4 2.6 2.8 3.2 3.4 3.6 3.8 4.2 4.4 4.6 4.8 Def - QD Difference Monthly Means (Z) 200 150 Count 100 50 0 -4 -3 -2 -1 20 3 0 1 2 4 -4.8 -4.6 -4.4 -4.2 -3.8 -3.6 -3.4 -3.2 -2.8 -2.6 -2.4 -2.2 -1.8 -1.6 -1.4 -1.2 -0.8 -0.6 -0.4 -0.2 0.2 0.4 0.6 0.8 1.2 1.4 1.6 1.8 2.2 2.4 2.6 2.8 3.2 3.4 3.6 3.8 4.2 4.4 4.6 4.8 -20 Def - QD Difference

Out of Specification X Channel Y Channel Z Channel DOU (Aug -5.8) MAB (Nov +5.8) MAB (Nov -17.4) MAB (May +6.6) THY (Dec +6.2) ASC (Mar -5.3) Almost Out of Specification X Channel Y Channel Z Channel ASC (Nov -4.8) MAB (Sep -4.9; Jun -4.3) MAB (Jun +4.9) DOU (Dec -4.8)

ASC 2015 X (North) Daily Mean Difference (Def - QD) DOU 2015 X (North) Daily Mean Difference (Def - QD) 6 100 4 80 2 60 0 40 1/1/2015 2/20/2015 4/11/2015 5/31/2015 7/20/2015 9/8/2015 10/28/2015 12/17/2015 -2 20 -4 0 1/1/2015 2/20/2015 4/11/2015 5/31/2015 7/20/2015 9/8/2015 10/28/2015 12/17/2015 -6 -20 -40 ASC 2015 Y (East) Daily Mean Difference (Def - QD) DOU 2015 Y (East) Daily Mean Difference (Def - QD) 4 20 10 2 0 0 -10 1/1/2015 2/20/2015 4/11/2015 5/31/2015 7/20/2015 9/8/2015 10/28/2015 12/17/2015 -2 1/1/2015 2/20/2015 4/11/2015 5/31/2015 7/20/2015 9/8/2015 10/28/2015 12/17/2015 -20 -4 -30 -6 -40 -8 -50 ASC 2015 Z (Vertical) Daily Mean Difference (Def - QD) DOU 2015 Z (Vertical) Daily Mean Difference (Def - QD) 5 30 20 3 10 1 0 1/1/2015 2/20/2015 4/11/2015 5/31/2015 7/20/2015 9/8/2015 10/28/2015 12/17/2015 -10 -1 1/1/2015 2/20/2015 4/11/2015 5/31/2015 7/20/2015 9/8/2015 10/28/201512/17/2015 -20 -3 -30 -5 -40

THY 2015 X (North) Daily Mean Differnce (Def - QD) MAB 2015 X (North) Daily Mean Difference (Def-QD) 30 35 30 25 25 20 20 15 15 10 10 5 5 0 0 -5 1/1/2015 2/20/2015 4/11/2015 5/31/2015 7/20/2015 9/8/2015 10/28/2015 12/17/2015 -5 1/1/2015 2/20/2015 4/11/2015 5/31/2015 7/20/2015 9/8/2015 10/28/2015 12/17/2015 -10 -10 -15 THY 2015 Y (East) Daily Mean Differnce (Def - QD) MAB 2015 Y (East) Daily Mean Difference (Def-QD) 6 5 4 2 0 1/1/2015 2/20/2015 4/11/2015 5/31/2015 7/20/2015 9/8/2015 10/28/2015 12/17/2015 0 -5 -2 1/1/2015 2/20/2015 4/11/2015 5/31/2015 7/20/2015 9/8/2015 10/28/2015 12/17/2015 -10 -4 -6 -15 -8 -20 THY 2015 Z (Vertical) Daily Mean Differnce (Def - QD) MAB 2015 Z (Vertical) Daily Mean Difference (Def-QD) 10 15 8 10 6 5 4 2 0 0 1/1/2015 2/20/2015 4/11/2015 5/31/2015 7/20/2015 9/8/2015 10/28/2015 12/17/2015 -5 -2 1/1/2015 2/20/2015 4/11/2015 5/31/2015 7/20/2015 9/8/2015 10/28/2015 12/17/2015 -10 -4 -6 -15

Publication Delay Publication Date is an optional field in the IAGA-2002 meta-data header. Some, but not all, IMOs include Publication Date in the IAGA-2002 QD daily data files. An alternative to the Publication Date field is the operating system file modification date. This is available for all QD daily IAGA files and was preserved during FTP data transfers. The OS file modification date correlates with IAGA-2002 header Publication Date for those files checked. OS file modification date was used as the measure of the publication date for the QD data for all IMOs. Acquisition date obtained from the IAGA-2002 daily file name eg CLF20150416qmin.min >90 days was used as an estimate of the three months delay time in some of the figures discussed below

Publication Delay 70 IMOs (N=23032 IMO_days of QD data) 900 800 Frequency of Occurrence 700 600 500 400 300 200 100 0 0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300 320 340 360 Publication Delay (days) 77 % delivered before 90 day delay 23 % delivered after 90 day delay 23/70 IMOs had no late data: ASP, BOU, BRW, CKI, CNB, CSY, CTA, EBR, EYR, FRD, GAN, GNG, GUA, HON, KHB, LRM, MGD, NGK, NVS, PET, PHU, SPT, THY Peak at 1 day: Peak at 10 day: Peak at 180 day: Peak > 365 days: ASC, KEP, PST, SBL, SFS, THY LER, NVS, PET CYG, KAK, MMB, KNY, NAQ, TDC, THL SBA, API Some IMOs deliver QD data daily, most deliver monthly. A small number deliver daily but 6 months after acquisition

Late QD daily files (70 IMOs) 400 350 Number of Late daily QD Files 300 250 200 150 100 50 0 sfs cnb kdu sbl cta khb mmb sba spt sua mab pst eyr kny maw nvs cki gan lyc mgd bel csy kou mbo cmo cyg ebr gdh had hyb ler naq new pet tam abk api gng ngk tdc brw gua ipm kak lrm tuc asc asp bou dlt esk frd hrn lzh mcq thl thy clf hua jco kep ups dou fur hon phu sjg box Late QD daily files - zoomed 60 Number of Late daily QD Files 50 40 30 20 10 0 cnb sfs cta new sba sbl sua kdu khb kny mab mmb spt nvs pst eyr maw tam cki gan lyc mgd api bel cmo csy cyg ebr kou ler mbo gdh had hyb naq pet abk gng gua ipm lrm ngk tdc tuc asp brw dlt frd hrn kak lzh thl thy asc esk box clf fur hua jco kep mcq sjg ups bou dou hon phu

Summary A total of 6 months (distributed across 4 IMOs) have monthly mean Def QD differences greater than 5 nT 1% of the total 603 months of QD data available for 2015 23 % daily QD files were delivered late (more than 90 days after acquisition 23 (33%) IMOs delivered all QD data within the three month publication delay