How to Check a Simulation Study: Methods and Considerations

 
H
o
w
 
t
o
 
c
h
e
c
k
 
a
 
s
i
m
u
l
a
t
i
o
n
 
s
t
u
d
y
 
 
T
r
a
 
M
y
 
P
h
a
m
 
|
 
t
r
a
.
p
h
a
m
.
0
9
@
u
c
l
.
a
c
.
u
k
 
C
o
-
a
u
t
h
o
r
s
:
 
I
a
n
 
W
h
i
t
e
,
 
M
a
t
t
e
o
 
Q
u
a
r
t
a
g
n
o
,
 
T
i
m
 
M
o
r
r
i
s
 
08/09/2023   |  2023 UK Stata Conference
 
1
 
M
o
t
i
v
a
t
i
o
n
 
We use simulation studies to assess e.g. statistical methods or power of a study
 
Sometimes they can go wrong
How do we check them?
Can we make them less likely to go wrong?
 
This work draws on our experience of checking simulation studies
Considerations discussed in 
Morris, White & Crowther (2019) 
and more recently in
White et al 2023
 (accepted by IJE and preprint)
 
M
orris TP, White IR, Crowther MJ. Using simulation studies to evaluate
statistical methods. 
Stat Med
 2019; 38: 2074–2102
White IR, Pham TM, Quartagno M, Morris TP. How to check a simulation
study. 
OSF
 2023; https://doi.org/10.31219/osf.io/cbr72
 
2
 
S
i
m
p
l
e
 
s
i
m
u
l
a
t
i
o
n
 
s
t
u
d
y
:
 
A
D
E
M
P
I
 
3
 
S
i
m
p
l
e
 
s
i
m
u
l
a
t
i
o
n
 
s
t
u
d
y
:
 
A
D
E
M
P
I
 
4
 
S
i
m
p
l
e
 
s
i
m
u
l
a
t
i
o
n
 
s
t
u
d
y
:
 
A
D
E
M
P
I
 
Varied parameters: Pr(E) & Pr(D), the strength of the
dependence of E and D on C, missingness mechanism
Fixed: n
sample
 = 500
 
N(0,1); initially MCAR
 
Logistic
regression | C
 
Logistic
regression | C
 
5
 
S
i
m
p
l
e
 
s
i
m
u
l
a
t
i
o
n
 
s
t
u
d
y
:
 
A
D
E
M
P
I
 
6
 
S
i
m
p
l
e
 
s
i
m
u
l
a
t
i
o
n
 
s
t
u
d
y
:
 
A
D
E
M
P
I
 
7
 
S
i
m
p
l
e
 
s
i
m
u
l
a
t
i
o
n
 
s
t
u
d
y
:
 
A
D
E
M
P
I
 
8
 
S
i
m
p
l
e
 
s
i
m
u
l
a
t
i
o
n
 
s
t
u
d
y
:
 
A
D
E
M
P
I
 
9
 
H
o
w
 
t
o
 
c
h
e
c
k
 
a
 
s
i
m
u
l
a
t
i
o
n
 
s
t
u
d
y
 
16 points of advice
A
r
r
a
n
g
e
d
 
b
y
 
D
e
s
i
g
n
,
 
C
o
n
d
u
c
t
,
 
A
n
a
l
y
s
i
s
Illustrated by simple simulation study
 
Stata code available at 
https://osf.io/egc73/
 
 
10
 
D
e
s
i
g
n
:
 
p
l
a
n
n
i
n
g
 
t
h
e
 
s
i
m
u
l
a
t
i
o
n
 
1
.
 
I
n
c
l
u
d
e
 
a
 
s
e
t
t
i
n
g
 
w
i
t
h
 
k
n
o
w
n
 
p
r
o
p
e
r
t
i
e
s
 
(
b
e
n
c
h
m
a
r
k
)
 
11
E
x
a
m
p
l
e
 
s
i
m
u
l
a
t
i
o
n
 
s
t
u
d
y
 
Complete case analysis unbiased under
Missing Completely At Random
N(0,1); initially MCAR
Logistic
regression | C
Logistic
regression | C
12
E
x
a
m
p
l
e
 
s
i
m
u
l
a
t
i
o
n
 
s
t
u
d
y
13
 
Unbiased & most precise
 
Least precise
 
C
o
n
d
u
c
t
:
 
c
o
d
i
n
g
 
t
h
e
 
s
i
m
u
l
a
t
i
o
n
 
14
 
2
.
 
W
r
i
t
e
 
w
e
l
l
-
s
t
r
u
c
t
u
r
e
d
 
c
o
d
e
 
prog define 
gendata
syntax, obs(int) logite(string) logitd(string) pmiss(string)
[generate Ctrue (full data), E, D, Cobs (MCAR)]
end
 
prog define 
anadata
//Method 1: full data before data deletion
 
logit D E Ctrue
 
[save results]
//Method 2: CCA
 
logit D E Cobs
 
[save results]
//Method 3: MI
 
mi impute regress Cobs D##E, add(5)
 
mi estimate: logit D E Cobs
 
[save results]
end
 
 
3 main tasks
Generate data
Analyse data
Save results in well-structured estimates
dataset
 
 
2 separate programs
g
e
n
d
a
t
a
 
d
a
t
a
 
g
e
n
e
r
a
t
i
o
n
anadata
 
 
data analysis & results
extraction
 
 
Results stored in estimates dataset via
postfile
 
15
 
C
o
n
d
u
c
t
:
 
c
o
d
i
n
g
 
t
h
e
 
s
i
m
u
l
a
t
i
o
n
 
3
.
 
S
t
u
d
y
 
a
 
s
i
n
g
l
e
 
v
e
r
y
 
l
a
r
g
e
 
d
a
t
a
s
e
t
Use 
gendata
 to generate a single very large dataset
Check 
anadata
 runs successfully on this dataset
 
 
16
 
C
o
n
d
u
c
t
:
 
c
o
d
i
n
g
 
t
h
e
 
s
i
m
u
l
a
t
i
o
n
 
3
.
 
S
t
u
d
y
 
a
 
s
i
n
g
l
e
 
v
e
r
y
 
l
a
r
g
e
 
d
a
t
a
s
e
t
Use 
gendata
 to generate a single very large dataset (e.g. with nsample=100000)
Check 
anadata
 runs successfully on this dataset
 
4
.
 
R
u
n
 
t
h
e
 
s
i
m
u
l
a
t
i
o
n
 
w
i
t
h
 
a
 
s
m
a
l
l
 
n
u
m
b
e
r
 
o
f
 
r
e
p
e
t
i
t
i
o
n
s
Run 3 repetitions using 
gendata
 & 
anadata
; check they all give different results
Check results stored match those displayed on screen
 
 
17
 
C
o
n
d
u
c
t
:
 
c
o
d
i
n
g
 
t
h
e
 
s
i
m
u
l
a
t
i
o
n
 
3
.
 
S
t
u
d
y
 
a
 
s
i
n
g
l
e
 
v
e
r
y
 
l
a
r
g
e
 
d
a
t
a
s
e
t
Use 
gendata
 to generate a single very large dataset (e.g. with nsample=100000)
Check 
anadata
 runs successfully on this dataset
 
4
.
 
R
u
n
 
t
h
e
 
s
i
m
u
l
a
t
i
o
n
 
w
i
t
h
 
a
 
s
m
a
l
l
 
n
u
m
b
e
r
 
o
f
 
r
e
p
e
t
i
t
i
o
n
s
Run 3 repetitions using 
gendata
 & 
anadata
; check they all give different results
Check results stored match those displayed on screen
 
5
.
 
A
n
t
i
c
i
p
a
t
e
 
a
n
a
l
y
s
i
s
 
f
a
i
l
u
r
e
s
Use 
capture
 in 
anadata
 to post empty results for repetitions with failures
 
18
 
C
o
n
d
u
c
t
:
 
c
o
d
i
n
g
 
t
h
e
 
s
i
m
u
l
a
t
i
o
n
 
3
.
 
S
t
u
d
y
 
a
 
s
i
n
g
l
e
 
v
e
r
y
 
l
a
r
g
e
 
d
a
t
a
s
e
t
Use 
gendata
 to generate a single very large dataset (e.g. with nsample=100000)
Check 
anadata
 runs successfully on this dataset
 
4
.
 
R
u
n
 
t
h
e
 
s
i
m
u
l
a
t
i
o
n
 
w
i
t
h
 
a
 
s
m
a
l
l
 
n
u
m
b
e
r
 
o
f
 
r
e
p
e
t
i
t
i
o
n
s
Run 3 repetitions using 
gendata
 & 
anadata
; check they all give different results
Check results stored match those displayed on screen
 
5
.
 
A
n
t
i
c
i
p
a
t
e
 
a
n
a
l
y
s
i
s
 
f
a
i
l
u
r
e
s
Use 
capture
 in 
anadata
 to post empty results for repetitions with failures
6
.
 
M
a
k
e
 
i
t
 
e
a
s
y
 
t
o
 
r
e
-
c
r
e
a
t
e
 
a
n
y
 
s
i
m
u
l
a
t
e
d
 
d
a
t
a
s
e
t
Use 
postfile
 to save 
c(rngstate)
at start of each repetition
 
 
19
 
A
n
a
l
y
s
i
s
:
 
m
e
t
h
o
d
 
f
a
i
l
u
r
e
s
 
a
n
d
 
o
u
t
l
i
e
r
s
 
7
.
 
C
o
u
n
t
 
a
n
d
 
u
n
d
e
r
s
t
a
n
d
 
m
e
t
h
o
d
 
f
a
i
l
u
r
e
s
Count method failures, by method & data
generating mechanism
 
 
 
Repetition 13
 
CCA occasionally failed under more extreme DGM
 
20
 
A
n
a
l
y
s
i
s
:
 
m
e
t
h
o
d
 
f
a
i
l
u
r
e
s
 
a
n
d
 
o
u
t
l
i
e
r
s
 
7
.
 
C
o
u
n
t
 
a
n
d
 
u
n
d
e
r
s
t
a
n
d
 
m
e
t
h
o
d
 
f
a
i
l
u
r
e
s
Count method failures, by method & data
generating mechanism (e.g. 
summ
 / 
tab
)
8
.
 
L
o
o
k
 
f
o
r
 
o
u
t
l
i
e
r
s
Plot the standard error estimates against the
point estimates
 
gendata
 
with more extreme DGM
 
21
 
A
n
a
l
y
s
i
s
:
 
m
e
t
h
o
d
 
f
a
i
l
u
r
e
s
 
a
n
d
 
o
u
t
l
i
e
r
s
 
7
.
 
C
o
u
n
t
 
a
n
d
 
u
n
d
e
r
s
t
a
n
d
 
m
e
t
h
o
d
 
f
a
i
l
u
r
e
s
Count method failures, by method & data
generating mechanism (e.g. 
summ
 / 
tab
)
8
.
 
L
o
o
k
 
f
o
r
 
o
u
t
l
i
e
r
s
Plot the standard error estimates against the
point estimates
9
.
 
U
n
d
e
r
s
t
a
n
d
 
o
u
t
l
i
e
r
s
Re-create and explore a simulated dataset
which gave outlying results
 
gendata
 
with more extreme DGM
 
Repetition 1
 
22
 
A
n
a
l
y
s
i
s
:
 
m
e
t
h
o
d
 
f
a
i
l
u
r
e
s
 
a
n
d
 
o
u
t
l
i
e
r
s
 
7
.
 
C
o
u
n
t
 
a
n
d
 
u
n
d
e
r
s
t
a
n
d
 
m
e
t
h
o
d
 
f
a
i
l
u
r
e
s
Count method failures, by method & data
generating mechanism (e.g. 
summ
 / 
tab
)
8
.
 
L
o
o
k
 
f
o
r
 
o
u
t
l
i
e
r
s
Plot the standard error estimates against the
point estimates
9
.
 
U
n
d
e
r
s
t
a
n
d
 
o
u
t
l
i
e
r
s
Re-create and explore a simulated dataset
which gave outlying results
1
0
.
 
D
e
a
l
 
w
i
t
h
 
o
u
t
l
i
e
r
s
Exclude from results? (selection bias)
Program a back-up analysis method?
C
hange the simulation design?
 
 
P
enalised logistic regression as alternative
 
23
 
A
n
a
l
y
s
i
s
:
 
u
n
e
x
p
e
c
t
e
d
 
f
i
n
d
i
n
g
s
 
24
 
A
n
a
l
y
s
i
s
:
 
u
n
e
x
p
e
c
t
e
d
 
f
i
n
d
i
n
g
s
 
25
 
A
n
a
l
y
s
i
s
:
 
u
n
e
x
p
e
c
t
e
d
 
f
i
n
d
i
n
g
s
 
https://www.stata.com/meeting/uk22/slides/UK22_Marley-Zagar.pptx
https://github.com/UCL/siman
 
26
 
A
n
a
l
y
s
i
s
:
 
u
n
e
x
p
e
c
t
e
d
 
f
i
n
d
i
n
g
s
 
27
 
https://www.stata.com/meeting/uk22/slides/UK22_Marley-Zagar.pptx
https://github.com/UCL/siman
 
A
n
a
l
y
s
i
s
:
 
u
n
e
x
p
e
c
t
e
d
 
f
i
n
d
i
n
g
s
 
28
 
 
https://www.stata.com/meeting/uk22/slides/UK22_Marley-Zagar.pptx
https://github.com/UCL/siman
 
A
n
a
l
y
s
i
s
:
 
u
n
e
x
p
e
c
t
e
d
 
f
i
n
d
i
n
g
s
 
29
 
https://www.stata.com/meeting/uk22/slides/UK22_Marley-Zagar.pptx
https://github.com/UCL/siman
 
C
o
n
c
l
u
s
i
o
n
 
Simulation studies should be designed that can be easy to check
They should be checked repeatedly during the conduct and analysis stages
This list is not exhaustive, suggestions are welcome!
 
30
 
 
S
t
u
d
y
 
w
i
t
h
 
u
s
 
o
n
l
i
n
e
 
f
o
r
 
a
n
 
M
S
c
 
C
l
i
n
i
c
a
l
 
T
r
i
a
l
s
o
r
 
a
n
 
M
S
c
 
S
t
a
t
i
s
t
i
c
s
 
i
n
 
C
l
i
n
i
c
a
l
 
T
r
i
a
l
s
.
F
i
n
d
 
o
u
t
 
m
o
r
e
:
 
h
t
t
p
s
:
/
/
b
i
t
.
l
y
/
3
E
w
r
v
J
K
 
A
c
k
n
o
w
l
e
d
g
e
m
e
n
t
s
 
Ella Marley-Zagar: 
siman
This work was supported by the Medical Research
Council [grant number MC_UU_00004/07]
Slide Note
Embed
Share

Simulation studies are often used to evaluate statistical methods and study power, but they can sometimes produce misleading results. This work discusses strategies to assess and improve the quality of simulation studies, drawing on experiences and considerations outlined in relevant literature. A simple simulation study comparing multiple imputation with complete case analysis is presented, focusing on data generating methods and estimands. Various methods of analysis are employed to compare the approaches.


Uploaded on Jul 29, 2024 | 0 Views


Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

E N D

Presentation Transcript


  1. How to check a simulation study Tra My Pham | tra.pham.09@ucl.ac.uk Co-authors: Ian White, Matteo Quartagno, Tim Morris 08/09/2023 | 2023 UK Stata Conference 1

  2. Motivation We use simulation studies to assess e.g. statistical methods or power of a study Sometimes they can go wrong How do we check them? Can we make them less likely to go wrong? This work draws on our experience of checking simulation studies Considerations discussed in Morris, White & Crowther (2019) and more recently in White et al 2023 (accepted by IJE and preprint) Morris TP, White IR, Crowther MJ. Using simulation studies to evaluate statistical methods. Stat Med 2019; 38: 2074 2102 White IR, Pham TM, Quartagno M, Morris TP. How to check a simulation study. OSF 2023; https://doi.org/10.31219/osf.io/cbr72 2

  3. Simple simulation study: ADEMPI 3

  4. Simple simulation study: ADEMPI Aim To compare multiple imputation with complete case analysis 4

  5. Simple simulation study: ADEMPI Aim Data generating methods To compare multiple imputation with complete case analysis N(0,1); initially MCAR C Logistic regression | C Logistic regression | C E D Varied parameters: Pr(E) & Pr(D), the strength of the dependence of E and D on C, missingness mechanism Fixed: nsample = 500 5

  6. Simple simulation study: ADEMPI Estimand Log odds ratio between E and D, conditional on C 6

  7. Simple simulation study: ADEMPI Estimand Log odds ratio between E and D, conditional on C Methods of analysis Logistic regression of D on E and C, using 1. Full data before setting some values of C to missing 2. Complete case analysis (excluding cases with missing C) 3. Multiple imputation of missing values of C (Missing values replaced with several plausible values based on observed data) 7

  8. Simple simulation study: ADEMPI Estimand Log odds ratio between E and D, conditional on C Methods of analysis Logistic regression of D on E and C, using 1. Full data before setting some values of C to missing 2. Complete case analysis (excluding cases with missing C) 3. Multiple imputation of missing values of C (Missing values replaced with several plausible values based on observed data) Performance measures Bias, empirical & model-based standard error, coverage 8

  9. Simple simulation study: ADEMPI Estimand Log odds ratio between E and D, conditional on C Methods of analysis Logistic regression of D on E and C, using 1. Full data before setting some values of C to missing 2. Complete case analysis (excluding cases with missing C) 3. Multiple imputation of missing values of C (Missing values replaced with several plausible values based on observed data) Performance measures Bias, empirical & model-based standard error, coverage Implementation nsim = 1000 repetitions 9

  10. How to check a simulation study 16 points of advice Arranged by Design, Conduct, Analysis Illustrated by simple simulation study Stata code available at https://osf.io/egc73/ 10

  11. Design: planning the simulation 1. Include a setting with known properties (benchmark) 11

  12. Example simulation study Data generating methods N(0,1); initially MCAR C Logistic regression | C Logistic regression | C E D Complete case analysis unbiased under Missing Completely At Random 12

  13. Example simulation study Methods of analysis Logistic regression of D on E and C, using 1. Full data before setting some values of C to missing 2. Complete case analysis (excluding cases with missing C) 3. Multiple imputation of missing values of C (Missing values are imputed with several plausible values based on observed data) Unbiased & most precise Least precise 13

  14. Conduct: coding the simulation 2. Write well-structured code 14

  15. prog define gendata syntax, obs(int) logite(string) logitd(string) pmiss(string) [generate Ctrue (full data), E, D, Cobs (MCAR)] 3 main tasks Generate data Analyse data Save results in well-structured estimates dataset end prog define anadata //Method 1: full data before data deletion logit D E Ctrue [save results] 2 separate programs gendata anadata extraction //Method 2: CCA data generation data analysis & results logit D E Cobs [save results] //Method 3: MI mi impute regress Cobs D##E, add(5) mi estimate: logit D E Cobs Results stored in estimates dataset via postfile [save results] end 15

  16. Conduct: coding the simulation 3. Study a single very large dataset Use gendata to generate a single very large dataset Check anadata runs successfully on this dataset 16

  17. Conduct: coding the simulation 3. Study a single very large dataset Use gendata to generate a single very large dataset (e.g. with nsample=100000) Check anadata runs successfully on this dataset 4. Run the simulation with a small number of repetitions Run 3 repetitions using gendata & anadata; check they all give different results Check results stored match those displayed on screen 17

  18. Conduct: coding the simulation 3. Study a single very large dataset Use gendata to generate a single very large dataset (e.g. with nsample=100000) Check anadata runs successfully on this dataset 4. Run the simulation with a small number of repetitions Run 3 repetitions using gendata & anadata; check they all give different results Check results stored match those displayed on screen 5. Anticipate analysis failures Use capture in anadata to post empty results for repetitions with failures 18

  19. Conduct: coding the simulation 3. Study a single very large dataset Use gendata to generate a single very large dataset (e.g. with nsample=100000) Check anadata runs successfully on this dataset 4. Run the simulation with a small number of repetitions Run 3 repetitions using gendata & anadata; check they all give different results Check results stored match those displayed on screen 5. Anticipate analysis failures Use capture in anadata to post empty results for repetitions with failures 6. Make it easy to re-create any simulated dataset Use postfile to save c(rngstate)at start of each repetition 19

  20. Analysis: method failures and outliers CCA occasionally failed under more extreme DGM 7. Count and understand method failures Count method failures, by method & data generating mechanism Repetition 13 20

  21. Analysis: method failures and outliers 7. Count and understand method failures gendata with more extreme DGM Count method failures, by method & data generating mechanism (e.g. summ / tab) 8. Look for outliers Plot the standard error estimates against the point estimates 21

  22. Analysis: method failures and outliers gendatawith more extreme DGM 7. Count and understand method failures Count method failures, by method & data generating mechanism (e.g. summ / tab) Repetition 1 8. Look for outliers Plot the standard error estimates against the point estimates 9. Understand outliers Re-create and explore a simulated dataset which gave outlying results 22

  23. Analysis: method failures and outliers 7. Count and understand method failures Penalised logistic regression as alternative Count method failures, by method & data generating mechanism (e.g. summ / tab) 8. Look for outliers Plot the standard error estimates against the point estimates 9. Understand outliers Re-create and explore a simulated dataset which gave outlying results 10. Deal with outliers Exclude from results? (selection bias) Program a back-up analysis method? Change the simulation design? 23

  24. Analysis: unexpected findings Is it really a bias / under-coverage, or is it noise? 11. Check Monte Carlo errors 24

  25. Analysis: unexpected findings Is it really a bias / under-coverage, or is it noise? 11. Check Monte Carlo errors 12. Why are model-based standard errors wrong? e.g. sources of variation in the DGM and analysis don t correspond 25

  26. Analysis: unexpected findings Is it really a bias / under-coverage, or is it noise? 11. Check Monte Carlo errors 12. Why are model-based standard errors wrong? e.g. sources of variation in the DGM and analysis don t correspond Is it driven by bias, wrong intervals, or both? (zip plot, produced by Stata package siman) 13. Why is coverage poor? https://www.stata.com/meeting/uk22/slides/UK22_Marley-Zagar.pptx https://github.com/UCL/siman 26

  27. Analysis: unexpected findings Is it really a bias / under-coverage, or is it noise? 11. Check Monte Carlo errors 12. Why are model-based standard errors wrong? e.g. sources of variation in the DGM and analysis don t correspond Is it driven by bias, wrong intervals, or both? (zip plot, produced by Stata package siman) 13. Why is coverage poor? e.g. interchanging power and type I error 14. Why are power and type I error wrong? https://www.stata.com/meeting/uk22/slides/UK22_Marley-Zagar.pptx https://github.com/UCL/siman 27

  28. Analysis: unexpected findings Is it really a bias / under-coverage, or is it noise? 11. Check Monte Carlo errors 12. Why are model-based standard errors wrong? e.g. sources of variation in the DGM and analysis don t correspond Is it driven by bias, wrong intervals, or both? (zip plot, produced by Stata package siman) 13. Why is coverage poor? e.g. interchanging power and type I error 14. Why are power and type I error wrong? e.g. they occur only when the DGM includes a particular source of variation while analyses aren t allowing for this source of variation? 15. When do unexpected findings occur? https://www.stata.com/meeting/uk22/slides/UK22_Marley-Zagar.pptx https://github.com/UCL/siman 28

  29. Analysis: unexpected findings Is it really a bias / under-coverage, or is it noise? 11. Check Monte Carlo errors 12. Why are model-based standard errors wrong? e.g. sources of variation in the DGM and analysis don t correspond Is it driven by bias, wrong intervals, or both? (zip plot, produced by Stata package siman) 13. Why is coverage poor? e.g. interchanging power and type I error 14. Why are power and type I error wrong? e.g. they occur only when the DGM includes a particular source of variation while analyses aren t allowing for this source of variation? 15. When do unexpected findings occur? Re-code the simulation study in a different statistical package? Have a different person code it? 16. General checking method https://www.stata.com/meeting/uk22/slides/UK22_Marley-Zagar.pptx https://github.com/UCL/siman 29

  30. Conclusion Simulation studies should be designed that can be easy to check They should be checked repeatedly during the conduct and analysis stages This list is not exhaustive, suggestions are welcome! 30

  31. Acknowledgements Study with us online for an MSc Clinical Trials or an MSc Statistics in Clinical Trials. Find out more: https://bit.ly/3EwrvJK Ella Marley-Zagar: siman This work was supported by the Medical Research Council [grant number MC_UU_00004/07] @MRCCTU www.mrcctu.ucl.ac.uk

Related


More Related Content

giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#