In-Service Monitoring and Reporting for Advanced Driver Assistance Systems (ADS)
SG3
In Service Monitoring and Reporting
VMAD#25 Meeting
11-12 April 2022
Espedito Rusciano
VMAD-25-08
SG3-In Service Monitoring and Reporting,
Objectives and Deliverables
•
In-Service Monitoring and Reporting (ISMR) addresses the in-
service safety of the ADS after its placing on the market
(operational experience feedback loop)
•
It relies on the collection of in-service data(fleet monitoring) to
assess whether the ADS continues to be safe when operated on
the road and to
identify safety risks
•
This data collection can also be used for the identification of new
scenarios to support the development of the Scenario Catalogue.
•
ISMR allows the whole ADS community to learn from major ADS
accidents/incidents through information sharing
•
SG3 is developing procedures to confirm the in-service ADS safety
and to improve the level of safety of ADS based on the evidence
collected from the field operation
Monitoring, Reporting and Investigation
M
o
n
i
t
o
r
i
n
g
R
e
p
o
r
t
i
n
g
I
n
v
e
s
t
i
g
a
t
i
o
n
S
e
v
e
r
i
t
y
F
r
e
q
u
e
n
c
y
In Service Monitoring
M
o
n
i
t
o
r
i
n
g
(
L
i
n
k
w
i
t
h
t
h
e
A
U
D
I
T
P
i
l
l
a
r
)
:
•
M
anufacturers should set up
a monitoring program according to the
SMS Requirements
•
Vehicle data collection and analysis by the manufactures for
reporting under ISMR, besides EDR/DSSAD
•
M
a
n
u
f
a
c
t
u
r
e
r
s
a
r
e
e
x
p
e
c
t
e
d
t
o
c
o
l
l
e
c
t
d
a
t
a
a
l
s
o
f
r
o
m
o
t
h
e
r
a
c
c
e
s
s
i
b
l
e
s
o
u
r
c
e
s
o
f
d
a
t
a
(
e
.
g
.
,
c
u
s
t
o
m
e
r
r
e
p
o
r
t
s
)
•
Proactive approach for Safety which shall be
integrated in the Safety Management System
•
Beyond the scope of the occurrences reporting
•
Increase safety by identifying trends and
unusual or unsafe circumstances
In Service Reporting
O
c
c
u
r
r
e
n
c
e
R
e
p
o
r
t
i
n
g
:
•
Occurrence refers to any safety-related event involving a vehicle
equipped with an ADS.
•
For reporting, two different categories of occurrences are defined:
critical and not critical occurrences.
•
The manufacturer should report, as required by the Authority, on
both critical and non-critical occurrences
The main purpose of in-service reporting is the
prevention of accidents and incidents and not to
attribute blame or liability. (Just culture)
Types of Reporting
•
S
h
o
r
t
-
t
e
r
m
r
e
p
o
r
t
i
n
g
–
o
c
c
u
r
r
e
n
c
e
s
t
h
a
t
r
e
q
u
i
r
e
t
h
e
m
a
n
u
f
a
c
t
u
r
e
r
t
o
t
a
k
e
r
e
m
e
d
i
a
l
a
c
t
i
o
n
w
h
e
n
d
a
t
a
p
r
o
v
i
d
e
s
e
v
i
d
e
n
c
e
o
f
t
h
e
A
D
S
p
o
s
i
n
g
a
n
u
n
a
c
c
e
p
t
a
b
l
e
i
n
-
s
e
r
v
i
c
e
r
i
s
k
.
(
w
i
t
h
i
n
1
m
o
n
t
h
)
a)
indications of failure to meet safety requirements
b)
critical occurrence where the ADS was at fault
c)
other safety-relevant performance issues
•
P
e
r
i
o
d
i
c
r
e
p
o
r
t
i
n
g
-
r
e
p
o
r
t
i
n
g
i
n
t
h
e
f
o
r
m
o
f
a
g
g
r
e
g
a
t
e
d
d
a
t
a
(
p
e
r
h
o
u
r
o
f
o
p
e
r
a
t
i
o
n
o
r
d
r
i
v
e
n
k
m
)
f
o
r
A
D
S
-
v
e
h
i
c
l
e
t
y
p
e
a
n
d
r
e
l
a
t
e
d
t
o
A
D
S
o
p
e
r
a
t
i
o
n
(
a
t
l
e
a
s
t
e
v
e
r
y
y
e
a
r
)
a)
no inconsistencies have been detected compared to the ADS safety performance assessed
prior to market introduction;
b)
the ADS respects the performance requirements set by FRAV and as evaluated in the test
methods developed by VMAD;
c)
any newly discovered significant ADS safety performance issues have been adequately
addressed and how this was achieved.
Occurrences list
Investigation
O
c
c
u
r
r
e
n
c
e
I
n
v
e
s
t
i
g
a
t
i
o
n
s
•
Focus on critical occurrences, but it can include also other occurrences
•
It is expected an immediate notification to the authority on occurrences which can pose an immediate
risk to public safety.
•
An
Independent
body
conducts investigations according to its mandate
•
The investigation report, containing where appropriate safety recommendations, should be made
available to all parties involved in the shortest possible time.
The main purpose is the safety investigations of
accidents and serious incidents to prevent future
occurrences
1.
Data elements vs occurrences:
I.
Need to identify specific data elements to be monitored (and reported) besides the high-level
occurrences listed by SG3;
II.
Non-critical occurrences reporting (e.g., near misses);
2.
ISMR roles and responsibilities:
I.
Identify roles of national/international authorities, including
I.
data accessibility/protection and
II.
development and sharing of safety recommendations
3.
Pending exchange with GRVA/WP29:
I.
Information sharing among safety authorities & Contracting Parties
II.
Reporting from other sources than the ADS manufacturers;
Outstanding Issues
(addressed)
“An unplanned event that did not result in injury, illness or
damage –
but had the potential to do so”
•
Near misses are precursors for crashes to come, their
analysis is of vital importance to prevent incidents
•
Near Misses could be included in the monitoring task to
fully exploit the possibility to find potentially harmful
behaviors in ADS before a collision occurs
•
Frequency of near miss is highly spread depending on
the field and on the way near misses are
detected/counted, nonetheless it is typically order of
magnitude higher than incident/accident
Near misses
1
Industrial Accident Prevention, A Scientific
Approach, Herbert W. Heinrich 1931.
ISMR Roles and responsibilities
Safety
recommendations
Short term and
Period reports
Safety
recommendations
SA Annual
Reports
WP29
•
Safety Authority (SA)
•
Can be the enforcement Authority or not; if not, it gives
recommendations to the enforcement Authority
•
Responsible for ISMR
data management at national level
•
Derives safety recommendations
and shares them at
higher level
•
Publish annual report
summarizing the level of ADS
safety.
Investigation Body (IB)
•
Independent/impartial body,
investigates occurrences
•
Issue an investigation report including (where applicable)
safety recommendations
•
Provides an Annual report
with evidence of investigations
ISMR Roles and responsibilities
MANUFACTURERS
AUTHORITIES
I
S
M
R
d
a
t
a
Reports & Safety
recommendations
•
Data collection should ensure its confidentiality, the protection of its
source and the confidence of the reporters
•
Sensitive safety information should be protected by preventing its use
for purposes other than safety.
•
Safety Authorities have to set up a confidential reporting scheme
and
to ensure that no personal details are ever recorded in the databases
both at national/international level.
Protection of Information
•
The final aim of ISM is to improve ADS safety through dissemination of lessons learned
•
A broader exchange of information and the dissemination of safety recommendations
should be ensured among the Contracting Parties, at international level
•
It is recommended that WP29 promotes and facilitates a broader exchange of information
Information sharing among safety authorities &
Contracting Parties
•
G
u
i
d
i
n
g
p
r
i
n
c
i
p
l
e
s
•
Access to the database for the authorities
•
Safety Recommendations publicly accessible
•
Promote cooperation among Contracting parties/authorities with regular exchange of
information
•
Limiting the reporting requirements to manufacturers will also limit the
amount and type of information covered by ISM, with a strong impact on
the achievable safety improvement.
•
E.g. identification of
traffic rules infringement is not possible through
data collected on-board the vehicle, and reporting by local authorities
and ADS vehicle users is needed.
•
Other transport sectors extend the operational reporting mechanism
also to
drivers
,
operators
,
users
,
traffic managers
, and
any other
person connected to the vehicle operation
.
Reporting from other sources
Interaction with other groups
I
n
S
e
r
v
i
c
e
M
o
n
i
t
o
r
i
n
g
A
n
d
R
e
p
o
r
t
i
n
g
F
R
A
V
-
I
S
M
R
c
o
u
l
d
i
n
t
e
r
a
c
t
f
o
r
t
h
e
i
d
e
n
t
i
f
i
c
a
t
i
o
n
o
f
t
h
e
p
e
r
f
o
r
m
a
n
c
e
t
o
b
e
m
o
n
i
t
o
r
e
d
a
n
d
r
e
l
e
v
a
n
t
m
e
t
r
i
c
s
/
t
h
r
e
s
h
o
l
d
s
E
D
R
/
D
S
S
A
D
-
I
S
M
R
c
o
u
l
d
i
n
t
e
r
a
c
t
t
o
i
d
e
n
t
i
f
y
t
h
e
d
a
t
a
t
o
b
e
c
o
l
l
e
c
t
e
d
f
o
r
t
h
e
c
h
a
r
a
c
t
e
r
i
z
a
t
i
o
n
o
f
t
h
e
o
c
c
u
r
r
e
n
c
e
s
F
G
4
A
I
A
D
-
I
S
M
R
could interact to identify the AI
performance and behaviors that
can be potentially monitored
during the operations
Thank you !
Interaction with other groups (Example)
I
S
M
R
F
R
A
V
Safety recommendations:
The ADS shall recognize the conditions and
boundaries of the ODD of its feature(s) pursuant to
the manufacturer’s declaration
Provision:
The ODD conditions and boundaries
(measurable limits) should be established by
the manufacturer.
Occurrence:
ADS operation outside its ODD
ODD Conditions to be monitored:
o
Precipitation (rain, snow)
o
Time of day (light intensity,
including the case of the use of
lighting devices)
o
Visibility
o
Road and lane markings
E
D
R
/
D
S
S
A
D
1
2
2
Data To be Collected
-
Environmental Condition data
-
Road data
-
Vehicle data
-
Etc..
3
Metric/Threshold:
Number of ODD excursion
4
In-Service Monitoring and Reporting (ISMR) is crucial for ensuring the ongoing safety of ADS vehicles on the road. It involves collecting in-service data to assess safety levels, identify risks, and facilitate continuous improvement. Manufacturers must establish monitoring programs, analyze critical occurrences, and report both critical and non-critical events to enhance safety culture and prevent accidents. ISMR also enables knowledge sharing and scenario catalogue development to further enhance ADS safety standards.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
VMAD-25-08 SG3 In Service Monitoring and Reporting VMAD#25 Meeting 11-12 April 2022 Espedito Rusciano
SG3-In Service Monitoring and Reporting, Objectives and Deliverables In-Service Monitoring and Reporting (ISMR) addresses the in- service safety of the ADS after its placing on the market (operational experience feedback loop) It relies on the collection of in-service data(fleet monitoring) to assess whether the ADS continues to be safe when operated on the road and to identify safety risks This data collection can also be used for the identification of new scenarios to support the development of the Scenario Catalogue. ISMR allows the whole ADS community to learn from major ADS accidents/incidents through information sharing SG3 is developing procedures to confirm the in-service ADS safety and to improve the level of safety of ADS based on the evidence collected from the field operation In Service Monitoring And Reporting 1. Confirmation of safety Assessment 2. Feed Scenario Catalogue 3. Share information and recommendations
Monitoring, Reporting and Investigation Monitoring Process Monitoring Focus Overall Data collection and analysis Critical and not critical occurrences Critical occurrences Reporting Frequency Reporting Investigation Investigation Severity
In Service Monitoring Monitoring (Link with the AUDIT Pillar): Manufacturers should set up a monitoring program according to the SMS Requirements Vehicle data collection and analysis by the manufactures for reporting under ISMR, besides EDR/DSSAD Manufacturers are expected to collect data also from other accessible sources of data (e.g., customer reports) Monitoring ISMR Reporting On Board Recording Proactive approach for Safety which shall be integrated in the Safety Management System Beyond the scope of the occurrences reporting Increase safety by identifying trends and unusual or unsafe circumstances Data Transfer to Ground Station Processing and Analysis
In Service Reporting Occurrence Reporting: Occurrence refers to any safety-related event involving a vehicle equipped with an ADS. For reporting, two different categories of occurrences are defined: critical and not critical occurrences. The manufacturer should report, as required by the Authority, on both critical and non-critical occurrences Monitoring ISMR Reporting Safety Culture The main purpose of in-service reporting is the prevention of accidents and incidents and not to attribute blame or liability. (Just culture) Just Culture
Types of Reporting Short-term reporting occurrences that require the manufacturer to take remedial action when data provides evidence of the ADS posing an unacceptable in-service risk. (within 1 month) a) indications of failure to meet safety requirements b) critical occurrence where the ADS was at fault c) other safety-relevant performance issues Periodic reporting - reporting in the form of aggregated data (per hour of operation or driven km) for ADS-vehicle type and related to ADS operation (at least every year) a) no inconsistencies have been detected compared to the ADS safety performance assessed prior to market introduction; b) the ADS respects the performance requirements set by FRAV and as evaluated in the test methods developed by VMAD; c) any newly discovered significant ADS safety performance issues have been adequately addressed and how this was achieved.
Occurrences list SHORT-TERM REPORTING [1 Month] X PERIODIC REPORTING [6 Month/1 Year] X OCCURRENCE 1.a. Safety critical occurrences known to the ADS manufacturer or OEM 1.b. Occurrences related to ADS operation outside its ODD 1.c. ADS failure to achieve a minimal risk condition when necessary 1.d. Communication-related occurrences 1.e. Cybersecurity-related occurrences 1.f. Interaction with remote operator if applicable 2.a. Driver unavailability (where applicable) and other user-related occurrences 2.b. Occurrences related to Transfer of Control failure 2.c. Prevention of takeover under unsafe conditions 3.a. Occurrences related ADS failure 3.b. Maintenance and repair problems 3.c. Occurrences related to unauthorized modifications 3.d. Modifications made by the ADS manufacturer or OEM to address an identified and significant ADS safety issue 4. Occurrences related to the identification of new safety-relevant scenarios X X X X X X X X X X X X X X X X
Investigation Occurrence Investigations Focus on critical occurrences, but it can include also other occurrences It is expected an immediate notification to the authority on occurrences which can pose an immediate risk to public safety. An Independent body conducts investigations according to its mandate The investigation report, containing where appropriate safety recommendations, should be made available to all parties involved in the shortest possible time. The main purpose is the safety investigations of accidents and serious incidents to prevent future occurrences
Outstanding Issues (addressed) 1. Data elements vs occurrences: I. Need to identify specific data elements to be monitored (and reported) besides the high-level occurrences listed by SG3; II. Non-critical occurrences reporting (e.g., near misses); 2. ISMR roles and responsibilities: I. Identify roles of national/international authorities, including I. data accessibility/protection and development and sharing of safety recommendations Pending exchange with GRVA/WP29: II. 3. I. Information sharing among safety authorities & Contracting Parties II. Reporting from other sources than the ADS manufacturers;
Near misses An unplanned event that did not result in injury, illness or damage but had the potential to do so Near misses are precursors for crashes to come, their analysis is of vital importance to prevent incidents Near Misses could be included in the monitoring task to fully exploit the possibility to find potentially harmful behaviors in ADS before a collision occurs Frequency of near miss is highly spread depending on the field and on the way near misses are detected/counted, nonetheless it is typically order of magnitude higher than incident/accident 1Industrial Accident Prevention, A Scientific Approach, Herbert W. Heinrich 1931.
ISMR Roles and responsibilities SA & IB SA & IB SA & IB SA & IB Short term and Period reports Safety recommendations WP29 SA Annual Reports Safety recommendations
ISMR Roles and responsibilities Safety Authority (SA) MANUFACTURERS Can be the enforcement Authority or not; if not, it gives recommendations to the enforcement Authority Responsible for ISMR data management at national level Derives safety recommendations and shares them at higher level AUTHORITIES Publish annual report summarizing the level of ADS safety. Investigation Body (IB) ISMR data Independent/impartial body, investigates occurrences Issue an investigation report including (where applicable) safety recommendations Reports & Safety recommendations Provides an Annual report with evidence of investigations
Protection of Information Data collection should ensure its confidentiality, the protection of its source and the confidence of the reporters Sensitive safety information should be protected by preventing its use for purposes other than safety. Safety Authorities have to set up a confidential reporting scheme and to ensure that no personal details are ever recorded in the databases both at national/international level.
Information sharing among safety authorities & Contracting Parties The final aim of ISM is to improve ADS safety through dissemination of lessons learned A broader exchange of information and the dissemination of safety recommendations should be ensured among the Contracting Parties, at international level It is recommended that WP29 promotes and facilitates a broader exchange of information Organisations databases National databases WP29 Repository Guiding principles Access to the database for the authorities Safety Recommendations publicly accessible Promote cooperation among Contracting parties/authorities with regular exchange of information
Reporting from other sources Limiting the reporting requirements to manufacturers will also limit the amount and type of information covered by ISM, with a strong impact on the achievable safety improvement. E.g. identification of traffic rules infringement is not possible through data collected on-board the vehicle, and reporting by local authorities and ADS vehicle users is needed. Other transport sectors extend the operational reporting mechanism also to drivers, operators, users, traffic managers, and any other person connected to the vehicle operation.
Interaction with other groups FRAV-ISMR could interact for the identification of the performance to be monitored and relevant metrics/thresholds Safety Performance Performance metrics/thresholds FRAV EDR/DSSAD-ISMR could interact to identify the data to be collected for the characterization of the occurrences Data vs Occurrences (including near misses) Data collection and storage In Service Monitoring And Reporting EDR DSSAD FG4AIAD-ISMR could interact to identify the AI performance and behaviors that can be potentially monitored during the operations AI real time monitoring AI behavioral drifting FGAI4AD
Interaction with other groups (Example) EDR/ DSSAD FRAV ISMR Data To be Collected Environmental Condition data Road data Vehicle data Etc.. Occurrence: ADS operation outside its ODD Safety recommendations: The ADS shall recognize the conditions and boundaries of the ODD of its feature(s) pursuant to the manufacturer s declaration Provision: The ODD conditions and boundaries (measurable limits) should be established by the manufacturer. - - - 1 - 2 ODD Conditions to be monitored: o Precipitation (rain, snow) o Time of day (light intensity, including the case of the use of lighting devices) o Visibility o Road and lane markings 3 2 4 Metric/Threshold: Number of ODD excursion
