- BACKGROUND
- SERVICE PORTFOLIO
- SERVICE VALIDATION
- PARTNERSHIP
- COMMUNICATION
- PUBLICATIONS & DOCUMENTS
- GLOSSARY
- PROJECT AREA
contenu.php?arbo=5&page=
TECHNICAL VALIDATION PROTOCOL - VALIDATION PLAN
OPERATIONAL VALIDATION - CONCEPTS
OPERATIONAL VALIDATION PROTOCOL - VALIDATION PLAN TEMPLATE
INTRODUCTION
The validation of the services is a key driver to the achievement of a truly operational status. In this respect, the RISK-EOS Validation Protocol has two main purposes:
(1) it defines a generic process and methodology for performing the validation of the RISK-EOS present and future services. The basic principle of the generic validation process is a two-steps approach:
First, a “technical validation of the Service chain” must, for each product, verify the compliance of the products to the key performance parameters (accuracy, reliability …) specified by the users. For this, test campaigns must be carried out, which consist in comparing the delivered products to ground truth data or reference data. The test campaign process, test results, and conclusions, are documented in the Service Validation Report (C6),
Then and in complement, the “operational validation” shall allow the end-users to assess the adequacy of the general service characteristics (delivery mode, delays, support…) to their needs. This is documented in the Service Utility Reports (U7).
(2) it defines the specific validation process and validation plans for the services which have been delivered as part of the Consolidation Phase (2003-2004) and for the new ones included in the Scaling Up Phase service portfolio (2005-2008).
The validation protocol document [C5] defines in particular:
the validation concepts,
the validation actors and roles,
the minimum performance per product for a given service,
the template to define the validation plan for the Technical Validation,
the template to define the validation plan for the Operational validation,
tests plans and procedures for each service chain and products to be performed.
VALIDATION CONCEPTS
| Type of validation | ||||
| Subjects | Purpose | Reference data | Output Report | |
| Scientific | Methods/ Algorithms | To perform a scientific review of the methodology used to derive products. | Scientific publications, bibliography | S2 – Methods Compendium |
| Technical | Product | To perform a quantitative measurement of a selected number of parameters useful to evaluate the service/products performances by comparing such parameters with the performances declared in the Service Specifications [S5] and in this document [C5]. This type of validation is needed to be a Risk EOS qualified service chain | Accurate set of ground truth data or reference data as defined in the validation protocol, chosen in order to exhaustively derive the figures needed to perform comparisons with the specified service/products performances | C6 – Service Validation Report |
| Operational | Service | Following the technical validation, this type of validation has to be performed in order to assess the results of the service deployment campaigns. |
The service operational validation will be based on the results of the service validation tests and the users feedbacks acquired by proposing proper questionnaires after service deployment |
U7 – Service Utility Report |
| Product | ||||
| At product level operational validation will be based on a subset of test according to reference data availability as routinely acquired by the authorities in charge of the specific risk management (existing ground surveys, aerial photographs, maps) | S6 – Service Operation Report | |||
VALIDATION ACTORS AND ROLES
| ACTOR | MEMBERS | TECHNICAL VALIDATION ROLE | OPERATIONAL VALIDATION ROLE |
| Independant body | ESA Representative | Reviews and Approvals Checks for Readiness Product Acceptance |
|
| Technical Validation Team | Service developer Service provider |
Algorithm tuning Data processing Algorithm performance analysis Technical support and maintenance Technical validation execution |
|
| Operational Validation Team | Service provider User operators Main Service Provider |
Operational Validation planning Collection/pre-processing of reference data Validation tools preparation Analysis and comparison Service/Product performance assessment by preparation and support to compilation of questionnaires Reporting |
|
| Service Network Coordination | Service Network Coordinator, Service Network Technical Coordinator. |
Maintain the Service documents links to RISK-EOS validation process | Review the Service Operation Reports [S6], Review the Service Utility Report [U7], Review the Service Validation Reports [C6], Update Validation Protocol document [C5], Update Service Prospectus document [S3], Update the Service Specification [S5]. |
| Reference User | Representatives of key users involved | Endorsement of Product Acceptance | Endorsement of Service Acceptance |
PRODUCT ACCURACY: GENERIC PARAMETERS AND MINIMUM EXPECTED PERFORMANCES
The Validation Protocol document recalls per service and products (when applicable), the performance parameters which will be used to assess the product quality and the reference minimum expected performances for these parameters.
The Service Network coordinator will compare the performances documented in the Validation Results document [C6] with the minimum expected performances for the service to approve the technical validation step.
For most of the products minimum expected performances for geometric and thematic accuracy are documented.
TECHNICAL VALIDATION PROTOCOL – VALIDATION PLAN
The Validation protocol document [C5] defines the validation plan related to the technical validation process for the Service Chain. The technical validation is a step to be performed for a service chain to be RISK-EOS qualified.
The validation plan must be structured as follows:
Specification of the product subject of validation
Information product description shall provide the characterization of the products in line with information fields presented in product specification standards:
Product general description,
Geographical coverage,
Information layers,
Expected product performance: resolution, scale, accuracy and potential sources of uncertainty,
Product formats,
Needs for main and auxiliary input data.
Selection and description of the validation site
The validation plan shall provide a justified selection of the validation site, describing:
Hazard phenomena in the area,
Vegetation and topography particularities,
Land use,
Statistics of incidence of the hazard in the area,
Impact on socio-economic assets.
Identification of actors and roles
Validation actors and roles shall be identified for each validation site.
Identification of validation requirements
The validation process will start with the definition of particular validation requirements established with involvement of users responsible for the management of the hazard to be dealt by the information product subject of validation. Those requirements shall regard aspects as those shown below:
Pre-requisites established for product performance,
Pre-requisites established by the user for field or independent data collection when applicable,
Constraints to not interfere user operational procedures when applicable,
Constraints in terms of personal availability when applicable,
Schedule constraints,
Normally, users involved in validation processes provide GIS information layers for information product generation. Formats, compatibility and content requirements shall be sketched.
Specification of validation data
The validation plan shall specify the data set required for all activities within the technical validation process.
Specification of validation means and facilities
Means and facilities required for validation shall be identified:
EO data processing facilities,
Product generation processors,
Database management tools,
GIS data management tools,
Validation analysis tools.
Brief description of validation tasks & scheduling
The plan shall depict a brief description of the tasks to be carried out for product validation. In addition this shall include a schedule of activities reflecting key milestones and expected outputs.
Preparation
Procurement of EO data,
Compilation of samples of field data,
Compilation GIS data.
Processing
EO data processing,
Tuning of product generation algorithms,
Product generation,
Product customization.
Analysis
Cross comparison of reference data and synthetic information products,
Product quality and performance assessment,
User feedback.
Technical validation analysis
Methodology definition
The methodology to be used for the technical validation analysis process shall be sketched for each product and for each validation site. The methodology shall be presented using a flow diagram stressing input data, steps for data comparison, iterative processes, decision blocks, outputs, etc.
Validation criteria and measures of success
The validation plan shall establish a set of validation criteria in order to ensure that the synthetic product performances fulfill the product characterization reflected in the specifications. Thus, product performances shall be derived from cross analysis and comparison of the synthetic products with independent and more accurate precise measurements collected during field campaigns.
Product quality and performance assessment
Once synthetic products are generated and the parameter databases are populated, a deep analysis shall be carried out to assess product quality and performances.
This activity will start with the production of statistics and figures of merit envisaged during the validation planning phase. Those figures shall be compared with user requirements and product specifications.
Special effort shall be dedicated to detection of unexpected deviations between measures with field reference data and observations from space. Features that might affect such deviations shall be conveniently recorded fur further analysis.
Validation results shall be conveniently justified through tabular and graphical outputs that demonstrate compliance to the validation criteria previously established.
Analysis of errors and deviations
Deviations detected during the comparison process shall be carefully analyzed looking for potential causes of errors and deviations detected: poor quality of reference data, lack of accuracy in GIS data aimed to support fire geo-referencing of space data, time lasted from hazard event occurrence until observations from space, lack in data sets aimed to support algorithm tuning before processing, under performing of processing algorithms, bad adaptation of algorithms to different sites, etc.
The analysis shall also identify uncertainties in the data sets used, analyzing the impact on the results obtained from the validation process.
Reporting of product quality
Service acceptance results and service quality shall be reported in the corresponding ‘Service Validation Report’ documentation [C6].
User evaluation
At the end of the validation process, a user evaluation and feedback is necessary in order to close the product generation procedure before facing the service acceptance under an operational scenario.
User evaluation shall be conducted through the following guidelines:
Presentation of product and service characterization,
Presentation of sample products overlapped with validation reference data and user GIS cartography,
Presentation of validation results, justifying possible errors or deviations, and if the case, the proposed enhancements,
Remark strong and weak points of products,
Fill in user questionnaires recording user feedback.
OPERATIONAL VALIDATION - CONCEPTS
There are 2 concepts linked to operational validation:
Service Operational Validation,
Product Operation Validation
Service Operational Validation
This validation has to be performed after every deployment campaign. The service Operational Validation results consist in the assessment of the users evaluation of the service over the complete use and life-cycle and the impact of products and services delivered to a single end-user-organization over a fixed period of time.
The Service Operational Validation reference specifications are agreed between the Main Service Provider and the Users in the relative Service Level Agreement.
The output report is the Service Utility Report [U7] whose production is mandatory in order to maintain the Service Qualification.
Product Operational Validation
This validation can be performed after every deployment campaign on the basis of the ground truth data made available from the users or other reference data. The Product Operational Validation reference specifications are the same applicable to the technical validation, which are specified in the Service Specification [S5] and in the present document [C5].
The results are documented in the Service Operation Report [S6] when applicable.
OPERATIONAL VALIDATION PROTOCOL – VALIDATION PLAN TEMPLATE
Specification of the subject of service validation
Service description shall provide the characterization of the service, including brief reference to associated products, in line with service specification documents:
Service concept: brief description of the service concept and purpose,
Geographical coverage: European, country, or targeted region,
Temporal coverage of the service: period of the year covered by the service,
Timeliness: time intervals for information refreshing,
Associated information products: summary of information products associated to the service,
Expected performance: timeliness and specific delivery times, processing times, delivery delays, product integrity and quality, effectiveness rate, false alarm rate, missed alarms rate,
Service supply and access: way under which the user access the service; some of them, might be supplied systematically after user subscription, whereas others could be only delivered on user demand,
Delivery channel: either FTP, email, CDROM, data streaming, etc.
Selection of the geographical coverage of the pilot site
RISK-EOS services will be validated on reduced pilot sites where all variables affecting the validation process may be under control. The site characteristics will be explained as commented in paragraph .
Identification of actors and roles
Identification of actors and roles as reflected in paragraph .
Identification of user acceptance requirements
As in the case of product validation, service user acceptance shall observe a set of requirements:
Pre-requisites established for service performances,
Pre-requisites established by the user for independent data collection,
Constraints to not interfere user operational procedures,
Constraints in terms of user personnel availability,
Means and facilities required at user premises for information product management and interpretation.
Identification of data sources and products
As per service validation user acceptance procedures shall identify sources for the different data sets to be used. In addition, the plan shall sketch the list of associated products.
Specification of service validation means and facilities
Means and facilities required for service validation and acceptance, taking into account that user acceptance will imply an end-to-end commissioning of the production chain:
Links to EO data sources,
EO data processors,
Product generation processors,
Communication links to end users,
Proper end user terminals for information product display and management.
Brief description of the end-to-end service validation procedures
The plan shall depict a brief description of the tasks to be carried out for product validation. In addition, this shall include a schedule of activities reflecting milestones and expected outputs.
Preparation
Setting up links to external data providers (EO data, and exogenous data),Setting up infrastructures for data reception,
Processing and information product distribution,
Setting up links from service providers to pilot end users.
Commissioning
Execution of end-to-end simulations and record of operations log and key parameters.
Pilot campaign
Training to users in service and data management and product interpretation,
Pilot operations of services with delivery of final products during a pilot campaign with high involvement of users,
Support to users during pilot operations and ensure record of events and key parameters for further statistics,
Support to users in evaluation of service quality during the pilot campaign, compiling user feedback and recommendations.
Analysis
Cross comparison of recorded parameters with service specifications and requirements,
Analysis of service quality and performance assessment,
User feedback.
Cross comparison with service specifications
Methodology definition
The methodology to be used for service operational validation and user acceptance shall be sketched for each validation site. The methodology shall be presented using flow diagrams, stressing operations, event recording, decision blocks, outputs, etc.
Service Validation criteria and measures of success
Service acceptance by the user involves end-to-end demonstrations to verify compliance of service specifications. Then, commissioning sessions shall be conducted to check a pre-define list of service key parameters aimed to allow measurement of operational validation success and satisfaction of user expectations. Thus, the top level criteria should consider:
Service management
Internal quality management,
Efficiency of the processing chain,
Handling of non-conformances,
Timely correction actions,
Sufficient allocation of resources.
Processing chain
Processing chain reliability,
Correct handling of input data,
Efficiency,
Scalability of the production chain.
Service delivery
Information refresh frequency,
Availability/ accessibility/ reliability,
Timeliness, information delivery delays with respect to tolerance specifications, in case of services systematically delivered; information delivery delay from user request, in case of services delivered on user demand; processing delays; response time of servers.
User interaction
End user contact,
End user information,
Compliance to user requirements,
Service evaluation by user,
User satisfaction.
Service performance assessment
Standard service commissioning and user evaluation forms shall be prepared as basis to record key parameters and events that might take place during the commissioning period.
All parameters defined in the validation criteria shall be checked and compared with specifications looking for compliance or deviations whose causes have to be determined, proposing the corresponding correction actions to bring them back to specifications.
Service quality assessment and audit
During the commissioning phase, the validation team shall conduct service key parameters measurements ensuring the collection of a representative sample of figures that allows generation of statistics aimed at determine the degree of compliance of specifications and user requirements, and therefore, the level of quality under which the service is provided. In addition, an independent audit of quality management is recommended.
Analysis of problems
Identification of errors and problems and non-conformances detected either during commissioning phase or during pilot operations campaign. Those problems or non-conformances shall receive the proper response in the shortest period of time. If as result of the analysis of causes major divergences are detected, corrective solutions shall be proposed to be implemented in the service provision chain for service improvement.
Reporting of service quality
Service acceptance results and service quality shall be reported in the corresponding ‘Service Validation Report’ documentation.
Reporting of user feedback
User acceptance of the service shall be reported in the corresponding ‘Service Utility Report’ documentation.
