Terrestrial File Transfer Concept: Design Goals and Protocols

Slide Note

This document discusses the concept of terrestrial file transfer as presented in a series of slides from an ESA event in Noordwijkerhout, The Netherlands. It covers the purpose, design goals, protocols, and transport protocols involved in exchanging files between agencies for mission design, operations, and post-mission activities. The emphasis is on simplicity, use of off-the-shelf technologies, and secure protocols like SSH and TLS for data transfer over networks.

lacrosse_g Follow

Uploaded on Dec 12, 2024 | 0 Views

Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

E N D

Presentation Transcript

Terrestrial File Transfer - Concept File Transfer BOF Noordwijkerhout, The Netherlands 02-Apr-14 Colin R. Haddow, ESA/ESOC ESA UNCLASSIFIED For Official Use

Its not rocket science File Transfer BOF| 02-Apr-14| ESA | Slide 2 ESA UNCLASSIFIED For Official Use

Purpose File exchanged between agencies may be used in, but are not limited to; Mission design, i.e. in investigating the feasibility of a mission with respect to the support available from another agency. Mission operations, i.e. to transfer files required for the successful operation of a mission between two or more agencies . Post Mission activities such as archiving data related to the mission. File Transfer BOF| 02-Apr-14| ESA | Slide 3 ESA UNCLASSIFIED For Official Use

Terrestrial File Transfer Conceptual Model Agency A Agency B Internet Internal Network Internal Network Firewall Firewall DMZ DMZ File File File Transfer Transfer Host Transfer Host File Transfer BOF| 02-Apr-14| ESA | Slide 4 ESA UNCLASSIFIED For Official Use

Design Goals The design goals behind this concept are; Keep the design as simple as possible whilst meeting the perceived needs Use off the shelf technologies, this concept is dealing with point-to-point terrestrial file transfer, something that occurs an enormous number of times per day over the internet File Transfer BOF| 02-Apr-14| ESA | Slide 5 ESA UNCLASSIFIED For Official Use

Protocols The protocol to be used needs to be able to provide security for the file transfer, e.g. it should not send user ID and/or passwords in plain text (thus ruling out standard FTP). should conform to a recognized standard readily available on a wide range of operating systems capable of supporting both IPv4 and IPv6 network addresses. designed for secure data communications over an insecure network. File Transfer BOF| 02-Apr-14| ESA | Slide 6 ESA UNCLASSIFIED For Official Use

Transport Protocols Two widely adopted transport protocols are potential candidates: Secure Shell (SSH) SSH is a cryptographic network protocol providing remote command-line login, remote command execution and other secure network services between two networked computers. Transport Layer Security (TLS) Transport Layer Security (TLS) and its predecessor, Secure Sockets Layer (SSL), are cryptographic protocols which are designed to provide communication security over the Internet. Both allow the use of certificates for login, thus making scripting of file transfer straightforward. Both are available for a wide range of operating systems File Transfer BOF| 02-Apr-14| ESA | Slide 7 ESA UNCLASSIFIED For Official Use

Transfer Protocols Selection of transfer protocols will largely be determined by the transport protocol. Some possibilities are outlined below. SCP and SFTP (both runs over SSH) SCP Secure Copy, copies files between hosts on a network and uses the same authentication and provides the same security as SSHSFTP Secure File Transfer Program, is an interactive file transfer program, similar to FTP. It may also use many features of SSH, such as public key authentication and compression WebDAV (runs over TLS) Web Distributed Authoring and Versioning (WebDAV) is an extension of the Hypertext Transfer Protocol (HTTP) that facilitates collaboration between users in editing and managing documents and files stored on World Wide Web servers. It provides a framework for users to create, change and move documents on a server; typically a web server or web share. File Transfer BOF| 02-Apr-14| ESA | Slide 8 ESA UNCLASSIFIED For Official Use

File Types A considerable number of the files that are likely candidates to be transferred via the TFT are likely to conform to CCSDS standards. it is feasible that a SANA repository could be set up which would contain details of what a particular file type was, possible including the definition of the appropriate metadata for that file type. It should however be recognized that it is also likely that a number of the files to be transferred may not conform to a CCSDS (or other recognized international standard). The transfer of these files by means of the TFT must therefore not be precluded and it is therefore essential that the method of associating metadata with files, and the specification of the metadata, is such that this is readily expandable such that it can readily be used to transfer files that are not defined in the SANA registry. File Transfer BOF| 02-Apr-14| ESA | Slide 9 ESA UNCLASSIFIED For Official Use

File Packaging There are a number of different technologies readily available on a variety of platforms that could be used for wrapping the files, ZIP TAR PAX JAR With exception of JAR none of the above provide a direct mechanism for associating metadata with the wrapped files. Need to ensure that what is chosen can support large data volumes Some versions of ZIP and TAR limited to approx 4Gbytes File Transfer BOF| 02-Apr-14| ESA | Slide 10 ESA UNCLASSIFIED For Official Use

Metadata As previously noted only JAR allows direct association of metadata with wrapped files. Can overcome this by providing a manifest files containing the required metadata Ways to encode required metadata include XML would require schema definition XFDU existing CCSDS standard File Transfer BOF| 02-Apr-14| ESA | Slide 11 ESA UNCLASSIFIED For Official Use

File considerations File and Directory naming Permitted characters File names Manifest file name Directory names Processing Metadata must contain any additional data required for further processing Ensuring all files have been received File counter in metadata ? Further transfers Injection into/from CFDP Injection into/from agency specific file transfer systems File Transfer BOF| 02-Apr-14| ESA | Slide 12 ESA UNCLASSIFIED For Official Use

Basic Operations Required Operations list get wildcards, i.e. mget) reget continue a failed file transfer from the apparent point of failure. put store a local file on the remote machine (probably want to support wildcards, i.e. mput) delete delete a file on the remote machine cd change directory on the remote machine mkdir make a directory on the remote machine (Is this really needed if we define a directory structure ?) obtains information about a file or directory retrieve a copy of the file (probably want to support reget acts like get, except that it can be used to File Transfer BOF| 02-Apr-14| ESA | Slide 13 ESA UNCLASSIFIED For Official Use

Hosts and Accounts Hosts Ideally the hosts for both the sender and receiver of the files would be transparently redundant, i.e. if the prime machine fails the backup machine should transparently replace it. Host names could be stored in a SANA registry. If an approach based on TLS/WebdAV were to be adopted then, as this is an extension of http it would be possible to store the appropriate end points in a SANA registry. Accounts To keep things simple it is suggested that one user account per mission be foreseen. Login to this account could be by conventional user account/password. It would be however be preferable (and would certainly make automation easier) if login authentication was carried out by means of public-private key pairs. . File Transfer BOF| 02-Apr-14| ESA | Slide 14 ESA UNCLASSIFIED For Official Use

Transfer of Files Transfer of files There are a number of possibilities with respect to where the transfer of files is instigated, particularly if it is assumed that the transfer of files is bi-directional, i.e. files go from one agency to a second, but files are also returned from the second agency to the first. Simplest approach would be to assume that files are always pushed. Can we assume this ? Is any form of notification of files being available required ? File Transfer BOF| 02-Apr-14| ESA | Slide 15 ESA UNCLASSIFIED For Official Use

Directory Structure Previously suggested that there is one account per mission. This being the case it is proposed that there are, within this account, only 2 directories; in-tray This directory would be used as the target directory for files pushed to the node. It is assumed that once a file has been found in the in-tray it will be moved to another location for further processing. out-tray This directory would be used either as a staging area for files that are to be pushed to the other agency or , if it is not permitted to push the files, would be where the files would be pulled from by the receiving agency. It is assumed that once a file has successfully been transferred from the out-tray it will be deleted from the out-tray or alternatively moved to an archive, in any event it will not remain in the out-tray directory File Transfer BOF| 02-Apr-14| ESA | Slide 16 ESA UNCLASSIFIED For Official Use

Security Not completely clear what is required but I would assume that at least the ability to optionally provide a hash value for each of the encapsulated files is The mechanism for permitting a SHA-2 hash value for all wrapped files is to enclose the hash value in the meta data in the manifest file. With respect to providing the hash value for the manifest and wrapper files this can be done by appending the appropriate hash value to the full filename. Is signing enclosed files needed ? Is encrypting enclosed files needed ? Is signing manifest file needed ? Is encrypting manifest file needed ? Is signing wrapper file needed ? Is encrypting wrapper files needed ? File Transfer BOF| 02-Apr-14| ESA | Slide 17 ESA UNCLASSIFIED For Official Use

Service Agreement For the terrestrial file transfer between agencies it should be foreseen that some form of service agreement will be required. This would need to address the following points. Points of contact Hosts Login Credentials Notification Mechanism (if required) Data Volume Retention Time Availability Transfer Rate Miscellaneous If multiple transport/transfer/wrapping mechanisms are in standard then this would indicate what is supported Anything else ? File Transfer BOF| 02-Apr-14| ESA | Slide 18 ESA UNCLASSIFIED For Official Use

Book Structure With respect to defining the most aspects this should be straightforward. Defining the required metadata for known file types is a bit trickier Don t want to have to issue the complete book again just because a new file type has been added or the metadata for an existing type has changed. Put each metadata definition in a separate annex so only that annex has to be reissued if it changes ? Put metadata definitions in SANA ? Other alternatives ? File Transfer BOF| 02-Apr-14| ESA | Slide 19 ESA UNCLASSIFIED For Official Use