DFS in Dyalog File Server
October 20-24
Dyalog File Server
Version 2.0
Morten Kromberg
CTO, Dyalog LTD
Dyalog’13
A
g
e
n
d
a
•
What is the DFS?
•
Motivation & Benefits
•
Version 2.0 Features
•
DFS Road Map
•
Availability
DFS v2.0
3
R
e
c
a
p
:
W
h
a
t
i
s
D
F
S
?
•
The
D
yalog
F
ile
S
erver
•
A ”plug compatible” replacement for direct
access to component files (DCF).
•
(and from v2.0, also ”native files”)
•
A Client/Server system in which file
operations are only possible by
communicating with DFS Server Processes
–
No direct access to component files from client
–
However, the underlying storage mechanism *is*
”normal” component files.
DFS v2.0
4
DFS v2.0
5
B
a
c
k
u
p
S
e
r
v
e
r
Takes backups
File Server 2 Delegate
1
Performs long-running
tasks like FCHK or FCOPY
on behalf of a server
V
2
.
0
DFS Architecture Diagram
F
e
a
t
u
r
e
s
•
Dyalog Component File Functions
•
Dyalog Native File Functions
–
Additional File & Folder utilities
•
SHAREFILE/AP features
–
FHIST, FUSERNO, FWHOIS
–
(but no ”Library Number” concept)
•
Management Console / Admin API
The Dyalog File Server
6
D
F
S
S
e
c
u
r
i
t
y
•
Access to the DFS requires a login
•
DFS has a table of valid users
•
Users can either have passwords
validated by DFS itself, or using
”Integrated Windows Authentication”
The Dyalog File Server
7
I
s
s
u
e
s
w
i
t
h
D
C
F
•
System administrators must create file shares to
accomodate APL applications
•
Security: The DCF essentially requires full access to
the underlying files
•
Access Matrices only apply to access via DCF
–
Notepad, vi and EMACS ignore them
–
Data is exposed and vulnerable
•
DCF is quite sensitive to network reliabiliy and
performance
–
In particular, File Holds / Locks can perform badly
DFS v2.0
8
B
e
n
e
f
i
t
s
o
f
D
F
S
(
1
/
2
)
•
Security: The DFS only allows access to files
through validated channels.
–
Architecture allows encryption and compression.
•
Access matrices have absolute authority
–
Other constraints can be added
•
No file shares are required
–
A handful of TCP ports is all you need for file
system access
DFS v2.0
9
B
e
n
e
f
i
t
s
o
f
D
F
S
(
2
/
2
)
•
Ability to monitor file system activity
–
Who has that file held (or tied)? (& Kill them!)
•
DFS recovers automatically from network glitches
and server restarts
–
If the server session is still active, no problem
–
If closed, reconnection happens unless FHOLDS existed
–
File Holds are queued by a separate process
•
Simultaneous connections from one client session
–
Two threads can be independent clients of the file system
–
Can do independent FHOLDs
DFS v2.0
10
P
o
t
e
n
t
i
a
l
L
o
n
g
e
r
T
e
r
m
B
e
n
e
f
i
t
s
.
.
.
•
Server-side compression and caching
•
Server-side ”Stored Procedures”
•
Mirroring to a 2nd DFS installation for
Disaster Recovery
•
Transactions
The Dyalog File Server
11
I
m
p
l
e
m
e
n
t
a
t
i
o
n
•
>90% of DFS-related code is written in APL
•
Monitor is a MiServer application
•
Some extensions to Conga and the Component Files
Integrated Windows Authentication
–
Support for single server task ”representing” multiple clients
–
Backups
•
Over time, more C code may be used, however ...
•
Keeping the core in APL allows
–
Stored procedures
–
Other user-defined extensions
–
... rapid progress
The Dyalog File Server
12
C
u
r
r
e
n
t
P
e
r
f
o
r
m
a
n
c
e
•
Sometimes slower on reads due to lack of caching
•
Faster on writes due to use of exclusive ties on the
server
•
Faster and (MUCH more predictable) on FHOLD with
high user loads
•
Current target is to have better overall performance
than Windows LAN files
•
Long term target is to be competitive on all platforms.
The Dyalog File Server
13
P
e
r
f
o
r
m
a
n
c
e
F
i
g
u
r
e
s
The Dyalog File Server
14
T
r
a
n
s
a
c
t
i
o
n
s
P
e
r
S
e
c
o
n
d
(
p
e
r
c
l
i
e
n
t
)
”Transaction” (on 10k components):
⎕
FHOLD tn
⍝
Start Transaction
data←
⎕
FREAD tn,cn
⍝
Read
data
⎕
FREPLACE tn,cn
⍝
Update
⎕
FHOLD
⍬
⍝
”Commit”
Hardware: i7 server & i7 laptop client.
D
C
F
–
D
F
S
D
i
f
f
e
r
e
n
c
e
s
•
DFS Access Matrix is always enforced; you
can not log in as ”user 0”.
•
(Optionally), all DFS functions that refer to
user numbers report user names:
FHIST, FRDAC, FSTAC, FRDCI
The Dyalog File Server
15
D
C
F
–
D
F
S
D
i
f
f
e
r
e
n
c
e
s
•
DFS
FLIB
reports exclusively tied files
–
So will
⎕
FLIB
in v13.2
•
DFS FHIST does not record latest file tie
–
From 13.2,
⎕
FHIST
will also not do this
•
DFS can tie more than 253 files at once
•
DFS file holds are not released on return to
immediate execution
–
Not a bug, it’s a feechure
The Dyalog File Server
16
F
e
a
t
u
r
e
s
o
f
v
2
.
0
•
Online Backup and Restore
•
Detailed Performance Monitor
•
Improved performance of long-running tasks
–
FCOPY, FCHK, MCOPY, MMOVE, ...
•
Faster FLIB
•
Support for Native File Functions
–
Including a number of extensions
•
Bootstrap applications direct from DFS
DFS v2.0
17
O
n
l
i
n
e
B
a
c
k
u
p
a
n
d
R
e
s
t
o
r
e
•
”24x7x365” operations: Backup and Restore without
taking DFS down
•
Takes snapshot of all component files managed by
DFS, at a single point in time
•
Slight performance degradation during backup
•
Files updated while backup is in process will grow
due to requirement to have old&new data available
•
Restored files must not be tied
–
Use monitor to identify (& if necessary kill) tasks using a file
DFS v2.0
18
V
2
.
0
S
p
e
e
d
u
p
s
•
Faster FLIB: Configuration switch to rely
on ”.DCF” extension
•
”Delegate Processes” to handle long-
running tasks:
•
FCHK, FCOPY
•
MCOPY, MMOVE, MDELETE
•
Queueing system rewritten
DFS v2.0
19
B
o
o
t
s
t
r
a
p
M
e
c
h
a
n
i
s
m
•
DFS server will support download of
application ”manifests” via HTTP
•
Bootstrap loader can maintain APL
applications without a network share
DFS v2.0
20
N
a
t
i
v
e
F
i
l
e
F
u
n
c
t
i
o
n
s
•
Full set of Native File Functions
supported by Dyalog APL
•
Several extensions
–
Metadata: NLIB, NATTRIBUTES, SPACE
–
Manage folders with NMKDIR, NRMDIR
–
Handle multiple files and/or folders with
MCOPY, MDELETE, MMOVE
DFS v2.0
21
E
x
t
e
n
d
e
d
F
i
l
e
/
F
o
l
d
e
r
F
n
s
DFS v2.0
22
N
a
t
i
v
e
F
i
l
e
I
s
s
u
e
s
•
Lots of small file operations may perform poorly (no
client-side cache)
•
For component files, security is enhanced (access
matrices have absolute authority)
–
For native files, user ”impersonation” is not possible
–
All file access is peformed using DFS server account
•
Use of native files via DFS is probably only
recommended for some applications
DFS v2.0
23
P
e
r
f
o
r
m
a
n
c
e
M
o
n
i
t
o
r
•
For each file operation DFS will log:
–
File and User/Session Ids,
–
Time of Day, Number of open files at time of op
–
Elapsed time and CPU consumption in main
process and in ”delegate”
–
Length of file operation queue and time spent in it
–
Number of bytes transmitted in & out
•
Monitoring can be switched on & off from the
web monitor
DFS v2.0
24
P
e
r
f
o
r
m
a
n
c
e
A
n
a
l
y
s
i
s
•
Web monitor provides on-line analysis
of last 15 minutes of data
•
Off-line analysis of historical data
DFS v2.0
25
M
o
n
i
t
o
r
S
c
r
e
e
n
S
h
o
t
s
DFS v2.0
26
M
o
n
i
t
o
r
S
c
r
e
e
n
S
h
o
t
s
DFS v2.0
27
M
o
n
i
t
o
r
S
c
r
e
e
n
S
h
o
t
s
DFS v2.0
28
M
o
n
i
t
o
r
S
c
r
e
e
n
S
h
o
t
s
DFS v2.0
29
M
o
n
i
t
o
r
S
c
r
e
e
n
S
h
o
t
s
DFS v2.0
30
M
o
n
i
t
o
r
S
c
r
e
e
n
S
h
o
t
s
DFS v2.0
31
M
o
n
i
t
o
r
S
c
r
e
e
n
S
h
o
t
s
DFS v2.0
32
M
o
n
i
t
o
r
S
c
r
e
e
n
S
h
o
t
s
DFS v2.0
33
D
F
S
R
o
a
d
M
a
p
•
Performance, Performance, Performance
–
Client and Server-side caching
–
Multiple file operations in a single operation
–
”Stored Procedures” (server-side APL)
–
Compression
•
”Transactions”
–
For example, (
⎕
FHOLD tn)
could trigger the
beginning of a transaction and (
⎕
FHOLD
⍬
)
be a
”commit”.
•
Mirroring / Disaster Recovery Features
DFS v2.0
34
A
v
a
i
l
a
b
i
l
i
t
y
•
V1 released in October 2013: Launch customer ”in
production” for over a year, 2nd customer at work
•
General Availability scheduled on V2.0 release in
Q1/2014
–
Requires Dyalog APL v13.2 for backup feature
•
Not bundled with APL – sold separately as add-on to
runtime licences
–
Annual licence for 50-user server:
£3,000/32bit, £4,500/64bit
DFS v2.0
35
DFS (Dyalog File Server) is a client/server system that replaces direct access to component files. It offers features like monitoring, backup, and user authentication. DFS enhances security and performance compared to traditional file access methods like DCF. Learn about its architecture, features, and security measures in this informative content.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Dyalog File Server Version 2.0 Morten Kromberg CTO, Dyalog LTD Dyalog 13
Agenda What is the DFS? Motivation & Benefits Version 2.0 Features DFS Road Map Availability DFS v2.0 3
Recap: What is DFS? The Dyalog File Server A plug compatible replacement for direct access to component files (DCF). (and from v2.0, also native files ) A Client/Server system in which file operations are only possible by communicating with DFS Server Processes No direct access to component files from client However, the underlying storage mechanism *is* normal component files. DFS v2.0 4
DFS Architecture Diagram HTTP Monitor (Browser) Monitor Provides monitoring and instrumentation Exposes HTTP Interface V2.0 Locks Server Handles all FHOLD and NLOCK requests Backup Server Takes backups APL Client Session 1 Master Server Manages Login requests APL Client Session 2 File Server 2 Delegate 1 Performs long-running tasks like FCHK or FCOPY on behalf of a server File Server 2 /test APL Client Session n File Server 1 /prod File1 File Handler Thread 1 File3 File Handler Thread 2 File2 DFS v2.0 5
Features Dyalog Component File Functions Dyalog Native File Functions Additional File & Folder utilities SHAREFILE/AP features FHIST, FUSERNO, FWHOIS (but no Library Number concept) Management Console / Admin API The Dyalog File Server 6
DFS Security Access to the DFS requires a login DFS has a table of valid users Users can either have passwords validated by DFS itself, or using Integrated Windows Authentication The Dyalog File Server 7
Issues with DCF System administrators must create file shares to accomodate APL applications Security: The DCF essentially requires full access to the underlying files Access Matrices only apply to access via DCF Notepad, vi and EMACS ignore them Data is exposed and vulnerable DCF is quite sensitive to network reliabiliy and performance In particular, File Holds / Locks can perform badly DFS v2.0 8
Benefits of DFS (1/2) Security: The DFS only allows access to files through validated channels. Architecture allows encryption and compression. Access matrices have absolute authority Other constraints can be added No file shares are required A handful of TCP ports is all you need for file system access DFS v2.0 9
Benefits of DFS (2/2) Ability to monitor file system activity Who has that file held (or tied)? (& Kill them!) DFS recovers automatically from network glitches and server restarts If the server session is still active, no problem If closed, reconnection happens unless FHOLDS existed File Holds are queued by a separate process Simultaneous connections from one client session Two threads can be independent clients of the file system Can do independent FHOLDs DFS v2.0 10
Potential Longer Term Benefits... Server-side compression and caching Server-side Stored Procedures Mirroring to a 2nd DFS installation for Disaster Recovery Transactions The Dyalog File Server 11
Implementation >90% of DFS-related code is written in APL Monitor is a MiServer application Some extensions to Conga and the Component Files Integrated Windows Authentication Support for single server task representing multiple clients Backups Over time, more C code may be used, however ... Keeping the core in APL allows Stored procedures Other user-defined extensions ... rapid progress The Dyalog File Server 12
Current Performance Sometimes slower on reads due to lack of caching Faster on writes due to use of exclusive ties on the server Faster and (MUCH more predictable) on FHOLD with high user loads Current target is to have better overall performance than Windows LAN files Long term target is to be competitive on all platforms. The Dyalog File Server 13
Performance Figures Transactions Per Second (per client) Platform FNS on LAN DFS 1 client 28<32<34 46<51<59 2 clients 15<20<23 25<34<38 3 clients 5<15<21 20<22<24 4 clients 3< 9<23 15<16<17 Transaction (on 10k components): FHOLD tn data FREAD tn,cn data FREPLACE tn,cn FHOLD Start Transaction Read Update Commit Hardware: i7 server & i7 laptop client. The Dyalog File Server 14
DCF DFS Differences DFS Access Matrix is always enforced; you can not log in as user 0 . (Optionally), all DFS functions that refer to user numbers report user names: FHIST, FRDAC, FSTAC, FRDCI The Dyalog File Server 15
DCF DFS Differences DFS FLIB reports exclusively tied files So will FLIB in v13.2 DFS FHIST does not record latest file tie From 13.2, FHIST will also not do this DFS can tie more than 253 files at once DFS file holds are not released on return to immediate execution Not a bug, it s a feechure The Dyalog File Server 16
Features of v2.0 Online Backup and Restore Detailed Performance Monitor Improved performance of long-running tasks FCOPY, FCHK, MCOPY, MMOVE, ... Faster FLIB Support for Native File Functions Including a number of extensions Bootstrap applications direct from DFS DFS v2.0 17
Online Backup and Restore 24x7x365 operations: Backup and Restore without taking DFS down Takes snapshot of all component files managed by DFS, at a single point in time Slight performance degradation during backup Files updated while backup is in process will grow due to requirement to have old&new data available Restored files must not be tied Use monitor to identify (& if necessary kill) tasks using a file DFS v2.0 18
V2.0 Speedups Faster FLIB: Configuration switch to rely on .DCF extension Delegate Processes to handle long- running tasks: FCHK, FCOPY MCOPY, MMOVE, MDELETE Queueing system rewritten DFS v2.0 19
Bootstrap Mechanism DFS server will support download of application manifests via HTTP Bootstrap loader can maintain APL applications without a network share DFS v2.0 20
Native File Functions Full set of Native File Functions supported by Dyalog APL Several extensions Metadata: NLIB, NATTRIBUTES, SPACE Manage folders with NMKDIR, NRMDIR Handle multiple files and/or folders with MCOPY, MDELETE, MMOVE DFS v2.0 21
Extended File/Folder Fns Function Name MCOPY MDELETE MMOVE NATTRIBUTES NLIB NMKDIR NRMDIR SPACE Description Copy multiple files or folders Delete multiple files Move (rename) multiple files Return and set file attributes Directory listing Create a directory (folder) Delete an empty directory Return the free space on a drive DFS v2.0 22
Native File Issues Lots of small file operations may perform poorly (no client-side cache) For component files, security is enhanced (access matrices have absolute authority) For native files, user impersonation is not possible All file access is peformed using DFS server account Use of native files via DFS is probably only recommended for some applications DFS v2.0 23
Performance Monitor For each file operation DFS will log: File and User/Session Ids, Time of Day, Number of open files at time of op Elapsed time and CPU consumption in main process and in delegate Length of file operation queue and time spent in it Number of bytes transmitted in & out Monitoring can be switched on & off from the web monitor DFS v2.0 24
Performance Analysis Web monitor provides on-line analysis of last 15 minutes of data Off-line analysis of historical data DFS v2.0 25
Monitor Screen Shots DFS v2.0 26
Monitor Screen Shots DFS v2.0 27
Monitor Screen Shots DFS v2.0 28
Monitor Screen Shots DFS v2.0 29
Monitor Screen Shots DFS v2.0 30
Monitor Screen Shots DFS v2.0 31
Monitor Screen Shots DFS v2.0 32
Monitor Screen Shots DFS v2.0 33
DFS Road Map Performance, Performance, Performance Client and Server-side caching Multiple file operations in a single operation Stored Procedures (server-side APL) Compression Transactions For example, ( FHOLD tn) could trigger the beginning of a transaction and ( FHOLD ) be a commit . Mirroring / Disaster Recovery Features DFS v2.0 34
Availability V1 released in October 2013: Launch customer in production for over a year, 2nd customer at work General Availability scheduled on V2.0 release in Q1/2014 Requires Dyalog APL v13.2 for backup feature Not bundled with APL sold separately as add-on to runtime licences Annual licence for 50-user server: 3,000/32bit, 4,500/64bit DFS v2.0 35
