richard smith component files and the dfs. topics 1. new features in v14.0 2. dyalog file server...
TRANSCRIPT
![Page 1: Richard Smith Component Files and the DFS. Topics 1. New features in V14.0 2. Dyalog File Server (DFS) V2.0 3. Future possibilities](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e5c5503460f94b54a97/html5/thumbnails/1.jpg)
Richard Smith
Component Files and the DFS
![Page 2: Richard Smith Component Files and the DFS. Topics 1. New features in V14.0 2. Dyalog File Server (DFS) V2.0 3. Future possibilities](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e5c5503460f94b54a97/html5/thumbnails/2.jpg)
Topics
1. New features in V14.02. Dyalog File Server (DFS) V2.03. Future possibilities
![Page 3: Richard Smith Component Files and the DFS. Topics 1. New features in V14.0 2. Dyalog File Server (DFS) V2.0 3. Future possibilities](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e5c5503460f94b54a97/html5/thumbnails/3.jpg)
Section 1New in V14.0
Slide 3
![Page 4: Richard Smith Component Files and the DFS. Topics 1. New features in V14.0 2. Dyalog File Server (DFS) V2.0 3. Future possibilities](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e5c5503460f94b54a97/html5/thumbnails/4.jpg)
Small Span files
• No new small span files
![Page 5: Richard Smith Component Files and the DFS. Topics 1. New features in V14.0 2. Dyalog File Server (DFS) V2.0 3. Future possibilities](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e5c5503460f94b54a97/html5/thumbnails/5.jpg)
V14.0 variant options
• ⎕FCREATE– fid (⎕FCREATE ⍠ 'J' 3) 1– J, C and Z properties may be
specified– S and U will be added in 14.1
• ⎕FCHK– (⎕FCHK ⍠ 'Repair' 1) fid
![Page 6: Richard Smith Component Files and the DFS. Topics 1. New features in V14.0 2. Dyalog File Server (DFS) V2.0 3. Future possibilities](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e5c5503460f94b54a97/html5/thumbnails/6.jpg)
V14.0 performance I
• Read/write re-implemented• Faster, with no application
changes• Best performance gains with:
– Fast networks– Congested networks– Arrays with many elements of mixed
type
![Page 7: Richard Smith Component Files and the DFS. Topics 1. New features in V14.0 2. Dyalog File Server (DFS) V2.0 3. Future possibilities](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e5c5503460f94b54a97/html5/thumbnails/7.jpg)
V14.0 performance II
• New syntax multiple component read– ⎕FREAD 1 (⍳10) ≡ ⎕FREAD¨1,¨⍳10
• File is locked only once– Semantic difference; performance
gain• Faster but requires application
changes• Benefits share tied files
![Page 8: Richard Smith Component Files and the DFS. Topics 1. New features in V14.0 2. Dyalog File Server (DFS) V2.0 3. Future possibilities](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e5c5503460f94b54a97/html5/thumbnails/8.jpg)
V14.0 performance III
• Compressed components– 'Z' 1 ⎕FPROPS 1– fid (⎕FCREATE ⍠ 'Z' 1) 1
• Compression on subsequent writes
• Automatic decompression– V14.0 onwards, only
• Performance gain on slow networks
![Page 9: Richard Smith Component Files and the DFS. Topics 1. New features in V14.0 2. Dyalog File Server (DFS) V2.0 3. Future possibilities](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e5c5503460f94b54a97/html5/thumbnails/9.jpg)
Section 2Dyalog File Server (DFS)
V2.0
Slide 9
![Page 10: Richard Smith Component Files and the DFS. Topics 1. New features in V14.0 2. Dyalog File Server (DFS) V2.0 3. Future possibilities](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e5c5503460f94b54a97/html5/thumbnails/10.jpg)
~DFS
File server
File1File2
File3File4
APL Client 1
APL Client 5
APL Client n
APL Client 4
APL Client 2
APL Client 3
![Page 11: Richard Smith Component Files and the DFS. Topics 1. New features in V14.0 2. Dyalog File Server (DFS) V2.0 3. Future possibilities](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e5c5503460f94b54a97/html5/thumbnails/11.jpg)
DFS
DFS
File1File2
File3File4
APL Client 1
APL Client 5
APL Client n
APL Client 4
APL Client 2
APL Client 3
![Page 12: Richard Smith Component Files and the DFS. Topics 1. New features in V14.0 2. Dyalog File Server (DFS) V2.0 3. Future possibilities](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e5c5503460f94b54a97/html5/thumbnails/12.jpg)
DFS overview
• Supports component and native files
• Client/server (local/remote/distributed)
• Written in APL– Underlying storage mechanism is
DCF– Conga, MiServer, Interpreter support– Client presents a namespace
containing DCF-compatible functions and extensions
![Page 13: Richard Smith Component Files and the DFS. Topics 1. New features in V14.0 2. Dyalog File Server (DFS) V2.0 3. Future possibilities](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e5c5503460f94b54a97/html5/thumbnails/13.jpg)
Slide 13
File Server 2/test
File Server 1/prod
File1
File3
APL Client Session 1
APL Client Session 2
APL Client Session n
File Handler Thread 1
Master ServerManages Login requests
HTTP Monitor (Browser)
File2File HandlerThread 2
MonitorProvides monitoring and instrumentationExposes HTTP Interface
Locks ServerHandles all FHOLD and NLOCK requests
Backup ServerTakes backups
File Server 2 Delegate 1Performs long-running tasks like FCHK or FCOPY on behalf of a server
![Page 14: Richard Smith Component Files and the DFS. Topics 1. New features in V14.0 2. Dyalog File Server (DFS) V2.0 3. Future possibilities](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e5c5503460f94b54a97/html5/thumbnails/14.jpg)
DFS advantages
• Security• Performance• Resilience• Additional functions• Monitoring and admin• Backup and restore• APL thread support
![Page 15: Richard Smith Component Files and the DFS. Topics 1. New features in V14.0 2. Dyalog File Server (DFS) V2.0 3. Future possibilities](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e5c5503460f94b54a97/html5/thumbnails/15.jpg)
DFS security
• No access to files except via DFS– No loading into editors and viewing
contents• User authentication required• No default User 0
– Access matrix honoured
![Page 16: Richard Smith Component Files and the DFS. Topics 1. New features in V14.0 2. Dyalog File Server (DFS) V2.0 3. Future possibilities](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e5c5503460f94b54a97/html5/thumbnails/16.jpg)
DFS performance
• Exclusive ties– Locking and caching benefit
• FHOLD is faster; scales better• Delegates• Overall better than DCF on
Win/LAN
![Page 17: Richard Smith Component Files and the DFS. Topics 1. New features in V14.0 2. Dyalog File Server (DFS) V2.0 3. Future possibilities](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e5c5503460f94b54a97/html5/thumbnails/17.jpg)
DFS resilience
• Files are managed locally by “always-up” servers– Less sensitive to network glitches– Less sensitive to client stability– Clients can reconnect and resume
• Journaling can cure problems; DFS can prevent them
![Page 18: Richard Smith Component Files and the DFS. Topics 1. New features in V14.0 2. Dyalog File Server (DFS) V2.0 3. Future possibilities](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e5c5503460f94b54a97/html5/thumbnails/18.jpg)
DFS additional functions
• SHAREFILE/AP compatibility– FHIST, FSTATUS, FUSERNO, FWHOIS– FRENAME share tied files– Usernames (configurable)– No “library number” concept
• MMOVE etc.• NMKDIR etc.
[1]
![Page 19: Richard Smith Component Files and the DFS. Topics 1. New features in V14.0 2. Dyalog File Server (DFS) V2.0 3. Future possibilities](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e5c5503460f94b54a97/html5/thumbnails/19.jpg)
DFS monitoring and admin
• Start and stop system• Monitor usage and performance
– By file, user, operation etc.• Edit configuration• Manage clients
– Users and their permissions– Kill connection
![Page 20: Richard Smith Component Files and the DFS. Topics 1. New features in V14.0 2. Dyalog File Server (DFS) V2.0 3. Future possibilities](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e5c5503460f94b54a97/html5/thumbnails/20.jpg)
Slide 20
![Page 21: Richard Smith Component Files and the DFS. Topics 1. New features in V14.0 2. Dyalog File Server (DFS) V2.0 3. Future possibilities](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e5c5503460f94b54a97/html5/thumbnails/21.jpg)
Slide 21
![Page 22: Richard Smith Component Files and the DFS. Topics 1. New features in V14.0 2. Dyalog File Server (DFS) V2.0 3. Future possibilities](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e5c5503460f94b54a97/html5/thumbnails/22.jpg)
DFS backup and restore
• Hot backups (100% uptime)– Slight performance/space hit during
backup and on first subsequent write• Full and/or incremental backups• Scheduled and/or on demand• Optional compression• Email notification
![Page 23: Richard Smith Component Files and the DFS. Topics 1. New features in V14.0 2. Dyalog File Server (DFS) V2.0 3. Future possibilities](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e5c5503460f94b54a97/html5/thumbnails/23.jpg)
DFS thread support
• FHOLD is thread safe• Multiple DFS client connections
– Different login credentials allowed– Each independent set of tie numbers
etc.
![Page 24: Richard Smith Component Files and the DFS. Topics 1. New features in V14.0 2. Dyalog File Server (DFS) V2.0 3. Future possibilities](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e5c5503460f94b54a97/html5/thumbnails/24.jpg)
DFS and DCF compatibility
• DFS requires login• Q.FTIE vs ⎕FTIE etc.• DFS extensions• No file conversion
![Page 25: Richard Smith Component Files and the DFS. Topics 1. New features in V14.0 2. Dyalog File Server (DFS) V2.0 3. Future possibilities](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e5c5503460f94b54a97/html5/thumbnails/25.jpg)
DFS availibility
• V2.0 release imminent– V1.0 (Oct 2012) mainframe migration
• Separately licensed
![Page 26: Richard Smith Component Files and the DFS. Topics 1. New features in V14.0 2. Dyalog File Server (DFS) V2.0 3. Future possibilities](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e5c5503460f94b54a97/html5/thumbnails/26.jpg)
Section 3Future possibilities
Slide 26
![Page 27: Richard Smith Component Files and the DFS. Topics 1. New features in V14.0 2. Dyalog File Server (DFS) V2.0 3. Future possibilities](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e5c5503460f94b54a97/html5/thumbnails/27.jpg)
DCF/DFS future possibilities
• Performance improvements– Multi-component ⎕FREPLACE, ⎕FAPPEND, ⎕FRDCI
– Asynchronous read/write• Encryption• Transactions
![Page 28: Richard Smith Component Files and the DFS. Topics 1. New features in V14.0 2. Dyalog File Server (DFS) V2.0 3. Future possibilities](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e5c5503460f94b54a97/html5/thumbnails/28.jpg)
DCF Future possibilities
• Internal code refactoring required– Getting difficult to maintain and
extend– 13.2 locking problem– Retention of small-span support
• Long-term experiment to implement a component file library– Alternative to the interpreter code
![Page 29: Richard Smith Component Files and the DFS. Topics 1. New features in V14.0 2. Dyalog File Server (DFS) V2.0 3. Future possibilities](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e5c5503460f94b54a97/html5/thumbnails/29.jpg)
Component file library
• Designed to:– Include greater internal validation– Be easier to maintain and extend– Overcome some current feature
“annoyances”– Be usable from other tools and
utilities
(Demonstration)
![Page 30: Richard Smith Component Files and the DFS. Topics 1. New features in V14.0 2. Dyalog File Server (DFS) V2.0 3. Future possibilities](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e5c5503460f94b54a97/html5/thumbnails/30.jpg)
Component file library
• Risks to completion:– Implementation effort– Does not perform adequately
• Currently reads 0-5% slower– Behavioural differences (esp. on error)– Existing code tried and tested
• Lots of additional QA
• Will not be released “hastily”
![Page 31: Richard Smith Component Files and the DFS. Topics 1. New features in V14.0 2. Dyalog File Server (DFS) V2.0 3. Future possibilities](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e5c5503460f94b54a97/html5/thumbnails/31.jpg)
DFS future possibilities
• Stored procedures• Component caching• Encrypted / compressed comms• Multiple DFS installations
(mirroring)• Distributed server
– Scalability– Load balancing
![Page 32: Richard Smith Component Files and the DFS. Topics 1. New features in V14.0 2. Dyalog File Server (DFS) V2.0 3. Future possibilities](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e5c5503460f94b54a97/html5/thumbnails/32.jpg)
Summary
• 14.0 performance improvements• Dyalog File Server release is
imminent– Ask if you want it sooner
• Further enhancements are coming