diffsync a file synchronization program class project – eel6883 by adam radwan gargi chipalkatti
TRANSCRIPT
![Page 1: DIFFSYNC A file synchronization program CLASS PROJECT – EEL6883 BY ADAM RADWAN GARGI CHIPALKATTI](https://reader038.vdocument.in/reader038/viewer/2022103111/55162735550346b2068b489e/html5/thumbnails/1.jpg)
DIFFSYNCA file synchronization program
CLASS PROJECT – EEL6883
BY
ADAM RADWAN
GARGI CHIPALKATTI
![Page 2: DIFFSYNC A file synchronization program CLASS PROJECT – EEL6883 BY ADAM RADWAN GARGI CHIPALKATTI](https://reader038.vdocument.in/reader038/viewer/2022103111/55162735550346b2068b489e/html5/thumbnails/2.jpg)
FILE SYNCHRONIZATION
Process of making sure that two or more locations contain the same up-to-date files
One-way (Mirroring) or two-way file synchronization Useful for mobile workers, multiple computer users;
used for backup purposes Version control tools and File synchronization tools Various existing solutions, none of which were found
by OpenPublic Networks, LLC to be robust enough for production use in a distributed environment
![Page 3: DIFFSYNC A file synchronization program CLASS PROJECT – EEL6883 BY ADAM RADWAN GARGI CHIPALKATTI](https://reader038.vdocument.in/reader038/viewer/2022103111/55162735550346b2068b489e/html5/thumbnails/3.jpg)
GOALS
Develop a file synchronization program which is capable of working over a diverse range of geographical placement
Must be capable of running securely over common public network backbone, such as the Internet
Capable of two-way synchronization of files for changes to served web pages
Ability for a non-functional server to gracefully degrade while other servers detect this and recover appropriately to keep services running
![Page 4: DIFFSYNC A file synchronization program CLASS PROJECT – EEL6883 BY ADAM RADWAN GARGI CHIPALKATTI](https://reader038.vdocument.in/reader038/viewer/2022103111/55162735550346b2068b489e/html5/thumbnails/4.jpg)
RSYNC
Open Source software for Unix systems Mirroring takes place with only one transmission in
each direction Used to synchronize local directories or directories
via a remote shell such as RSH or SSH Two checksums: a weak “rolling“ 32-bit checksum
and a strong 128-bit MD4 checksum
![Page 5: DIFFSYNC A file synchronization program CLASS PROJECT – EEL6883 BY ADAM RADWAN GARGI CHIPALKATTI](https://reader038.vdocument.in/reader038/viewer/2022103111/55162735550346b2068b489e/html5/thumbnails/5.jpg)
UNISON
Runs on Unix-like operating systems (Linux, Mac OS X, and Solaris), and on Windows
Can synchronize files across platforms Detects 'conflicts' where a file has been modified on
both sources, and displays these to the user Uses the rsync algorithm Designed to be robust in the event of a program or
system crash or a communication failure
![Page 6: DIFFSYNC A file synchronization program CLASS PROJECT – EEL6883 BY ADAM RADWAN GARGI CHIPALKATTI](https://reader038.vdocument.in/reader038/viewer/2022103111/55162735550346b2068b489e/html5/thumbnails/6.jpg)
DIFFSYNC
Uses open source software – rsync, inotify Uses PHP scripting language Built on top of a MySQL 5.1 triple-master replicated
database system running in chrooted jail for security Custom SSH-based VPN for three independent point-
to-point TUN links between the servers Apache-based web server for verification running in
chrooted jail for security phpMyAdmin used for database management
![Page 7: DIFFSYNC A file synchronization program CLASS PROJECT – EEL6883 BY ADAM RADWAN GARGI CHIPALKATTI](https://reader038.vdocument.in/reader038/viewer/2022103111/55162735550346b2068b489e/html5/thumbnails/7.jpg)
DIFFSYNC FOR THREE SERVERS
Akimbo
Location: Orlando, FLIP: 72.189.195.130CPU: Pentium III 800 MhzStorage: 20GB, 128 MBOS: Gentoo Linux Philomena
Location: Ft. Lauderdale, FLIP: 65.111.166.49CPU: AMD 2000+Storage: 40GB, 256 MBOS: Gentoo Linux
Podice
Location: Fayetteville, NCIP: 216.117.160.235CPU: Pentium 4 2.8 GhzStorage: 80GB, 512 MBOS: Gentoo Linux
![Page 8: DIFFSYNC A file synchronization program CLASS PROJECT – EEL6883 BY ADAM RADWAN GARGI CHIPALKATTI](https://reader038.vdocument.in/reader038/viewer/2022103111/55162735550346b2068b489e/html5/thumbnails/8.jpg)
CREATION OF DIRECTORY & FILE~/TestArea $ mkdir 1 Array( [0] => Array ( [dir] => ./TestArea/ [event] => CREATE,ISDIR [file] => 1 ) [1] => Array ( [dir] => ./TestArea/ [event] => OPEN,ISDIR [file] => 1 ) [2] => Array ( [dir] => ./TestArea/ [event] => CLOSE_NOWRITE,CLOSE,ISDIR [file] => 1 ))
~/TestArea $ nano ./1/X Array( [0] => Array ( [dir] => ./TestArea/1/ [event] => CREATE [file] => X ) [1] => Array ( [dir] => ./TestArea/1/ [event] => OPEN [file] => X ) [2] => Array ( [dir] => ./TestArea/1/ [event] => MODIFY [file] => X ) [3] => Array ( [dir] => ./TestArea/1/ [event] => CLOSE_WRITE,CLOSE [file] => X ))
![Page 9: DIFFSYNC A file synchronization program CLASS PROJECT – EEL6883 BY ADAM RADWAN GARGI CHIPALKATTI](https://reader038.vdocument.in/reader038/viewer/2022103111/55162735550346b2068b489e/html5/thumbnails/9.jpg)
MOVE DIRECTORY & FILE
~/TestArea $ mv ./1/X ./1/Y Array( [0] => Array ( [dir] => ./TestArea/1/ [event] => MOVED_FROM [file] => X ) [1] => Array ( [dir] => ./TestArea/1/ [event] => MOVED_TO [file] => Y ) )
~/TestArea $ mv ./1 ./2 Array( [0] => Array ( [dir] => ./TestArea/ [event] => MOVED_FROM [file] => 1 ) [1] => Array ( [dir] => ./TestArea/ [event] => MOVED_TO [file] => 2 ) )
![Page 10: DIFFSYNC A file synchronization program CLASS PROJECT – EEL6883 BY ADAM RADWAN GARGI CHIPALKATTI](https://reader038.vdocument.in/reader038/viewer/2022103111/55162735550346b2068b489e/html5/thumbnails/10.jpg)
ATTRIB DIRECTORY & FILE
~/TestArea $ chmod 777 ./1 Array( [0] => Array ( [dir] => ./TestArea/ [event] => ATTRIB [file] => 1 ))
~/TestArea $ chmod 777 ./1/X Array( [0] => Array ( [dir] => ./TestArea/1/ [event] => ATTRIB [file] => X ))
![Page 11: DIFFSYNC A file synchronization program CLASS PROJECT – EEL6883 BY ADAM RADWAN GARGI CHIPALKATTI](https://reader038.vdocument.in/reader038/viewer/2022103111/55162735550346b2068b489e/html5/thumbnails/11.jpg)
DELETE DIRECTORY [4] => Array ( [dir] => ./TestArea/1/ [event] => CLOSE_NOWRITE,CLOSE,ISDIR [file] => ) [5] => Array ( [dir] => ./TestArea/ [event] => DELETE,ISDIR [file] => 1 ) [6] => Array ( [dir] => ./TestArea/1/ [event] => DELETE_SELF [file] => ) [7] => Array ( [dir] => ./TestArea/1/ [event] => IGNORED [file] => ))
~/TestArea $ rm –r ./1
Array( [0] => Array ( [dir] => ./TestArea/ [event] => OPEN,ISDIR [file] => 1 ) [1] => Array ( [dir] => ./TestArea/1/ [event] => OPEN,ISDIR [file] => ) [2] => Array ( [dir] => ./TestArea/1/ [event] => DELETE [file] => X ) [3] => Array ( [dir] => ./TestArea/ [event] => CLOSE_NOWRITE,CLOSE,ISDIR [file] => 1 )
![Page 12: DIFFSYNC A file synchronization program CLASS PROJECT – EEL6883 BY ADAM RADWAN GARGI CHIPALKATTI](https://reader038.vdocument.in/reader038/viewer/2022103111/55162735550346b2068b489e/html5/thumbnails/12.jpg)
DELETE FILE
~/TestArea $ rm ./1/X Array( [0] => Array ( [dir] => ./TestArea/1/ [event] => DELETE [file] => X ) )
![Page 13: DIFFSYNC A file synchronization program CLASS PROJECT – EEL6883 BY ADAM RADWAN GARGI CHIPALKATTI](https://reader038.vdocument.in/reader038/viewer/2022103111/55162735550346b2068b489e/html5/thumbnails/13.jpg)
EDIT FILE
~/TestArea $ nano ./1/X Array( [0] => Array ( [dir] => ./TestArea/1/ [event] => OPEN [file] => X ) [1] => Array ( [dir] => ./TestArea/1/ [event] => ACCESS [file] => X ) [2] => Array ( [dir] => ./TestArea/1/ [event] => CLOSE_NOWRITE,CLOSE [file] => X ))
Array( [0] => Array ( [dir] => ./TestArea/1/ [event] => MODIFY [file] => X ) [1] => Array ( [dir] => ./TestArea/1/ [event] => OPEN [file] => X ) [2] => Array ( [dir] => ./TestArea/1/ [event] => CLOSE_WRITE,CLOSE [file] => X ))
![Page 14: DIFFSYNC A file synchronization program CLASS PROJECT – EEL6883 BY ADAM RADWAN GARGI CHIPALKATTI](https://reader038.vdocument.in/reader038/viewer/2022103111/55162735550346b2068b489e/html5/thumbnails/14.jpg)
COPY FILE~/TestArea $ cp ./1/X ./1/Y Array( [0] => Array ( [dir] => ./TestArea/1/ [event] => OPEN [file] => X ) [1] => Array ( [dir] => ./TestArea/1/ [event] => CREATE [file] => Y ) [2] => Array ( [dir] => ./TestArea/1/ [event] => OPEN [file] => Y ))
Array( [3] => Array ( [dir] => ./TestArea/1/ [event] => ACCESS [file] => X ) [4] => Array ( [dir] => ./TestArea/1/ [event] => MODIFY [file] => Y ) [5] => Array ( [dir] => ./TestArea/1/ [event] => CLOSE_WRITE,CLOSE [file] => Y ) [6] => Array ( [dir] => ./TestArea/1/ [event] => CLOSE_NOWRITE,CLOSE [file] => X ))
![Page 15: DIFFSYNC A file synchronization program CLASS PROJECT – EEL6883 BY ADAM RADWAN GARGI CHIPALKATTI](https://reader038.vdocument.in/reader038/viewer/2022103111/55162735550346b2068b489e/html5/thumbnails/15.jpg)
DIFFICULTIES ENCOUNTERED
Must have a kernel module installed on the host server for inotify events to be recorded
Problems detecting changes made to newly-created directories within the specified inotify watch point
MySQL replication is sensitive to binary log changes when one node is disconnected
Accurate prediction and parsing of inotify events for complete analysis of changes made to filesystem
Not enough time in the semester to complete integration of all features into program
![Page 16: DIFFSYNC A file synchronization program CLASS PROJECT – EEL6883 BY ADAM RADWAN GARGI CHIPALKATTI](https://reader038.vdocument.in/reader038/viewer/2022103111/55162735550346b2068b489e/html5/thumbnails/16.jpg)
CONCLUSION
Fundamentals of DiffSync program prove that file synchronization is possible for servers in remote locations
Additional work will be completed to make this program ready for a production environment
Any questions?
![Page 17: DIFFSYNC A file synchronization program CLASS PROJECT – EEL6883 BY ADAM RADWAN GARGI CHIPALKATTI](https://reader038.vdocument.in/reader038/viewer/2022103111/55162735550346b2068b489e/html5/thumbnails/17.jpg)
THE END
THANK YOU!