![Page 1: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/1.jpg)
Getting data home
?
![Page 2: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/2.jpg)
Getting data home
?
FTP
![Page 3: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/3.jpg)
Getting data home
?
FTP tapes
![Page 4: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/4.jpg)
Getting data home
?
FTP: not securetapes: incompatible
![Page 5: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/5.jpg)
Getting data home
?
FTP: not securetapes: incompatible
distracting
![Page 6: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/6.jpg)
Getting data home
?
http://bl831.als.lbl.gov/~jamesh/beamline/moving_data.html
![Page 7: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/7.jpg)
Getting data home
?
http://bl831.als.lbl.gov/~jamesh/beamline/moving_data.html
http://bcsb-web.als.lbl.gov/beamline/Sector8/DataBackup.htm
![Page 8: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/8.jpg)
Getting data home
?
http://bl831.als.lbl.gov/~jamesh/beamline/moving_data.html
http://bcsb-web.als.lbl.gov/beamline/Sector8/DataBackup.htm
http://ucxray.berkeley.edu/~jamesh/elves/manual/tricks.html
![Page 9: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/9.jpg)
Can you login?
ssh user@home
![Page 10: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/10.jpg)
Universal directory-transfer
tar cf - . |
ssh user@home
“cd /dir ; tar xvf -”
![Page 11: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/11.jpg)
Universal directory-transfer
tar cf - . |
ssh user@home
“cd /dir ; tar xvf -”
![Page 12: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/12.jpg)
Universal directory-transfer
tar cf - . |
ssh user@home
“cd /dir ; tar xvf -”
Password:
Ctrl-z
bg
![Page 13: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/13.jpg)
Making it faster
tar cf - . |
compress |
ssh user@home
“cd /dir ; uncompress | tar xvf -”
![Page 14: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/14.jpg)
The easy way
http://bl831.als.lbl.gov/~jamesh/elves/scripts/sendhome
sendhome . user@home:/dir
![Page 15: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/15.jpg)
Real Time
http://bl831.als.lbl.gov/~jamesh/elves/scripts/sendhome
sendhome first_001.img user@home:/dir
![Page 16: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/16.jpg)
Through a firewall
tar cf - . |
ssh user@firewall
‘ssh fileserver
“cd /dir ; tar xvf -”’
![Page 17: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/17.jpg)
The other way
ssh user@home
cd /dir
ssh user@beamline
“cd /data ; tar cf - .” |
tar xvf –
![Page 18: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/18.jpg)
The other way, faster
ssh user@beamline
“cd /dir ; tar cf - . | compress” |
uncompress |
tar xvf –
![Page 19: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/19.jpg)
The easy other way
ssh user@beamline
“download yourname” |
tar xvf –
![Page 20: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/20.jpg)
What if it stops?
![Page 21: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/21.jpg)
What if it stops?
ssh user@beamline
“download yourname frame_046.img” |
tar xvf –
![Page 22: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/22.jpg)
What if it stops?
http://bl831.als.lbl.gov/~jamesh/elves/scripts/sendhome
sendhome frame_046.img user@home:/dir
![Page 23: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/23.jpg)
What if I’m confused?
![Page 24: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/24.jpg)
What if I’m confused?
4 steps
![Page 25: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/25.jpg)
1) Generate a filename list
find . -name ‘*.img’
>! here.log
![Page 26: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/26.jpg)
2) Generate a “done” list
ssh user@home
“cd /dir ; find .”
>! there.log
![Page 27: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/27.jpg)
3) Make the “todo” list
diff there.log here.log |
awk ‘/^>/{print $2}’
>! 2bsent.log
![Page 28: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/28.jpg)
4) Feed “still left” list to tar
cat 2bsent.log |
tar cTf - - |
ssh user@home
“cd /dir ; tar xvf -”
![Page 29: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/29.jpg)
What if I can’t login?
ssh user@home warning: Connecting to home failed: Connection Refused
![Page 30: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/30.jpg)
“SneakerNet”
![Page 31: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/31.jpg)
“SneakerNet”
Tape
Firewire
DVD
![Page 32: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/32.jpg)
FireWire
OS Format
Mac HFS
Windows NTFS
Linux FAT32
![Page 33: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/33.jpg)
Realtime Firewire
1)plug in
![Page 34: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/34.jpg)
Realtime Firewire
1)plug in
2)share it
![Page 35: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/35.jpg)
Realtime Firewire
1)plug in
2)share it
3)smbclient
![Page 36: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/36.jpg)
Realtime Firewire
tar cf - . |
smbclient
//laptop/share –U user%password" -N -Tx -
![Page 37: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/37.jpg)
Realtime Firewire
find . |
tar cTf - - |
smbclient
//laptop/share –U user%password" -N -Tx -
![Page 38: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/38.jpg)
The easy way
http://bl831.als.lbl.gov/~jamesh/beamline/backup_firewire.com.txt
backup_firewire.com .
![Page 39: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/39.jpg)
DVD
OS Format
Mac HFS
Windows NTFS
Linux UFS
![Page 40: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/40.jpg)
DVD
OS Format
Mac ISO
Windows ISO
Linux ISO
![Page 41: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/41.jpg)
HOWTO make a DVD
mkisofs -r -J .
>! image.iso
![Page 42: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/42.jpg)
HOWTO make a DVD
mkisofs -r -J .
>! image.iso
dvdrecord dev=0,0,0
–data image.iso
![Page 43: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/43.jpg)
The easy way
http://bl831.als.lbl.gov/~jamesh/beamline/backup_local.com.txt
backup_local.com .
![Page 44: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/44.jpg)
The easy way
http://bl831.als.lbl.gov/~jamesh/beamline/backup_local.com.txt
backup_local.com .
http://bl831.als.lbl.gov/~jamesh/beamline/autolabel.com.txt
autolabel.com
![Page 45: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/45.jpg)
![Page 46: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/46.jpg)
DVD data archive
![Page 47: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/47.jpg)
DVD data archive
![Page 48: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/48.jpg)
DVD data archive
![Page 49: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/49.jpg)
DVD data archive
![Page 50: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/50.jpg)
DVD data archive
![Page 51: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/51.jpg)
DVD data archive
![Page 52: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/52.jpg)
DVD data archive
![Page 53: Getting data home ?. FTP Getting data home ? FTP tapes](https://reader036.vdocument.in/reader036/viewer/2022062422/56649f225503460f94c3b7eb/html5/thumbnails/53.jpg)
George Meigs
Jane Tanamachi
Beamline 8.3.1 staff
Acknowledgments