SlideShare une entreprise Scribd logo
1  sur  54
Télécharger pour lire hors ligne
Rsync on HP-UX
Brief Presentation By Unix/Linux Apprentice
with 26 Years of Experience

Dusan Baljevic
Sydney, Australia
2011
Why This Document?
•

As a classical electronics/telecommunications engineer, I
believe in proper planning process – measure three times
before cutting:
http://www.circlingcycle.com.au/dusan.html

•

It is based on my 26-year practical experiences in
Unix/Linux.

•

My way of Operations Acceptance Testing and other tools:
http://www.circlingcycle.com.au/Unix-sources/
August 18, 2011

Rsync on HP-UX Webinar

2
Rsync History
•

rsync is an open source utility that provides fast
incremental file transfer.

•

rsync is freely available under the GNU General Public
License.

•

It has been available for more than 10 years now.

•

Available for Unix, Linux, and Windows.

•

Official site: http://rsync.samba.org/
August 18, 2011

Rsync on HP-UX Webinar

3
Rsync Features…
•

File based.

•

Incremental writes.

•

File delta.

•

Full restore.

•

Can use RSH, SSH or direct sockets as the transport.

•

Transmission security via SSH.

•

Internal pipelining reduces latency for multiple files.

•

File security via Encrypted File System (EvFS).

•

Cannot handle open-files (skips transferring them).

•

Cannot handle raw volumes.

•

Does not detect file renames and conflicts.

•

Scheduling done through O/S tools (cron, at, batch).
August 18, 2011

Rsync on HP-UX Webinar

4
Rsync Alternatives and Add-ons…
•

There are over 90 projects that deal with rsync in some
way:
http://unix.freshmeat.net/search/?Go.x=1&Go.y=1&q=rsync&section=projects

•

Other tools for file/directory synchronization:
http://en.wikipedia.org/wiki/Comparison_of_file_synchronization_software

August 18, 2011

Rsync on HP-UX Webinar

5
Rsync Algorithm
•

The rsync utility uses an algorithm (invented by Australian computer
programmer Andrew Tridgell) for efficiently transmitting a structure (such
as a file) across a communications link when the receiving computer
already has a different version of the same structure.

•

The recipient splits its copy of the file into fixed-size non-overlapping
chunks, of size S, and computes two checksums for each chunk: the
MD4 hash, and a weaker “rolling checksum”. It sends these checksums
to the sender.

•

The sender then compares its rolling checksums with the set sent by the
recipient to determine if any matches exist. If they do, it verifies the match
by computing the MD4/MD5* checksum for the matching block and by
comparing it with the MD4/MD5 checksum sent by the recipient.

•

The sender then sends the recipient those parts of its file that didn't
match any of the recipient's blocks, along with assembly instructions on
how to merge these blocks into the recipient's version to create a file
identical to the sender's copy.

•

If the sender's and recipient's versions of the file have many sections in
common, the utility needs to transfer relatively little data to synchronize
the files.
August 18, 2011

Rsync on HP-UX Webinar

6
Current Rsync Checksum Weaknesses – Part 1*
•

As mentioned on the previous slide, the recipient splits its copy of the file
into fixed-size non-overlapping chunks and computes two checksums for
each chunk: the MD4 hash, and a weaker 'rolling checksum' (version 30
of the protocol, released with rsync version 3.0.0, now uses MD5 hashes
rather than MD4). It sends these checksums to the sender.

•

The sender computes the rolling checksum for every chunk of size S in
its own version of the file, even overlapping chunks. This can be
calculated efficiently because of a special property of the rolling
checksum: if the rolling checksum of bytes n through n + S − 1 is R, the
rolling checksum of bytes n + 1 through n + S can be computed from R,
byte n, and byte n + S without having to examine the intervening bytes.
Thus, if one had already calculated the rolling checksum of bytes 1–25,
one could calculate the rolling checksum of bytes 2–26 solely from the
previous checksum, and from bytes 1 and 26.

•

The rolling checksum used in rsync is based on Mark Adler's adler-32
checksum, which is used in zlib, and is itself based on Fletcher's
checksum.
August 18, 2011

Rsync on HP-UX Webinar

7
Current Rsync Checksum Weaknesses – Part
2
•

The sender then compares its rolling checksums with the set sent by the
recipient to determine if any matches exist. If they do, it verifies the match
by computing the hash for the matching block and by comparing it with
the hash for that block sent by the recipient.

•

The sender then sends the recipient those parts of its file that did not
match the recipient's blocks, along with information on where to merge
these blocks into the recipient's version. This makes the copies identical.
However, there is a small probability that differences between chunks in
the sender and recipient are not detected, and thus remains uncorrected.
This requires a simultaneous hash collision in MD5 and the rolling
checksum. It is possible to generate MD5 collisions, and the rolling
checksum is not cryptographically strong, but the chance for this to occur
by accident is nevertheless extremely remote. With 128 bits from MD5
plus 32 bits from the rolling checksum, and assuming maximum entropy
in these bits, the probability of a hash collision with this combined
checksum is 2exp(−(128+32)) = 2exp(−160). The actual probability is a
few times higher, since good checksums approach maximum output
entropy but very rarely achieve it.
August 18, 2011

Rsync on HP-UX Webinar

8
Current Rsync Checksum Weaknesses – Part
3
•

If the sender's and recipient's versions of the file have many
sections in common, the utility needs to transfer relatively
little data to synchronize the files.

•

The MD5 Message-Digest Algorithm is a widely used
cryptographic hash function that produces a 128-bit (16-byte)
hash value. Specified in RFC 1321, MD5 has been employed
in a wide variety of security applications, and is also
commonly used to check data integrity. However, it has been
shown that MD5 is not collision resistant;[3] as such, MD5 is
not suitable for applications like SSL certificates or digital
signatures that rely on this property. An MD5 hash is typically
expressed as a 32-digit hexadecimal number.

August 18, 2011

Rsync on HP-UX Webinar

9
Current Rsync Checksum Weaknesses – Part
4
•

MD5 was designed by Ron Rivest in 1991 to replace an earlier hash
function, MD4. In 1996, a flaw was found with the design of MD5. While it
was not a clearly fatal weakness, cryptographers began recommending
the use of other algorithms, such as SHA-1 (which has since been found
also to be vulnerable). In 2004, more serious flaws were discovered,
making further use of the algorithm for security purposes questionable;
specifically, a group of researchers described how to create a pair of files
that share the same MD5 checksum. Further advances were made in
breaking MD5 in 2005, 2006, and 2007. In an attack on MD5 published in
December 2008, a group of researchers used this technique to fake SSL
certificate validity.

•
•

US-CERT says MD5 "should be considered cryptographically broken and
unsuitable for further use“.

•

Most U.S. government applications now require the SHA-2 family of hash
functions.
August 18, 2011

Rsync on HP-UX Webinar

10
Rsync – Memory Utilization
•

At one time, we used rsync version 2.x at a large private
hospital group in Australia (53 hospitals). It had one major
issue when we used it extensively: huge directory trees
were taking lot of RAM.

•

However, if you use rsync version 3.x, it does not keep the
whole tree in memory if it is big – it uses an incrementalrecursion algorithm.

August 18, 2011

Rsync on HP-UX Webinar

11
Rsync on HP-UX
•

For HP-UX 11iv2 and 11iv3, it is delivered on HP-UX Internet Express *
media, or can be obtained as free download at:
http://www.software.hp.com/portal/swdepot/displayProductInfo.do?productNumber=HPUXIEXP1131

•

For all currently supported HP-UX releases (11iv1 to 11iv3), there is
also The Porting and Archiving Centre for HP-UX:
http://hpux.connect.org.uk/hppd/hpux/Networking/Admin/rsync-3.0.8/

Make sure when installing depots from this site to also install ALL runtime dependencies (next slide).
•

Compile from sources.
August 18, 2011

Rsync on HP-UX Webinar

12
Rsync at The Porting and Archiving Centre
for HP-UX
Description: Rsync uses an algorithm which provides a very fast method for bringing
remote files into sync. It does this by sending just the differences in the files,
optionally
with compression, across the link, without requiring that both sets of files are present
at one of the ends of the link beforehand.

Author: Andrew Tridgell, Paul Mackerras tridge@samba.org
Home URL: http://rsync.samba.org/
License: GNU General Public License v3
Installation Tree: /usr/local
Languages used: C
Build-time dependencies: gettext libiconv make popt
Run-time dependencies: gettext libiconv popt
Documentation: Installation README Man Page

August 18, 2011

Rsync on HP-UX Webinar

13
Rsync Typical Scenario – Part 1
•

Day 1 – the first backup. No data at the remote location, so
a complete file transfer is required:
Local File

4GB

Remote File

Data transferred:
Rsync ~2GB (2:1 compression)
FTP/rcp/scp ~ 4GB

August 18, 2011

Rsync on HP-UX Webinar

14
Rsync Typical Scenario – Part 2
•

Day 2 – 0.5GB is added at the start of the file and the rest of
the file is left intact (red blocks of data) *:
Local File

4.5GB

Remote File

Data transferred:
Rsync ~0.25GB (2:1 compression)
FTP/rcp/scp ~ 4.5GB

August 18, 2011

Rsync on HP-UX Webinar

15
Rsync Typical Scenario – Part 3
•

Day 3 – The green (0.4GB) and yellow (0.2GB) blocks of
data are moved around (total of 0.6GB – the file contents are
the same, just block moved around):
Local File

4.5GB

Remote File

Data transferred:
Rsync ~0 (a small overhead)
FTP/rcp/scp ~ 4.5GB

August 18, 2011

Rsync on HP-UX Webinar

16
Rsync Command-line Options

# /usr/local/bin/rsync -v

August 18, 2011

Rsync on HP-UX Webinar

17
Rsync from Internet Express
It is important to pass the full path of the rsync command if it is not in the
PATH:

srvA# /opt/iexpress/rsync/bin/rsync -az -H -v --stats 
--rsync-path=/opt/iexpress/rsync/bin/rsync 
testfile.gz userB@srvB:/somedir

August 18, 2011

Rsync on HP-UX Webinar

18
Rsync from Porting and Archiving for HPUX
srvA# /usr/local/bin/rsync -a -r -v -t -z --stats 
--progress --rsh=/usr/bin/ssh 

--rsync-path=/usr/local/bin/rsync /labs srvB:

August 18, 2011

Rsync on HP-UX Webinar

19
Rsync Example when Previous Run
Interrupted or Failed
•

The flag “–partial” keeps the partially downloaded files on the
target.

•

This option is useful if the data transfer process gets interrupted by
some error of malfunction.

•

For example, if the rsync command terminated before all the data was
transported we could launch the same command again:
srvA# rsync -P myproject.gz userB@srvB:/somedir
rsync would use partial check sums to test the validity of that part of the
file that was already transported previously and start the actual data
transport only for the missing parts of the myproject.gz file.

August 18, 2011

Rsync on HP-UX Webinar

20
Rsync and SSH Key Exchange
userA@srvA# ssh-keygen -t rsa
Do not enter a passphrase, or use ssh-agent is passphrase is important.
userA@srvA# ssh userB@serverB mkdir -p .ssh
userA@srvA# cat .ssh/id_rsa.pub | ssh userB@srvB 
cat >> .ssh/authorized_keys
userA@srvA# ssh userB@srvB chmod 0700 .ssh/
userA@srvA# ssh userB@srvB chmod 0600 .ssh/authorized_keys
userA@srvA# rsync -a -r -v -t -z --stats --progress 
-e ssh /sourcedir/ userB@srvB:/targetdir/
August 18, 2011

Rsync on HP-UX Webinar

21
Rsync – SSH Chains and Different Port
•

rsync files between hosts that can not talk directly – use ssh

chain):
# rsync

-av --rsh="ssh -TA userA@srvA ssh -TA -l userB" 

/mydir/ srvB:/somedir/
•

rsync on different port and forcing IP protocol:

# rsync --progress --partial --rsh="ssh -p 8322" 

--bwlimit=300 --ipv4 remuser@remsrv:~/myfile.tgz .

August 18, 2011

Rsync on HP-UX Webinar

22
Rsync Example with Full Copy
srvA# /usr/local/bin/rsync -a -r -v -t -z --stats 

--progress --rsh=/usr/bin/ssh 
--rsync-path=/usr/local/bin/rsync /labs srvB:
...

Number of files: 118

Number of files transferred: 109
Total file size: 483583369 bytes
Total transferred file size: 483583369 bytes
Literal data: 483583369 bytes
Matched data: 0 bytes

File list size: 2002
File list generation time: 0.016 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 393465005
Total bytes received: 2119
August 18, 2011

Rsync on HP-UX Webinar

23
Rsync Example with Sparse Files * – Part 1
srvA# prealloc sparsefile1 1024000
srvA# /usr/local/bin/rsync -a -A -r -v -t -z 
--stats --progress 

--r sync-path=/usr/local/bin/rsync /src srvB:
srvB# ll sparsefile1
2000 -rw-r----sparsefile3

August 18, 2011

1 root sys

1024000 Aug 18 14:50

Rsync on HP-UX Webinar

24
Rsync Example with Sparse Files * – Part 2
srvA# prealloc sparsefile2 1024000
srvA# /usr/local/bin/rsync -a -A -S -r -v -t -z 
--stats --progress



--rsync-path=/usr/local/bin/rsync /src srvB:
sending incremental file list

src/sparsefile2
1024000 100%

24.88MB/s 0:00:00 (xfer#1, to-check=1/8)

Total transferred file size: 1024000 bytes
Literal data: 1024000 bytes
...
Total bytes sent: 1329
Total bytes received: 35
sent 1329 bytes

received 35 bytes

total size is 28825484
August 18, 2011

2728.00 bytes/sec

speedup is 21133.05
Rsync on HP-UX Webinar

25
Rsync Example with Sparse Files – Part 3
srvA# ll sparsefile*
2000 -rw-r-----

1 root

sys

1024000 Aug 18 14:45 sparsefile1

2000 -rw-r-----

1 root

sys

1024000 Aug 18 14:50 sparsefile2

srvB# ll sparsefile*
2000 -rw-r-----

1 root

sys

1024000 Aug 18 14:45 sparsefile1

0 -rw-r-----

1 root

sys

1024000 Aug 18 14:50 sparsefile2

August 18, 2011

Rsync on HP-UX Webinar

26
Rsync Example with ACLs * – Part 1
srvA# getacl vxdump
# file: vxdump
# owner: root

# group: sys
user::r-x
group::r-x
class:r-x
other:---

srvA# setacl -m u:dusan:rx vxdump
srvA# pwget -n dusan
dusan:*:111:1::/home/dusan:/usr/bin/sh

(User “dusan” has UID 111)

# getacl vxdump

# file: vxdump
# owner: root
# group: sys
user::r-x
user:dusan:r-x
group::r-x
class:r-x
other:--August 18, 2011

Rsync on HP-UX Webinar

27
Rsync Example with ACLs * – Part 2
srvA# /usr/local/bin/rsync -a -r -v -t -z --stats 
--progress --rsync-path=/usr/local/bin/rsync 

/src srvB:
Check the file on the target srvB (ACL is missing):
srvB# getacl /src/vxdump
# file: /src/vxdump
# owner: root
# group: sys
user::r-x
group::r-x
class:r-x
other:--August 18, 2011

Rsync on HP-UX Webinar

28
Rsync Example with ACLs * – Part 3
srvA# /usr/local/bin/rsync -a –A -r -v -t -z --stats 
--progress --rsync-path=/usr/local/bin/rsync 
/src srvB:
Check the file on the target srvB (note that the ACL is owned by user “111” because
“dusan” is obviously not in the password database, but the ACL is copied!).
srvB# getacl /src/vxdump

# file: /src/vxdump
# owner: root
# group: sys
user::r-x
user:111:r-x

group::r-x
class:r-x
other:--August 18, 2011

Rsync on HP-UX Webinar

29
Rsync with Dry Run
•

When doing something other than non-trivial copies or using
features of rsync that you have never used before, add the
“-n” or “--dry-run” switch to make it a dry run.

# rsync -avhn /mydir /otherdir/
# rsync -nbrvvhn --del /mydir /otherdir/
# rsync -rn --size-only --exclude=*.iso 
/mydir/ /otherdir/

August 18, 2011

Rsync on HP-UX Webinar

30
Rsync Anonymous Server from Porting and
Archiving for HP-UX – Part 1
•

Install the depot.

•

Add into /etc/group

nobody::-2:
•

Add into /etc/services
rsync

•

873/tcp

Add into /etc/inetd.conf
rsync stream tcp nowait root /usr/local/bin/rsync rsyncd –daemon

... And reload the daemon (inetd –c)
August 18, 2011

Rsync on HP-UX Webinar

31
Rsync Anonymous Server from Porting and
Archiving for HP-UX – Part 2
•

Create /etc/rsyncd.conf
uid = nobody
gid = nobody
use chroot = yes
read only=yes
max connections = 5
log file = /var/adm/syslog/rsyncd.log
[ftp] path = /src
comment = HP-UX Source export area

•

Check Bastille or disable it.

August 18, 2011

Rsync on HP-UX Webinar

32
Rsync Anonymous Server from Porting and
Archiving for HP-UX – Part 3
To check what is available on the rsync server:
# /usr/local/bin/rsync -avlH rsync://myhost
src

August 18, 2011

HP-UX Source export area

Rsync on HP-UX Webinar

33
Mirror By Rsync
If a remote server runs anonymous rsync server, mirroring is

achieved in the following manner:
# /usr/local/bin/rsync

-avvlH --rsync-path=/usr/local/bin/rsync 

myhost:/src/ /src
opening connection using: ssh myhost /usr/local/bin/rsync --server --sender vvlHogDtpre.isf . /src/
Password:
receiving incremental file list
created directory /src
delta-transmission enabled

rsync-3.0.8-ia64-11.31.depot
sparsefile2
sparsefile3
total: matches=0

hash_hits=0

•

sent 166 bytes

•

total size is 29849484
August 18, 2011

false_alarms=0 data=29849484

received 29853794 bytes

5427992.73 bytes/sec

speedup is 1.00
Rsync on HP-UX Webinar

34
Rsync and LVM Snapshots – Part 1
•

There is no official way to copy data from a snapshot back to the original
volume. Here is how I normally accomplish this task, although it
depends on what kind of files I am trying to copy, any special file
attributes, and the amount of data

•

If the amount of data is not massive, and there are not many files in the
file system), use rsync to re-synchronize data between a snapshot and
the original. Be very careful about the command line, because if you use
the wrong order of the volumes, the copy of the data goes in the wrong
direction!

# rsync -aHv /snap/ /orig
Rsync is known to be VERY slow for big file transfers and when there
are lot of files.
August 18, 2011

Rsync on HP-UX Webinar

35
Rsync and LVM Snapshots – Part 2
•

Since file systems are VxFS, vxdump(1M) and vxrestore(1M) are very
convenient. Much faster:
# fsck /snap
# vxdump 0f - /snap | (cd /orig && vxrestore xf -)

•

Use native HP-UX tools, like tar(1M), pax(1M), fbackup(1M), and
cpio(1M). Here are some simple examples:
# cd /snap && tar cpf - * | (cd /orig && tar xvpf -)
# cd /snap && fbackup -i . -f - | ( cd /orig && frecover -Xsrf - )
# cd /snap && pax -w . | ( cd /orig && pax -r -pe )

August 18, 2011

Rsync on HP-UX Webinar

36
Rsync and Expanded Remote File List
Pass the listing of remote command as list of files to rsync.
The following line will run the find command on the remote
machine in the /remdir directory and rsync all “*.conf"
files it finds to the local machine in the /somedir directory.
# rsync -avR remsrv:'`find /remdir -name "*.[conf]"`„ 
/somedir/

August 18, 2011

Rsync on HP-UX Webinar

37
Rsync and Relative Directory
•

If you want the target directory structure to be relative
(chroot in a way) you can add flag"-R".

•

The directory structure /mydir/data would then look like
/BACKUP/mydir/data/ as the sync path name starts
from / on the source machine:

# rsync -Ravx --timeout=30 --delete-excluded 
user@remsrv:/mydir/data/ /BACKUP/

August 18, 2011

Rsync on HP-UX Webinar

38
Rsync and “Snapshots”
•

Use “--link-dest” option to create space-efficient
snapshot-based backups. It appears to have multiple
complete copies of the backed up data (one for each
backup run) but files that do not change between runs are
hard linked instead of creating new copies, thus saving
space.

•

The major disadvantage of this technique is that if a file is
corrupted due to disk error it is just as corrupt in all
snapshots that link to that file. Having offline backups would
protect against this possibility.

August 18, 2011

Rsync on HP-UX Webinar

39
Rsync - File Size and Bandwidth Limits
If you need to update a site over a slow link, run two passes
of rsync.
•

Transfer the small files firstly:
# rsync -a --max-size=150K /mydir/ remsrv:/somedir/

•

Then do the same for the large files and limit bandwidth to
100 KBytes per second:
# rsync -a --min-size=150K --bwlimit=100 

/mydir/ remsrv:/somedir/
August 18, 2011

Rsync on HP-UX Webinar

40
Rsync – Exclude Files
An example how to exclude files and directories from rsync
transfer.

# rsync -azhve ssh --stats --progress 
--exclude-from '/somedir/EXCLUDE.txt' 
--delete-excluded /mydir remsrv:/somedir/

August 18, 2011

Rsync on HP-UX Webinar

41
Rsync – Avoid Checksum For Large Files
Before Transfer
•

It is a common error to use option “-c” when huge files are
transferred. rsync is going to have to read/checksum the
entire file, and reading them is going to take a long time,
unless the file is stored on SSDs or some very fast storage.

•

Instead, try:

# rsync -vhz --partial --inplace …
“-c” means that it checksums the entire file BEFORE doing
any transfers, rather than using the timestamp to see if it
has changed, which means reading the whole file twice.
August 18, 2011

Rsync on HP-UX Webinar

42
Rsync and “Clean Shell”
•

The "is your shell clean" message and the "protocol mismatch"
message are usually caused by having some program or command
running in Shell’s profiles (.cshrc, .profile, .kshrc, .bashrc or equivalent)
every time when using a remote-shell program (such as ssh or rsh).
Data written in this way corrupts the rsync data stream. rsync detects
this at startup and produces those error messages. However, if you are
using rsync-daemon syntax (host::path or rsync://) without
using a remote-shell program (no --rsh or -e option), there is not
remote-shell program involved, and the problem is probably caused by
an error on the daemon side (so check the daemon logs). To test it:
# ssh user@myhost.domain.dom /bin/true

The above command should not output anything at all (except an ssh
password prompt, if applicable). If there is any output on stdout, disable
whatever is generating that output so that rsync does not get “garbage”
trying to talk to the remote rsync.
August 18, 2011

Rsync on HP-UX Webinar

43
Rsync Common Problems
Several common causes for a remote rsync process going away:
•

The destination disk is full (at least the size of the largest file that
needs to be updated available must be in free disk space for the
transfer to succeed).

•

An idle connection caused a router or remote-shell server to
close the connection.

•

A network error caused the connection to be dropped.

•

The remote rsync executable was not found.

•

Remote Shell setup is not working right or is not "clean" (it is
sending spurious text to rsync).

•

If the problem might be an idle connection getting closed, use a
“--timeout” option (newer rsync versions send keep-alive
messages during periods of no activity).
August 18, 2011

Rsync on HP-UX Webinar

44
Rsync Performance Analysis and Tuning

There is no silver bullet!
Test everything and rsync will certainly bring
benefits.

August 18, 2011

Rsync on HP-UX Webinar

45
Thank You!

Dusan Baljevic
Sydney, Australia
2011
Appendix

Dusan Baljevic
Sydney, Australia
2011
Rsync Command-line Options – Part 1
Usage: rsync [OPTION]... SRC [SRC]... DEST
or rsync [OPTION]... SRC [SRC]... [USER@]HOST:DEST
or rsync [OPTION]... SRC [SRC]... [USER@]HOST::DEST
or rsync [OPTION]... SRC [SRC]... rsync://[USER@]HOST[:PORT]/DEST
or rsync [OPTION]... [USER@]HOST:SRC [DEST]

or rsync [OPTION]... [USER@]HOST::SRC [DEST]
or rsync [OPTION]... rsync://[USER@]HOST[:PORT]/SRC [DEST]
The ':' usages connect via remote shell, while '::' & 'rsync://' usages connect

to an rsync daemon, and require SRC or DEST to start with a module name.

August 18, 2011

Appendix- Rsync on HP-UX Webinar

48
Rsync Command-line Options – Part 2
Options
-v, --verbose

increase verbosity

-q, --quiet

suppress non-error messages

--no-motd

suppress daemon-mode MOTD (see manpage caveat)

-c, --checksum

skip based on checksum, not mod-time & size

-a, --archive

archive mode; equals -rlptgoD (no -H,-A,-X)

--no-OPTION

turn off an implied OPTION (e.g. --no-D)

-r, --recursive

recurse into directories

-R, --relative

use relative path names

--no-implied-dirs
-b, --backup

don't send implied dirs with --relative
make backups (see --suffix & --backup-dir)

--backup-dir=DIR

make backups into hierarchy based in DIR

--suffix=SUFFIX

set backup suffix (default ~ w/o --backup-dir)

-u, --update

skip files that are newer on the receiver

--inplace

update destination files in-place (SEE MAN PAGE)

--append

append data onto shorter files

--append-verify

like --append, but with old data in file checksum

August 18, 2011

Appendix- Rsync on HP-UX Webinar

49
Rsync Command-line Options – Part 3
Options
-d, --dirs

transfer directories without recursing

-l, --links

copy symlinks as symlinks

-L, --copy-links

transform symlink into referent file/dir

--copy-unsafe-links

only "unsafe" symlinks are transformed

--safe-links

ignore symlinks that point outside the source tree

-k, --copy-dirlinks

transform symlink to a dir into referent dir

-K, --keep-dirlinks

treat symlinked dir on receiver as dir

-H, --hard-links

preserve hard links

-p, --perms

preserve permissions

-E, --executability

preserve the file's executability

--chmod=CHMOD

affect file and/or directory permissions

-A, --acls

preserve ACLs (implies --perms)

-o, --owner

preserve owner (super-user only)

-g, --group

preserve group

--devices

preserve device files (super-user only)

--specials

preserve special files

August 18, 2011

Appendix- Rsync on HP-UX Webinar

50
Rsync Command-line Options – Part 4
Options
-D

same as --devices --specials

-t, --times

preserve modification times

-O, --omit-dir-times

omit directories from --times

--super

receiver attempts super-user activities

-S, --sparse

handle sparse files efficiently

-n, --dry-run

perform a trial run with no changes made

-W, --whole-file

copy files whole (without delta-xfer algorithm)

-x, --one-file-system

don't cross filesystem boundaries

-B, --block-size=SIZE

force a fixed checksum block-size

-e, --rsh=COMMAND

specify the remote shell to use

--rsync-path=PROGRAM

specify the rsync to run on the remote machine

--existing

skip creating new files on receiver

--ignore-existing

skip updating files that already exist on receiver

--remove-source-files

sender removes synchronized files (non-dirs)

--del

an alias for --delete-during

--delete

delete extraneous files from destination dirs

--delete-before

receiver deletes before transfer, not during

--delete-during

receiver deletes during transfer (default)

--delete-delay

find deletions during, delete after

--delete-after

receiver deletes after transfer, not during

--delete-excluded

also delete excluded files from destination dirs

--ignore-errors

delete even if there are I/O errors

--force

force deletion of directories even if not empty

--max-delete=NUM

don't delete more than NUM files

--max-size=SIZE

don't transfer any file larger than SIZE

--min-size=SIZE

don't transfer any file smaller than SIZE

--partial

keep partially transferred files

--partial-dir=DIR

put a partially transferred file into DIR

--delay-updates

put all updated files into place at transfer's end

August 18, 2011

Appendix- Rsync on HP-UX Webinar

51
Rsync Command-line Options – Part 5
Options
-m, --prune-empty-dirs

prune empty directory chains from the file-list

--numeric-ids

don't map uid/gid values by user/group name

--timeout=SECONDS

set I/O timeout in seconds

--contimeout=SECONDS

set daemon connection timeout in seconds

-I, --ignore-times

don't skip files that match in size and mod-time

--size-only

skip files that match in size

--modify-window=NUM

compare mod-times with reduced accuracy

-T, --temp-dir=DIR

create temporary files in directory DIR

-y, --fuzzy

find similar file for basis if no dest file

--compare-dest=DIR

also compare destination files relative to DIR

--copy-dest=DIR

... and include copies of unchanged files

--link-dest=DIR

hardlink to files in DIR when unchanged

-z, --compress

compress file data during the transfer

--compress-level=NUM

explicitly set compression level

--skip-compress=LIST

skip compressing files with a suffix in LIST

-C, --cvs-exclude

August 18, 2011

auto-ignore files the same way CVS does

Appendix- Rsync on HP-UX Webinar

52
Rsync Command-line Options – Part 6
Options
-f, --filter=RULE

add a file-filtering RULE

-F

same as --filter='dir-merge /.rsync-filter'
repeated: --filter='- .rsync-filter'
--exclude=PATTERN

exclude files matching PATTERN

--exclude-from=FILE

read exclude patterns from FILE

--include=PATTERN

don't exclude files matching PATTERN

--include-from=FILE

read include patterns from FILE

--files-from=FILE

read list of source-file names from FILE

-0, --from0

all *-from/filter files are delimited by 0s

-s, --protect-args

no space-splitting; only wildcard special-chars

--address=ADDRESS

bind address for outgoing socket to daemon

--port=PORT

specify double-colon alternate port number

--sockopts=OPTIONS

specify custom TCP options

--blocking-io

use blocking I/O for the remote shell

--stats

give some file-transfer stats

-8, --8-bit-output

leave high-bit chars unescaped in output

-h, --human-readable

output numbers in a human-readable format

--progress
August 18, 2011

show progress during transfer
Appendix- Rsync on HP-UX Webinar

53
Rsync Command-line Options – Part 7
Options
-P

same as --partial --progress

-i, --itemize-changes

output a change-summary for all updates

--out-format=FORMAT

output updates using the specified FORMAT

--log-file=FILE

log what we're doing to the specified FILE

--log-file-format=FMT

log updates using the specified FMT

--password-file=FILE

read daemon-access password from FILE

--list-only

list the files instead of copying them

--bwlimit=KBPS

limit I/O bandwidth; KBytes per second

--write-batch=FILE

write a batched update to FILE

--only-write-batch=FILE like --write-batch but w/o updating destination
--read-batch=FILE

read a batched update from FILE

--protocol=NUM

force an older protocol version to be used

--iconv=CONVERT_SPEC

request charset conversion of filenames

-4, --ipv4

prefer IPv4

-6, --ipv6

prefer IPv6

--version
(-h) --help

August 18, 2011

print version number
show this help (-h works with no other options)

Appendix- Rsync on HP-UX Webinar

54

Contenu connexe

En vedette

HP-UX Dynamic Root Disk Boot Disk Cloning Benefits and Use Cases by Dusan Bal...
HP-UX Dynamic Root Disk Boot Disk Cloning Benefits and Use Cases by Dusan Bal...HP-UX Dynamic Root Disk Boot Disk Cloning Benefits and Use Cases by Dusan Bal...
HP-UX Dynamic Root Disk Boot Disk Cloning Benefits and Use Cases by Dusan Bal...Circling Cycle
 
StoreGrid : Restore & Recovery
StoreGrid : Restore & RecoveryStoreGrid : Restore & Recovery
StoreGrid : Restore & RecoveryRevolucion
 
FreeNAS backup solution
FreeNAS backup solutionFreeNAS backup solution
FreeNAS backup solutiona3
 
FreeNAS installation and setup for shared storage (1/2)
FreeNAS installation and setup for shared storage (1/2)FreeNAS installation and setup for shared storage (1/2)
FreeNAS installation and setup for shared storage (1/2)康志強 大人
 
Three CLI Methods to Find Console IP details on HP-UX by Dusan Baljevic
Three CLI Methods to Find Console IP details on HP-UX by Dusan BaljevicThree CLI Methods to Find Console IP details on HP-UX by Dusan Baljevic
Three CLI Methods to Find Console IP details on HP-UX by Dusan BaljevicCircling Cycle
 
HP-UX RBAC Audsys Setup by Dusan Baljevic
HP-UX RBAC Audsys Setup by Dusan BaljevicHP-UX RBAC Audsys Setup by Dusan Baljevic
HP-UX RBAC Audsys Setup by Dusan BaljevicCircling Cycle
 
Better Settings for /tmp Filesystem on HP-UX by Dusan Baljevic
Better Settings for /tmp Filesystem on HP-UX by Dusan BaljevicBetter Settings for /tmp Filesystem on HP-UX by Dusan Baljevic
Better Settings for /tmp Filesystem on HP-UX by Dusan BaljevicCircling Cycle
 
HP-UX 11iv3 How to Change Root Volume Group Name vg00 by Dusan Baljevic
HP-UX 11iv3 How to Change Root Volume Group Name vg00 by Dusan BaljevicHP-UX 11iv3 How to Change Root Volume Group Name vg00 by Dusan Baljevic
HP-UX 11iv3 How to Change Root Volume Group Name vg00 by Dusan BaljevicCircling Cycle
 
Hp ux-11iv3-multiple-clones-with-dynamic-root-disks-dusan-baljevic-mar2014
Hp ux-11iv3-multiple-clones-with-dynamic-root-disks-dusan-baljevic-mar2014Hp ux-11iv3-multiple-clones-with-dynamic-root-disks-dusan-baljevic-mar2014
Hp ux-11iv3-multiple-clones-with-dynamic-root-disks-dusan-baljevic-mar2014Circling Cycle
 
Sizing your alfresco platform
Sizing your alfresco platformSizing your alfresco platform
Sizing your alfresco platformLuis Cabaceira
 
HP-UX 11i Log File Management with Logrotate by Dusan Baljevic
HP-UX 11i Log File Management with Logrotate by Dusan BaljevicHP-UX 11i Log File Management with Logrotate by Dusan Baljevic
HP-UX 11i Log File Management with Logrotate by Dusan BaljevicCircling Cycle
 
HP-UX 11i LVM Mirroring Features and Multi-threads by Dusan Baljevic
HP-UX 11i LVM Mirroring Features and Multi-threads by Dusan BaljevicHP-UX 11i LVM Mirroring Features and Multi-threads by Dusan Baljevic
HP-UX 11i LVM Mirroring Features and Multi-threads by Dusan BaljevicCircling Cycle
 
HP-UX Swap and Dump Unleashed by Dusan Baljevic
HP-UX Swap and Dump Unleashed by Dusan BaljevicHP-UX Swap and Dump Unleashed by Dusan Baljevic
HP-UX Swap and Dump Unleashed by Dusan BaljevicCircling Cycle
 
HPUX Update Seminar Session 1 Dan Taipala
HPUX Update Seminar Session 1   Dan TaipalaHPUX Update Seminar Session 1   Dan Taipala
HPUX Update Seminar Session 1 Dan Taipaladtaipala
 
How to Remove Primary Swap on HP-UX 11iv3 Online by Dusan Baljevic
How to Remove Primary Swap on HP-UX 11iv3 Online by Dusan BaljevicHow to Remove Primary Swap on HP-UX 11iv3 Online by Dusan Baljevic
How to Remove Primary Swap on HP-UX 11iv3 Online by Dusan BaljevicCircling Cycle
 
HP-UX 11iv3 Ignite-UX with NFSv4 and SSH Tunnel by Dusan Baljevic
HP-UX 11iv3 Ignite-UX with NFSv4 and SSH Tunnel by Dusan BaljevicHP-UX 11iv3 Ignite-UX with NFSv4 and SSH Tunnel by Dusan Baljevic
HP-UX 11iv3 Ignite-UX with NFSv4 and SSH Tunnel by Dusan BaljevicCircling Cycle
 

En vedette (20)

HP-UX Dynamic Root Disk Boot Disk Cloning Benefits and Use Cases by Dusan Bal...
HP-UX Dynamic Root Disk Boot Disk Cloning Benefits and Use Cases by Dusan Bal...HP-UX Dynamic Root Disk Boot Disk Cloning Benefits and Use Cases by Dusan Bal...
HP-UX Dynamic Root Disk Boot Disk Cloning Benefits and Use Cases by Dusan Bal...
 
StoreGrid : Restore & Recovery
StoreGrid : Restore & RecoveryStoreGrid : Restore & Recovery
StoreGrid : Restore & Recovery
 
Elastix backup-guide
Elastix backup-guideElastix backup-guide
Elastix backup-guide
 
FreeNAS backup solution
FreeNAS backup solutionFreeNAS backup solution
FreeNAS backup solution
 
FreeNAS installation and setup for shared storage (1/2)
FreeNAS installation and setup for shared storage (1/2)FreeNAS installation and setup for shared storage (1/2)
FreeNAS installation and setup for shared storage (1/2)
 
Three CLI Methods to Find Console IP details on HP-UX by Dusan Baljevic
Three CLI Methods to Find Console IP details on HP-UX by Dusan BaljevicThree CLI Methods to Find Console IP details on HP-UX by Dusan Baljevic
Three CLI Methods to Find Console IP details on HP-UX by Dusan Baljevic
 
HP-UX RBAC Audsys Setup by Dusan Baljevic
HP-UX RBAC Audsys Setup by Dusan BaljevicHP-UX RBAC Audsys Setup by Dusan Baljevic
HP-UX RBAC Audsys Setup by Dusan Baljevic
 
Better Settings for /tmp Filesystem on HP-UX by Dusan Baljevic
Better Settings for /tmp Filesystem on HP-UX by Dusan BaljevicBetter Settings for /tmp Filesystem on HP-UX by Dusan Baljevic
Better Settings for /tmp Filesystem on HP-UX by Dusan Baljevic
 
HP-UX 11iv3 How to Change Root Volume Group Name vg00 by Dusan Baljevic
HP-UX 11iv3 How to Change Root Volume Group Name vg00 by Dusan BaljevicHP-UX 11iv3 How to Change Root Volume Group Name vg00 by Dusan Baljevic
HP-UX 11iv3 How to Change Root Volume Group Name vg00 by Dusan Baljevic
 
Hp ux-11iv3-multiple-clones-with-dynamic-root-disks-dusan-baljevic-mar2014
Hp ux-11iv3-multiple-clones-with-dynamic-root-disks-dusan-baljevic-mar2014Hp ux-11iv3-multiple-clones-with-dynamic-root-disks-dusan-baljevic-mar2014
Hp ux-11iv3-multiple-clones-with-dynamic-root-disks-dusan-baljevic-mar2014
 
Superdome
SuperdomeSuperdome
Superdome
 
Hp Integrity Servers
Hp Integrity ServersHp Integrity Servers
Hp Integrity Servers
 
Sizing your alfresco platform
Sizing your alfresco platformSizing your alfresco platform
Sizing your alfresco platform
 
HP-UX 11i Log File Management with Logrotate by Dusan Baljevic
HP-UX 11i Log File Management with Logrotate by Dusan BaljevicHP-UX 11i Log File Management with Logrotate by Dusan Baljevic
HP-UX 11i Log File Management with Logrotate by Dusan Baljevic
 
HP-UX 11i LVM Mirroring Features and Multi-threads by Dusan Baljevic
HP-UX 11i LVM Mirroring Features and Multi-threads by Dusan BaljevicHP-UX 11i LVM Mirroring Features and Multi-threads by Dusan Baljevic
HP-UX 11i LVM Mirroring Features and Multi-threads by Dusan Baljevic
 
UX at HP Enterprise
UX at HP Enterprise UX at HP Enterprise
UX at HP Enterprise
 
HP-UX Swap and Dump Unleashed by Dusan Baljevic
HP-UX Swap and Dump Unleashed by Dusan BaljevicHP-UX Swap and Dump Unleashed by Dusan Baljevic
HP-UX Swap and Dump Unleashed by Dusan Baljevic
 
HPUX Update Seminar Session 1 Dan Taipala
HPUX Update Seminar Session 1   Dan TaipalaHPUX Update Seminar Session 1   Dan Taipala
HPUX Update Seminar Session 1 Dan Taipala
 
How to Remove Primary Swap on HP-UX 11iv3 Online by Dusan Baljevic
How to Remove Primary Swap on HP-UX 11iv3 Online by Dusan BaljevicHow to Remove Primary Swap on HP-UX 11iv3 Online by Dusan Baljevic
How to Remove Primary Swap on HP-UX 11iv3 Online by Dusan Baljevic
 
HP-UX 11iv3 Ignite-UX with NFSv4 and SSH Tunnel by Dusan Baljevic
HP-UX 11iv3 Ignite-UX with NFSv4 and SSH Tunnel by Dusan BaljevicHP-UX 11iv3 Ignite-UX with NFSv4 and SSH Tunnel by Dusan Baljevic
HP-UX 11iv3 Ignite-UX with NFSv4 and SSH Tunnel by Dusan Baljevic
 

Similaire à HP-UX with Rsync by Dusan Baljevic

Teoria efectului defectului hardware: GoogleFS
Teoria efectului defectului hardware: GoogleFSTeoria efectului defectului hardware: GoogleFS
Teoria efectului defectului hardware: GoogleFSAsociatia ProLinux
 
Pacemaker+DRBD
Pacemaker+DRBDPacemaker+DRBD
Pacemaker+DRBDDan Frincu
 
The Google File System (GFS)
The Google File System (GFS)The Google File System (GFS)
The Google File System (GFS)Romain Jacotin
 
Analysis of the LAN Sync Protocol
Analysis of the LAN Sync ProtocolAnalysis of the LAN Sync Protocol
Analysis of the LAN Sync ProtocolSJSU
 
Realizing the promise of portable data processing with Apache Beam
Realizing the promise of portable data processing with Apache BeamRealizing the promise of portable data processing with Apache Beam
Realizing the promise of portable data processing with Apache BeamDataWorks Summit
 
Distributed operating system
Distributed operating systemDistributed operating system
Distributed operating systemMoeez Ahmad
 
Parallel Database description in database management
Parallel Database description in database managementParallel Database description in database management
Parallel Database description in database managementchandugoswami
 
ch22a_ParallelDBs how parallel Datab.ppt
ch22a_ParallelDBs how parallel Datab.pptch22a_ParallelDBs how parallel Datab.ppt
ch22a_ParallelDBs how parallel Datab.pptRahulBhole12
 
Load Balancing with HAproxy
Load Balancing with HAproxyLoad Balancing with HAproxy
Load Balancing with HAproxyBrendan Jennings
 
For your final step, you will synthesize the previous steps and la
For your final step, you will synthesize the previous steps and laFor your final step, you will synthesize the previous steps and la
For your final step, you will synthesize the previous steps and laShainaBoling829
 
Network simulator 2 a simulation tool for linux
Network simulator 2 a simulation tool for linuxNetwork simulator 2 a simulation tool for linux
Network simulator 2 a simulation tool for linuxPratik Joshi
 
Efficient File Sharing Scheme in Mobile Adhoc Network
Efficient File Sharing Scheme in Mobile Adhoc NetworkEfficient File Sharing Scheme in Mobile Adhoc Network
Efficient File Sharing Scheme in Mobile Adhoc Networkpaperpublications3
 
Laboratory 2PRACTICE EXAMINING TRAFFIC WITH A PROTOCOL ANALYZER.docx
Laboratory 2PRACTICE EXAMINING TRAFFIC WITH A PROTOCOL ANALYZER.docxLaboratory 2PRACTICE EXAMINING TRAFFIC WITH A PROTOCOL ANALYZER.docx
Laboratory 2PRACTICE EXAMINING TRAFFIC WITH A PROTOCOL ANALYZER.docxsleeperfindley
 
Secured Authorized Deduplication Based Hybrid Cloud
Secured Authorized Deduplication Based Hybrid CloudSecured Authorized Deduplication Based Hybrid Cloud
Secured Authorized Deduplication Based Hybrid Cloudtheijes
 
E031102034039
E031102034039E031102034039
E031102034039theijes
 
Driver Programming Report
Driver Programming ReportDriver Programming Report
Driver Programming ReportShivek Khurana
 
Performance improvement by
Performance improvement byPerformance improvement by
Performance improvement byIJCNCJournal
 

Similaire à HP-UX with Rsync by Dusan Baljevic (20)

Teoria efectului defectului hardware: GoogleFS
Teoria efectului defectului hardware: GoogleFSTeoria efectului defectului hardware: GoogleFS
Teoria efectului defectului hardware: GoogleFS
 
Pacemaker+DRBD
Pacemaker+DRBDPacemaker+DRBD
Pacemaker+DRBD
 
The Google File System (GFS)
The Google File System (GFS)The Google File System (GFS)
The Google File System (GFS)
 
Analysis of the LAN Sync Protocol
Analysis of the LAN Sync ProtocolAnalysis of the LAN Sync Protocol
Analysis of the LAN Sync Protocol
 
Realizing the promise of portable data processing with Apache Beam
Realizing the promise of portable data processing with Apache BeamRealizing the promise of portable data processing with Apache Beam
Realizing the promise of portable data processing with Apache Beam
 
Distributed operating system
Distributed operating systemDistributed operating system
Distributed operating system
 
Parallel Database description in database management
Parallel Database description in database managementParallel Database description in database management
Parallel Database description in database management
 
ch22a_ParallelDBs how parallel Datab.ppt
ch22a_ParallelDBs how parallel Datab.pptch22a_ParallelDBs how parallel Datab.ppt
ch22a_ParallelDBs how parallel Datab.ppt
 
4 026
4 0264 026
4 026
 
Load Balancing with HAproxy
Load Balancing with HAproxyLoad Balancing with HAproxy
Load Balancing with HAproxy
 
For your final step, you will synthesize the previous steps and la
For your final step, you will synthesize the previous steps and laFor your final step, you will synthesize the previous steps and la
For your final step, you will synthesize the previous steps and la
 
Network simulator 2 a simulation tool for linux
Network simulator 2 a simulation tool for linuxNetwork simulator 2 a simulation tool for linux
Network simulator 2 a simulation tool for linux
 
Debunking Common Myths in Stream Processing
Debunking Common Myths in Stream ProcessingDebunking Common Myths in Stream Processing
Debunking Common Myths in Stream Processing
 
Document 22.pdf
Document 22.pdfDocument 22.pdf
Document 22.pdf
 
Efficient File Sharing Scheme in Mobile Adhoc Network
Efficient File Sharing Scheme in Mobile Adhoc NetworkEfficient File Sharing Scheme in Mobile Adhoc Network
Efficient File Sharing Scheme in Mobile Adhoc Network
 
Laboratory 2PRACTICE EXAMINING TRAFFIC WITH A PROTOCOL ANALYZER.docx
Laboratory 2PRACTICE EXAMINING TRAFFIC WITH A PROTOCOL ANALYZER.docxLaboratory 2PRACTICE EXAMINING TRAFFIC WITH A PROTOCOL ANALYZER.docx
Laboratory 2PRACTICE EXAMINING TRAFFIC WITH A PROTOCOL ANALYZER.docx
 
Secured Authorized Deduplication Based Hybrid Cloud
Secured Authorized Deduplication Based Hybrid CloudSecured Authorized Deduplication Based Hybrid Cloud
Secured Authorized Deduplication Based Hybrid Cloud
 
E031102034039
E031102034039E031102034039
E031102034039
 
Driver Programming Report
Driver Programming ReportDriver Programming Report
Driver Programming Report
 
Performance improvement by
Performance improvement byPerformance improvement by
Performance improvement by
 

Plus de Circling Cycle

Brief summary-standard-password-hashes-Aix-FreeBSD-Linux-Solaris-HP-UX-May-20...
Brief summary-standard-password-hashes-Aix-FreeBSD-Linux-Solaris-HP-UX-May-20...Brief summary-standard-password-hashes-Aix-FreeBSD-Linux-Solaris-HP-UX-May-20...
Brief summary-standard-password-hashes-Aix-FreeBSD-Linux-Solaris-HP-UX-May-20...Circling Cycle
 
How to-mount-3 par-san-virtual-copy-onto-rhel-servers-by-Dusan-Baljevic
How to-mount-3 par-san-virtual-copy-onto-rhel-servers-by-Dusan-BaljevicHow to-mount-3 par-san-virtual-copy-onto-rhel-servers-by-Dusan-Baljevic
How to-mount-3 par-san-virtual-copy-onto-rhel-servers-by-Dusan-BaljevicCircling Cycle
 
Ovclusterinfo command by Dusan Baljevic
Ovclusterinfo command by Dusan BaljevicOvclusterinfo command by Dusan Baljevic
Ovclusterinfo command by Dusan BaljevicCircling Cycle
 
Unix and Linux Common Boot Disk Disaster Recovery Tools by Dusan Baljevic
Unix and Linux Common Boot Disk Disaster Recovery Tools by Dusan BaljevicUnix and Linux Common Boot Disk Disaster Recovery Tools by Dusan Baljevic
Unix and Linux Common Boot Disk Disaster Recovery Tools by Dusan BaljevicCircling Cycle
 
HP-UX 11iv3 Private Kernel Parameter nfile by Dusan Baljevic
HP-UX 11iv3 Private Kernel Parameter nfile by Dusan BaljevicHP-UX 11iv3 Private Kernel Parameter nfile by Dusan Baljevic
HP-UX 11iv3 Private Kernel Parameter nfile by Dusan BaljevicCircling Cycle
 
HP-UX Dynamic Root Disk vs Solaris Live Upgrade vs AIX Multibos by Dusan Balj...
HP-UX Dynamic Root Disk vs Solaris Live Upgrade vs AIX Multibos by Dusan Balj...HP-UX Dynamic Root Disk vs Solaris Live Upgrade vs AIX Multibos by Dusan Balj...
HP-UX Dynamic Root Disk vs Solaris Live Upgrade vs AIX Multibos by Dusan Balj...Circling Cycle
 
Comparison of Unix and Linux Log File Management Tools by Dusan Baljevic
Comparison of Unix and Linux Log File Management Tools by Dusan BaljevicComparison of Unix and Linux Log File Management Tools by Dusan Baljevic
Comparison of Unix and Linux Log File Management Tools by Dusan BaljevicCircling Cycle
 

Plus de Circling Cycle (7)

Brief summary-standard-password-hashes-Aix-FreeBSD-Linux-Solaris-HP-UX-May-20...
Brief summary-standard-password-hashes-Aix-FreeBSD-Linux-Solaris-HP-UX-May-20...Brief summary-standard-password-hashes-Aix-FreeBSD-Linux-Solaris-HP-UX-May-20...
Brief summary-standard-password-hashes-Aix-FreeBSD-Linux-Solaris-HP-UX-May-20...
 
How to-mount-3 par-san-virtual-copy-onto-rhel-servers-by-Dusan-Baljevic
How to-mount-3 par-san-virtual-copy-onto-rhel-servers-by-Dusan-BaljevicHow to-mount-3 par-san-virtual-copy-onto-rhel-servers-by-Dusan-Baljevic
How to-mount-3 par-san-virtual-copy-onto-rhel-servers-by-Dusan-Baljevic
 
Ovclusterinfo command by Dusan Baljevic
Ovclusterinfo command by Dusan BaljevicOvclusterinfo command by Dusan Baljevic
Ovclusterinfo command by Dusan Baljevic
 
Unix and Linux Common Boot Disk Disaster Recovery Tools by Dusan Baljevic
Unix and Linux Common Boot Disk Disaster Recovery Tools by Dusan BaljevicUnix and Linux Common Boot Disk Disaster Recovery Tools by Dusan Baljevic
Unix and Linux Common Boot Disk Disaster Recovery Tools by Dusan Baljevic
 
HP-UX 11iv3 Private Kernel Parameter nfile by Dusan Baljevic
HP-UX 11iv3 Private Kernel Parameter nfile by Dusan BaljevicHP-UX 11iv3 Private Kernel Parameter nfile by Dusan Baljevic
HP-UX 11iv3 Private Kernel Parameter nfile by Dusan Baljevic
 
HP-UX Dynamic Root Disk vs Solaris Live Upgrade vs AIX Multibos by Dusan Balj...
HP-UX Dynamic Root Disk vs Solaris Live Upgrade vs AIX Multibos by Dusan Balj...HP-UX Dynamic Root Disk vs Solaris Live Upgrade vs AIX Multibos by Dusan Balj...
HP-UX Dynamic Root Disk vs Solaris Live Upgrade vs AIX Multibos by Dusan Balj...
 
Comparison of Unix and Linux Log File Management Tools by Dusan Baljevic
Comparison of Unix and Linux Log File Management Tools by Dusan BaljevicComparison of Unix and Linux Log File Management Tools by Dusan Baljevic
Comparison of Unix and Linux Log File Management Tools by Dusan Baljevic
 

Dernier

Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfDaniel Santiago Silva Capera
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxUdaiappa Ramachandran
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?IES VE
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-pyJamie (Taka) Wang
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Brian Pichman
 

Dernier (20)

Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptx
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
20150722 - AGV
20150722 - AGV20150722 - AGV
20150722 - AGV
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-py
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )
 

HP-UX with Rsync by Dusan Baljevic

  • 1. Rsync on HP-UX Brief Presentation By Unix/Linux Apprentice with 26 Years of Experience Dusan Baljevic Sydney, Australia 2011
  • 2. Why This Document? • As a classical electronics/telecommunications engineer, I believe in proper planning process – measure three times before cutting: http://www.circlingcycle.com.au/dusan.html • It is based on my 26-year practical experiences in Unix/Linux. • My way of Operations Acceptance Testing and other tools: http://www.circlingcycle.com.au/Unix-sources/ August 18, 2011 Rsync on HP-UX Webinar 2
  • 3. Rsync History • rsync is an open source utility that provides fast incremental file transfer. • rsync is freely available under the GNU General Public License. • It has been available for more than 10 years now. • Available for Unix, Linux, and Windows. • Official site: http://rsync.samba.org/ August 18, 2011 Rsync on HP-UX Webinar 3
  • 4. Rsync Features… • File based. • Incremental writes. • File delta. • Full restore. • Can use RSH, SSH or direct sockets as the transport. • Transmission security via SSH. • Internal pipelining reduces latency for multiple files. • File security via Encrypted File System (EvFS). • Cannot handle open-files (skips transferring them). • Cannot handle raw volumes. • Does not detect file renames and conflicts. • Scheduling done through O/S tools (cron, at, batch). August 18, 2011 Rsync on HP-UX Webinar 4
  • 5. Rsync Alternatives and Add-ons… • There are over 90 projects that deal with rsync in some way: http://unix.freshmeat.net/search/?Go.x=1&Go.y=1&q=rsync&section=projects • Other tools for file/directory synchronization: http://en.wikipedia.org/wiki/Comparison_of_file_synchronization_software August 18, 2011 Rsync on HP-UX Webinar 5
  • 6. Rsync Algorithm • The rsync utility uses an algorithm (invented by Australian computer programmer Andrew Tridgell) for efficiently transmitting a structure (such as a file) across a communications link when the receiving computer already has a different version of the same structure. • The recipient splits its copy of the file into fixed-size non-overlapping chunks, of size S, and computes two checksums for each chunk: the MD4 hash, and a weaker “rolling checksum”. It sends these checksums to the sender. • The sender then compares its rolling checksums with the set sent by the recipient to determine if any matches exist. If they do, it verifies the match by computing the MD4/MD5* checksum for the matching block and by comparing it with the MD4/MD5 checksum sent by the recipient. • The sender then sends the recipient those parts of its file that didn't match any of the recipient's blocks, along with assembly instructions on how to merge these blocks into the recipient's version to create a file identical to the sender's copy. • If the sender's and recipient's versions of the file have many sections in common, the utility needs to transfer relatively little data to synchronize the files. August 18, 2011 Rsync on HP-UX Webinar 6
  • 7. Current Rsync Checksum Weaknesses – Part 1* • As mentioned on the previous slide, the recipient splits its copy of the file into fixed-size non-overlapping chunks and computes two checksums for each chunk: the MD4 hash, and a weaker 'rolling checksum' (version 30 of the protocol, released with rsync version 3.0.0, now uses MD5 hashes rather than MD4). It sends these checksums to the sender. • The sender computes the rolling checksum for every chunk of size S in its own version of the file, even overlapping chunks. This can be calculated efficiently because of a special property of the rolling checksum: if the rolling checksum of bytes n through n + S − 1 is R, the rolling checksum of bytes n + 1 through n + S can be computed from R, byte n, and byte n + S without having to examine the intervening bytes. Thus, if one had already calculated the rolling checksum of bytes 1–25, one could calculate the rolling checksum of bytes 2–26 solely from the previous checksum, and from bytes 1 and 26. • The rolling checksum used in rsync is based on Mark Adler's adler-32 checksum, which is used in zlib, and is itself based on Fletcher's checksum. August 18, 2011 Rsync on HP-UX Webinar 7
  • 8. Current Rsync Checksum Weaknesses – Part 2 • The sender then compares its rolling checksums with the set sent by the recipient to determine if any matches exist. If they do, it verifies the match by computing the hash for the matching block and by comparing it with the hash for that block sent by the recipient. • The sender then sends the recipient those parts of its file that did not match the recipient's blocks, along with information on where to merge these blocks into the recipient's version. This makes the copies identical. However, there is a small probability that differences between chunks in the sender and recipient are not detected, and thus remains uncorrected. This requires a simultaneous hash collision in MD5 and the rolling checksum. It is possible to generate MD5 collisions, and the rolling checksum is not cryptographically strong, but the chance for this to occur by accident is nevertheless extremely remote. With 128 bits from MD5 plus 32 bits from the rolling checksum, and assuming maximum entropy in these bits, the probability of a hash collision with this combined checksum is 2exp(−(128+32)) = 2exp(−160). The actual probability is a few times higher, since good checksums approach maximum output entropy but very rarely achieve it. August 18, 2011 Rsync on HP-UX Webinar 8
  • 9. Current Rsync Checksum Weaknesses – Part 3 • If the sender's and recipient's versions of the file have many sections in common, the utility needs to transfer relatively little data to synchronize the files. • The MD5 Message-Digest Algorithm is a widely used cryptographic hash function that produces a 128-bit (16-byte) hash value. Specified in RFC 1321, MD5 has been employed in a wide variety of security applications, and is also commonly used to check data integrity. However, it has been shown that MD5 is not collision resistant;[3] as such, MD5 is not suitable for applications like SSL certificates or digital signatures that rely on this property. An MD5 hash is typically expressed as a 32-digit hexadecimal number. August 18, 2011 Rsync on HP-UX Webinar 9
  • 10. Current Rsync Checksum Weaknesses – Part 4 • MD5 was designed by Ron Rivest in 1991 to replace an earlier hash function, MD4. In 1996, a flaw was found with the design of MD5. While it was not a clearly fatal weakness, cryptographers began recommending the use of other algorithms, such as SHA-1 (which has since been found also to be vulnerable). In 2004, more serious flaws were discovered, making further use of the algorithm for security purposes questionable; specifically, a group of researchers described how to create a pair of files that share the same MD5 checksum. Further advances were made in breaking MD5 in 2005, 2006, and 2007. In an attack on MD5 published in December 2008, a group of researchers used this technique to fake SSL certificate validity. • • US-CERT says MD5 "should be considered cryptographically broken and unsuitable for further use“. • Most U.S. government applications now require the SHA-2 family of hash functions. August 18, 2011 Rsync on HP-UX Webinar 10
  • 11. Rsync – Memory Utilization • At one time, we used rsync version 2.x at a large private hospital group in Australia (53 hospitals). It had one major issue when we used it extensively: huge directory trees were taking lot of RAM. • However, if you use rsync version 3.x, it does not keep the whole tree in memory if it is big – it uses an incrementalrecursion algorithm. August 18, 2011 Rsync on HP-UX Webinar 11
  • 12. Rsync on HP-UX • For HP-UX 11iv2 and 11iv3, it is delivered on HP-UX Internet Express * media, or can be obtained as free download at: http://www.software.hp.com/portal/swdepot/displayProductInfo.do?productNumber=HPUXIEXP1131 • For all currently supported HP-UX releases (11iv1 to 11iv3), there is also The Porting and Archiving Centre for HP-UX: http://hpux.connect.org.uk/hppd/hpux/Networking/Admin/rsync-3.0.8/ Make sure when installing depots from this site to also install ALL runtime dependencies (next slide). • Compile from sources. August 18, 2011 Rsync on HP-UX Webinar 12
  • 13. Rsync at The Porting and Archiving Centre for HP-UX Description: Rsync uses an algorithm which provides a very fast method for bringing remote files into sync. It does this by sending just the differences in the files, optionally with compression, across the link, without requiring that both sets of files are present at one of the ends of the link beforehand. Author: Andrew Tridgell, Paul Mackerras tridge@samba.org Home URL: http://rsync.samba.org/ License: GNU General Public License v3 Installation Tree: /usr/local Languages used: C Build-time dependencies: gettext libiconv make popt Run-time dependencies: gettext libiconv popt Documentation: Installation README Man Page August 18, 2011 Rsync on HP-UX Webinar 13
  • 14. Rsync Typical Scenario – Part 1 • Day 1 – the first backup. No data at the remote location, so a complete file transfer is required: Local File 4GB Remote File Data transferred: Rsync ~2GB (2:1 compression) FTP/rcp/scp ~ 4GB August 18, 2011 Rsync on HP-UX Webinar 14
  • 15. Rsync Typical Scenario – Part 2 • Day 2 – 0.5GB is added at the start of the file and the rest of the file is left intact (red blocks of data) *: Local File 4.5GB Remote File Data transferred: Rsync ~0.25GB (2:1 compression) FTP/rcp/scp ~ 4.5GB August 18, 2011 Rsync on HP-UX Webinar 15
  • 16. Rsync Typical Scenario – Part 3 • Day 3 – The green (0.4GB) and yellow (0.2GB) blocks of data are moved around (total of 0.6GB – the file contents are the same, just block moved around): Local File 4.5GB Remote File Data transferred: Rsync ~0 (a small overhead) FTP/rcp/scp ~ 4.5GB August 18, 2011 Rsync on HP-UX Webinar 16
  • 17. Rsync Command-line Options # /usr/local/bin/rsync -v August 18, 2011 Rsync on HP-UX Webinar 17
  • 18. Rsync from Internet Express It is important to pass the full path of the rsync command if it is not in the PATH: srvA# /opt/iexpress/rsync/bin/rsync -az -H -v --stats --rsync-path=/opt/iexpress/rsync/bin/rsync testfile.gz userB@srvB:/somedir August 18, 2011 Rsync on HP-UX Webinar 18
  • 19. Rsync from Porting and Archiving for HPUX srvA# /usr/local/bin/rsync -a -r -v -t -z --stats --progress --rsh=/usr/bin/ssh --rsync-path=/usr/local/bin/rsync /labs srvB: August 18, 2011 Rsync on HP-UX Webinar 19
  • 20. Rsync Example when Previous Run Interrupted or Failed • The flag “–partial” keeps the partially downloaded files on the target. • This option is useful if the data transfer process gets interrupted by some error of malfunction. • For example, if the rsync command terminated before all the data was transported we could launch the same command again: srvA# rsync -P myproject.gz userB@srvB:/somedir rsync would use partial check sums to test the validity of that part of the file that was already transported previously and start the actual data transport only for the missing parts of the myproject.gz file. August 18, 2011 Rsync on HP-UX Webinar 20
  • 21. Rsync and SSH Key Exchange userA@srvA# ssh-keygen -t rsa Do not enter a passphrase, or use ssh-agent is passphrase is important. userA@srvA# ssh userB@serverB mkdir -p .ssh userA@srvA# cat .ssh/id_rsa.pub | ssh userB@srvB cat >> .ssh/authorized_keys userA@srvA# ssh userB@srvB chmod 0700 .ssh/ userA@srvA# ssh userB@srvB chmod 0600 .ssh/authorized_keys userA@srvA# rsync -a -r -v -t -z --stats --progress -e ssh /sourcedir/ userB@srvB:/targetdir/ August 18, 2011 Rsync on HP-UX Webinar 21
  • 22. Rsync – SSH Chains and Different Port • rsync files between hosts that can not talk directly – use ssh chain): # rsync -av --rsh="ssh -TA userA@srvA ssh -TA -l userB" /mydir/ srvB:/somedir/ • rsync on different port and forcing IP protocol: # rsync --progress --partial --rsh="ssh -p 8322" --bwlimit=300 --ipv4 remuser@remsrv:~/myfile.tgz . August 18, 2011 Rsync on HP-UX Webinar 22
  • 23. Rsync Example with Full Copy srvA# /usr/local/bin/rsync -a -r -v -t -z --stats --progress --rsh=/usr/bin/ssh --rsync-path=/usr/local/bin/rsync /labs srvB: ... Number of files: 118 Number of files transferred: 109 Total file size: 483583369 bytes Total transferred file size: 483583369 bytes Literal data: 483583369 bytes Matched data: 0 bytes File list size: 2002 File list generation time: 0.016 seconds File list transfer time: 0.000 seconds Total bytes sent: 393465005 Total bytes received: 2119 August 18, 2011 Rsync on HP-UX Webinar 23
  • 24. Rsync Example with Sparse Files * – Part 1 srvA# prealloc sparsefile1 1024000 srvA# /usr/local/bin/rsync -a -A -r -v -t -z --stats --progress --r sync-path=/usr/local/bin/rsync /src srvB: srvB# ll sparsefile1 2000 -rw-r----sparsefile3 August 18, 2011 1 root sys 1024000 Aug 18 14:50 Rsync on HP-UX Webinar 24
  • 25. Rsync Example with Sparse Files * – Part 2 srvA# prealloc sparsefile2 1024000 srvA# /usr/local/bin/rsync -a -A -S -r -v -t -z --stats --progress --rsync-path=/usr/local/bin/rsync /src srvB: sending incremental file list src/sparsefile2 1024000 100% 24.88MB/s 0:00:00 (xfer#1, to-check=1/8) Total transferred file size: 1024000 bytes Literal data: 1024000 bytes ... Total bytes sent: 1329 Total bytes received: 35 sent 1329 bytes received 35 bytes total size is 28825484 August 18, 2011 2728.00 bytes/sec speedup is 21133.05 Rsync on HP-UX Webinar 25
  • 26. Rsync Example with Sparse Files – Part 3 srvA# ll sparsefile* 2000 -rw-r----- 1 root sys 1024000 Aug 18 14:45 sparsefile1 2000 -rw-r----- 1 root sys 1024000 Aug 18 14:50 sparsefile2 srvB# ll sparsefile* 2000 -rw-r----- 1 root sys 1024000 Aug 18 14:45 sparsefile1 0 -rw-r----- 1 root sys 1024000 Aug 18 14:50 sparsefile2 August 18, 2011 Rsync on HP-UX Webinar 26
  • 27. Rsync Example with ACLs * – Part 1 srvA# getacl vxdump # file: vxdump # owner: root # group: sys user::r-x group::r-x class:r-x other:--- srvA# setacl -m u:dusan:rx vxdump srvA# pwget -n dusan dusan:*:111:1::/home/dusan:/usr/bin/sh (User “dusan” has UID 111) # getacl vxdump # file: vxdump # owner: root # group: sys user::r-x user:dusan:r-x group::r-x class:r-x other:--August 18, 2011 Rsync on HP-UX Webinar 27
  • 28. Rsync Example with ACLs * – Part 2 srvA# /usr/local/bin/rsync -a -r -v -t -z --stats --progress --rsync-path=/usr/local/bin/rsync /src srvB: Check the file on the target srvB (ACL is missing): srvB# getacl /src/vxdump # file: /src/vxdump # owner: root # group: sys user::r-x group::r-x class:r-x other:--August 18, 2011 Rsync on HP-UX Webinar 28
  • 29. Rsync Example with ACLs * – Part 3 srvA# /usr/local/bin/rsync -a –A -r -v -t -z --stats --progress --rsync-path=/usr/local/bin/rsync /src srvB: Check the file on the target srvB (note that the ACL is owned by user “111” because “dusan” is obviously not in the password database, but the ACL is copied!). srvB# getacl /src/vxdump # file: /src/vxdump # owner: root # group: sys user::r-x user:111:r-x group::r-x class:r-x other:--August 18, 2011 Rsync on HP-UX Webinar 29
  • 30. Rsync with Dry Run • When doing something other than non-trivial copies or using features of rsync that you have never used before, add the “-n” or “--dry-run” switch to make it a dry run. # rsync -avhn /mydir /otherdir/ # rsync -nbrvvhn --del /mydir /otherdir/ # rsync -rn --size-only --exclude=*.iso /mydir/ /otherdir/ August 18, 2011 Rsync on HP-UX Webinar 30
  • 31. Rsync Anonymous Server from Porting and Archiving for HP-UX – Part 1 • Install the depot. • Add into /etc/group nobody::-2: • Add into /etc/services rsync • 873/tcp Add into /etc/inetd.conf rsync stream tcp nowait root /usr/local/bin/rsync rsyncd –daemon ... And reload the daemon (inetd –c) August 18, 2011 Rsync on HP-UX Webinar 31
  • 32. Rsync Anonymous Server from Porting and Archiving for HP-UX – Part 2 • Create /etc/rsyncd.conf uid = nobody gid = nobody use chroot = yes read only=yes max connections = 5 log file = /var/adm/syslog/rsyncd.log [ftp] path = /src comment = HP-UX Source export area • Check Bastille or disable it. August 18, 2011 Rsync on HP-UX Webinar 32
  • 33. Rsync Anonymous Server from Porting and Archiving for HP-UX – Part 3 To check what is available on the rsync server: # /usr/local/bin/rsync -avlH rsync://myhost src August 18, 2011 HP-UX Source export area Rsync on HP-UX Webinar 33
  • 34. Mirror By Rsync If a remote server runs anonymous rsync server, mirroring is achieved in the following manner: # /usr/local/bin/rsync -avvlH --rsync-path=/usr/local/bin/rsync myhost:/src/ /src opening connection using: ssh myhost /usr/local/bin/rsync --server --sender vvlHogDtpre.isf . /src/ Password: receiving incremental file list created directory /src delta-transmission enabled rsync-3.0.8-ia64-11.31.depot sparsefile2 sparsefile3 total: matches=0 hash_hits=0 • sent 166 bytes • total size is 29849484 August 18, 2011 false_alarms=0 data=29849484 received 29853794 bytes 5427992.73 bytes/sec speedup is 1.00 Rsync on HP-UX Webinar 34
  • 35. Rsync and LVM Snapshots – Part 1 • There is no official way to copy data from a snapshot back to the original volume. Here is how I normally accomplish this task, although it depends on what kind of files I am trying to copy, any special file attributes, and the amount of data • If the amount of data is not massive, and there are not many files in the file system), use rsync to re-synchronize data between a snapshot and the original. Be very careful about the command line, because if you use the wrong order of the volumes, the copy of the data goes in the wrong direction! # rsync -aHv /snap/ /orig Rsync is known to be VERY slow for big file transfers and when there are lot of files. August 18, 2011 Rsync on HP-UX Webinar 35
  • 36. Rsync and LVM Snapshots – Part 2 • Since file systems are VxFS, vxdump(1M) and vxrestore(1M) are very convenient. Much faster: # fsck /snap # vxdump 0f - /snap | (cd /orig && vxrestore xf -) • Use native HP-UX tools, like tar(1M), pax(1M), fbackup(1M), and cpio(1M). Here are some simple examples: # cd /snap && tar cpf - * | (cd /orig && tar xvpf -) # cd /snap && fbackup -i . -f - | ( cd /orig && frecover -Xsrf - ) # cd /snap && pax -w . | ( cd /orig && pax -r -pe ) August 18, 2011 Rsync on HP-UX Webinar 36
  • 37. Rsync and Expanded Remote File List Pass the listing of remote command as list of files to rsync. The following line will run the find command on the remote machine in the /remdir directory and rsync all “*.conf" files it finds to the local machine in the /somedir directory. # rsync -avR remsrv:'`find /remdir -name "*.[conf]"`„ /somedir/ August 18, 2011 Rsync on HP-UX Webinar 37
  • 38. Rsync and Relative Directory • If you want the target directory structure to be relative (chroot in a way) you can add flag"-R". • The directory structure /mydir/data would then look like /BACKUP/mydir/data/ as the sync path name starts from / on the source machine: # rsync -Ravx --timeout=30 --delete-excluded user@remsrv:/mydir/data/ /BACKUP/ August 18, 2011 Rsync on HP-UX Webinar 38
  • 39. Rsync and “Snapshots” • Use “--link-dest” option to create space-efficient snapshot-based backups. It appears to have multiple complete copies of the backed up data (one for each backup run) but files that do not change between runs are hard linked instead of creating new copies, thus saving space. • The major disadvantage of this technique is that if a file is corrupted due to disk error it is just as corrupt in all snapshots that link to that file. Having offline backups would protect against this possibility. August 18, 2011 Rsync on HP-UX Webinar 39
  • 40. Rsync - File Size and Bandwidth Limits If you need to update a site over a slow link, run two passes of rsync. • Transfer the small files firstly: # rsync -a --max-size=150K /mydir/ remsrv:/somedir/ • Then do the same for the large files and limit bandwidth to 100 KBytes per second: # rsync -a --min-size=150K --bwlimit=100 /mydir/ remsrv:/somedir/ August 18, 2011 Rsync on HP-UX Webinar 40
  • 41. Rsync – Exclude Files An example how to exclude files and directories from rsync transfer. # rsync -azhve ssh --stats --progress --exclude-from '/somedir/EXCLUDE.txt' --delete-excluded /mydir remsrv:/somedir/ August 18, 2011 Rsync on HP-UX Webinar 41
  • 42. Rsync – Avoid Checksum For Large Files Before Transfer • It is a common error to use option “-c” when huge files are transferred. rsync is going to have to read/checksum the entire file, and reading them is going to take a long time, unless the file is stored on SSDs or some very fast storage. • Instead, try: # rsync -vhz --partial --inplace … “-c” means that it checksums the entire file BEFORE doing any transfers, rather than using the timestamp to see if it has changed, which means reading the whole file twice. August 18, 2011 Rsync on HP-UX Webinar 42
  • 43. Rsync and “Clean Shell” • The "is your shell clean" message and the "protocol mismatch" message are usually caused by having some program or command running in Shell’s profiles (.cshrc, .profile, .kshrc, .bashrc or equivalent) every time when using a remote-shell program (such as ssh or rsh). Data written in this way corrupts the rsync data stream. rsync detects this at startup and produces those error messages. However, if you are using rsync-daemon syntax (host::path or rsync://) without using a remote-shell program (no --rsh or -e option), there is not remote-shell program involved, and the problem is probably caused by an error on the daemon side (so check the daemon logs). To test it: # ssh user@myhost.domain.dom /bin/true The above command should not output anything at all (except an ssh password prompt, if applicable). If there is any output on stdout, disable whatever is generating that output so that rsync does not get “garbage” trying to talk to the remote rsync. August 18, 2011 Rsync on HP-UX Webinar 43
  • 44. Rsync Common Problems Several common causes for a remote rsync process going away: • The destination disk is full (at least the size of the largest file that needs to be updated available must be in free disk space for the transfer to succeed). • An idle connection caused a router or remote-shell server to close the connection. • A network error caused the connection to be dropped. • The remote rsync executable was not found. • Remote Shell setup is not working right or is not "clean" (it is sending spurious text to rsync). • If the problem might be an idle connection getting closed, use a “--timeout” option (newer rsync versions send keep-alive messages during periods of no activity). August 18, 2011 Rsync on HP-UX Webinar 44
  • 45. Rsync Performance Analysis and Tuning There is no silver bullet! Test everything and rsync will certainly bring benefits. August 18, 2011 Rsync on HP-UX Webinar 45
  • 48. Rsync Command-line Options – Part 1 Usage: rsync [OPTION]... SRC [SRC]... DEST or rsync [OPTION]... SRC [SRC]... [USER@]HOST:DEST or rsync [OPTION]... SRC [SRC]... [USER@]HOST::DEST or rsync [OPTION]... SRC [SRC]... rsync://[USER@]HOST[:PORT]/DEST or rsync [OPTION]... [USER@]HOST:SRC [DEST] or rsync [OPTION]... [USER@]HOST::SRC [DEST] or rsync [OPTION]... rsync://[USER@]HOST[:PORT]/SRC [DEST] The ':' usages connect via remote shell, while '::' & 'rsync://' usages connect to an rsync daemon, and require SRC or DEST to start with a module name. August 18, 2011 Appendix- Rsync on HP-UX Webinar 48
  • 49. Rsync Command-line Options – Part 2 Options -v, --verbose increase verbosity -q, --quiet suppress non-error messages --no-motd suppress daemon-mode MOTD (see manpage caveat) -c, --checksum skip based on checksum, not mod-time & size -a, --archive archive mode; equals -rlptgoD (no -H,-A,-X) --no-OPTION turn off an implied OPTION (e.g. --no-D) -r, --recursive recurse into directories -R, --relative use relative path names --no-implied-dirs -b, --backup don't send implied dirs with --relative make backups (see --suffix & --backup-dir) --backup-dir=DIR make backups into hierarchy based in DIR --suffix=SUFFIX set backup suffix (default ~ w/o --backup-dir) -u, --update skip files that are newer on the receiver --inplace update destination files in-place (SEE MAN PAGE) --append append data onto shorter files --append-verify like --append, but with old data in file checksum August 18, 2011 Appendix- Rsync on HP-UX Webinar 49
  • 50. Rsync Command-line Options – Part 3 Options -d, --dirs transfer directories without recursing -l, --links copy symlinks as symlinks -L, --copy-links transform symlink into referent file/dir --copy-unsafe-links only "unsafe" symlinks are transformed --safe-links ignore symlinks that point outside the source tree -k, --copy-dirlinks transform symlink to a dir into referent dir -K, --keep-dirlinks treat symlinked dir on receiver as dir -H, --hard-links preserve hard links -p, --perms preserve permissions -E, --executability preserve the file's executability --chmod=CHMOD affect file and/or directory permissions -A, --acls preserve ACLs (implies --perms) -o, --owner preserve owner (super-user only) -g, --group preserve group --devices preserve device files (super-user only) --specials preserve special files August 18, 2011 Appendix- Rsync on HP-UX Webinar 50
  • 51. Rsync Command-line Options – Part 4 Options -D same as --devices --specials -t, --times preserve modification times -O, --omit-dir-times omit directories from --times --super receiver attempts super-user activities -S, --sparse handle sparse files efficiently -n, --dry-run perform a trial run with no changes made -W, --whole-file copy files whole (without delta-xfer algorithm) -x, --one-file-system don't cross filesystem boundaries -B, --block-size=SIZE force a fixed checksum block-size -e, --rsh=COMMAND specify the remote shell to use --rsync-path=PROGRAM specify the rsync to run on the remote machine --existing skip creating new files on receiver --ignore-existing skip updating files that already exist on receiver --remove-source-files sender removes synchronized files (non-dirs) --del an alias for --delete-during --delete delete extraneous files from destination dirs --delete-before receiver deletes before transfer, not during --delete-during receiver deletes during transfer (default) --delete-delay find deletions during, delete after --delete-after receiver deletes after transfer, not during --delete-excluded also delete excluded files from destination dirs --ignore-errors delete even if there are I/O errors --force force deletion of directories even if not empty --max-delete=NUM don't delete more than NUM files --max-size=SIZE don't transfer any file larger than SIZE --min-size=SIZE don't transfer any file smaller than SIZE --partial keep partially transferred files --partial-dir=DIR put a partially transferred file into DIR --delay-updates put all updated files into place at transfer's end August 18, 2011 Appendix- Rsync on HP-UX Webinar 51
  • 52. Rsync Command-line Options – Part 5 Options -m, --prune-empty-dirs prune empty directory chains from the file-list --numeric-ids don't map uid/gid values by user/group name --timeout=SECONDS set I/O timeout in seconds --contimeout=SECONDS set daemon connection timeout in seconds -I, --ignore-times don't skip files that match in size and mod-time --size-only skip files that match in size --modify-window=NUM compare mod-times with reduced accuracy -T, --temp-dir=DIR create temporary files in directory DIR -y, --fuzzy find similar file for basis if no dest file --compare-dest=DIR also compare destination files relative to DIR --copy-dest=DIR ... and include copies of unchanged files --link-dest=DIR hardlink to files in DIR when unchanged -z, --compress compress file data during the transfer --compress-level=NUM explicitly set compression level --skip-compress=LIST skip compressing files with a suffix in LIST -C, --cvs-exclude August 18, 2011 auto-ignore files the same way CVS does Appendix- Rsync on HP-UX Webinar 52
  • 53. Rsync Command-line Options – Part 6 Options -f, --filter=RULE add a file-filtering RULE -F same as --filter='dir-merge /.rsync-filter' repeated: --filter='- .rsync-filter' --exclude=PATTERN exclude files matching PATTERN --exclude-from=FILE read exclude patterns from FILE --include=PATTERN don't exclude files matching PATTERN --include-from=FILE read include patterns from FILE --files-from=FILE read list of source-file names from FILE -0, --from0 all *-from/filter files are delimited by 0s -s, --protect-args no space-splitting; only wildcard special-chars --address=ADDRESS bind address for outgoing socket to daemon --port=PORT specify double-colon alternate port number --sockopts=OPTIONS specify custom TCP options --blocking-io use blocking I/O for the remote shell --stats give some file-transfer stats -8, --8-bit-output leave high-bit chars unescaped in output -h, --human-readable output numbers in a human-readable format --progress August 18, 2011 show progress during transfer Appendix- Rsync on HP-UX Webinar 53
  • 54. Rsync Command-line Options – Part 7 Options -P same as --partial --progress -i, --itemize-changes output a change-summary for all updates --out-format=FORMAT output updates using the specified FORMAT --log-file=FILE log what we're doing to the specified FILE --log-file-format=FMT log updates using the specified FMT --password-file=FILE read daemon-access password from FILE --list-only list the files instead of copying them --bwlimit=KBPS limit I/O bandwidth; KBytes per second --write-batch=FILE write a batched update to FILE --only-write-batch=FILE like --write-batch but w/o updating destination --read-batch=FILE read a batched update from FILE --protocol=NUM force an older protocol version to be used --iconv=CONVERT_SPEC request charset conversion of filenames -4, --ipv4 prefer IPv4 -6, --ipv6 prefer IPv6 --version (-h) --help August 18, 2011 print version number show this help (-h works with no other options) Appendix- Rsync on HP-UX Webinar 54