业精于勤,荒于嬉
全部博文(763)
分类: 系统运维
2017-07-24 15:38:40
官网对Unison的描述:
Unison is a file-synchronization tool for Unix and Windows. It allows two replicas of a collection of files and directories to be stored on different hosts (or different disks on the same host), modified separately, and then brought up to date by propagating the changes in each replica to the other.
可以知道,Unison是windows和unix平台下都可以使用的双向文件同步工具,它能使两个文件夹(本地或网络 上的)保持内容的一致。 unison 拥有其它一些同步工具或文件系统 的相同特性,但也有自己的特点:
1、跨平台使用;
2、对内核和用户 权限 没有特别要求;
3、unison 是双向的,它能自动 处理两分拷贝中更新没有冲突的部分,有冲突的部分将会显示出来让用户选择更新策略;
只要是能连通的两台主机 ,就可以运行 unison ,可以直接使用 socket 连接或安全的 ssh 连接方式,对带宽 的要求不高,使用类似 rsync 的压缩传输协议。
Unison双向同步的一个缺点是,对于同名文件在两个同步文件夹中都被修改时,unison是不会去同步的,因为unison无法判断以那个为准.
Unison双向同步基本原理是:假如有A B两个文件夹,A文件夹把自己的改动同步到B,B文件夹也把自己的改动同步到A,最后A B两文件夹的内容相同,是A B文件夹的合集.
Unison 有文字界面和图形界面,这里只介绍如何在文字界面下使用。
~bcpierce/unison//download.html
有二进制包的 ,也有源码包的。二进制包可以直接使用 。这里只介绍源码安装
下载Unison最新源码包:
[root@www tmp]# wget ~bcpierce/unison//download/releases/stable/unison-2.40.63.tar.gz
解压:
[root@www tmp]# tar -zxvf unison-2.40.63.tar.gz
进入unison-2.40.63
[root@www tmp]# cd unison-2.40.63
阅读里面的INSTALL文件:
只有一句话:
For installation instructions, see the the INSTALLATION section of the
user manual.
只好去下载用户手册(以前的版本不用去下载用户手册,安装说明就直接在INSTALL文件里)
[root@www unison-2.40.63]# wget ~bcpierce/unison//download/releases/stable/unison-2.40.63-manual.pdf
打开下载到的用户手册,里面有一段文字是关于在Unix/Linux下编译安装unison的:
3.4.1 Unix
You’ll need the Objective Caml compiler (version 3.11.2 or later), which is available from
Building and installing OCaml on Unix systems is very straightforward; just follow the instructions in the
distribution. You’ll probably want to build the native-code compiler in addition to the bytecode compiler,
as Unison runs much faster when compiled to native code, but this is not absolutely necessary. (Quick start:
on many systems, the following sequence of commands will get you a working and installed compiler: ?rst
do make world opt, then su to root and do make install.)
You’ll also need the GNU make utility, standard on many Unix systems. (Type make –version to check
that you’ve got the GNU version.)
Once you’ve got OCaml installed, grab a copy of the Unison sources, unzip and untar them, change to
the new unison directory, and type “make UISTYLE=text.” The result should be an executable ?le called
unison. Type ./unison to make sure the program is executable. You should get back a usage message.
If you want to build the graphical user interface, you will need to install two additional things:
? The Gtk2 libraries. These areavailable from and are standard on many Unix
installations.
? The lablgtk2 OCaml library. Grab the developers’ tarball from
untar it, and follow the instructions to build and install it.
(Quick start: make configure, then make, then make opt, then su and make install.)
Now build unison. If your search paths are set up correctly, simply typing make again should build a
unison executable with a Gtk2 graphical interface. (In previous releases of Unison, it was necessary to add
UISTYLE=gtk2 to the ’make’ command above. This requirement has been removed: the make?le should
detect automatically when lablgtk2 is present and set this ?ag automatically.)
Put the unison executable somewhere in your search path, either by adding the Unison directory to your
PATH variable or by copying the executable to some standard directory where executables are stored.
前面四段的说明是关于在Unix/Linux下安装编译文本模式的unison,而后面几段是关于在Unix/Linux下安装图形界面的Unison。
在这里只讲文本模式下的Unison(因为在文本模式下可以使用脚本,所以不用图形界面)
从第一段可以知道,在Unix/Linux下从源码包编译安装unison,需要一个叫做Objective Caml compiler 的编译器,而且版本最低至少为version 3.11.2,下载地址为 。
下载ocaml最新版本:
[root@www tmp]# wget /pub/distrib/ocaml-3.12/ocaml-3.12.0.tar.gz
# tar -zxf ocaml-3.09.3.tar.gz
第二段则是关于编译安装ocaml编译器的说明,按照说明进行编译安装:
解压:
[root@www tmp]# tar -zxvf ocaml-3.12.0.tar.gz
进入:
[root@www tmp]# cd ocaml-3.12.0
执行./configure:
[root@www ocaml-3.12.0]# ./configure
编译:
[root@www ocaml-3.12.0]# make world opt
(编译时间比较长,请耐心等待)
安装:
[root@www ocaml-3.12.0]# make install
清理现场:
[root@www ocaml-3.12.0]# make clean
第四段则是安装unison的步骤
(Unison 对版本要求很高,进行同步的两台主机需要相同版本的 unison)
编译:
[root@www unison-2.40.63]# make UISTYLE=text
安装:
[root@www unison-2.40.63]# make install
安装报错:
mv /root/bin//unison /tmp/unison-12798
mv: 无法 stat “/root/bin//unison”: 没有那个文件或目录
make: [doinstall] 错误 1 (忽略)
cp unison /root/bin/
cp: 无法创建一般文件“/root/bin/”: 是一个目录
make: *** [doinstall] 错误 1
解决办法:
创建/root/bin文件夹
[root@www unison-2.40.63]# mkdir /root/bin
再次执行make install,还是报如下错误:
[root@www unison-2.40.63]# make install
mv /root/bin//unison /tmp/unison-12868
mv: 无法 stat “/root/bin//unison”: 没有那个文件或目录
make: [doinstall] 错误 1 (忽略)
cp unison /root/bin/
cp unison /root/bin/unison-2.40
尝试再次执行make install,阴差阳错地解决了问题:
[root@www unison-2.40.63]# make install
mv /root/bin//unison /tmp/unison-12899
cp unison /root/bin/
cp unison /root/bin/unison-2.40
(以前的版本安装完毕后还需要把生成的可执行文件unison拷贝到系统PATH中去,而现在的这个版本在安装时已经自行拷贝,其路径为/root/bin/unison)
至此,全部安装完毕
安装好的Unison的默认配置文件为/root/.unison/default.prf
由于 unison 在远程同步文件夹要登陆远程服务器 , 因此要配置两机互相信任
本地机:
[root@bogon ~]# uname -a
Linux bogon 2.6.18-238.9.1.el5 #1 SMP Tue Apr 12 18:10:56 EDT 2011 i686 athlon i386 GNU/Linux
[root@bogon ~]# ifconfig
eth0 Link encap:Ethernet HWaddr 1C:6F:65:D4:57:16
inet addr:172.16.10.189 Bcast:172.16.10.255 Mask:255.255.255.0
inet6 addr: fe80::1e6f:65ff:fed4:5716/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:401 errors:0 dropped:0 overruns:0 frame:0
TX packets:511 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:178233 (174.0 KiB) TX bytes:56965 (55.6 KiB)
Interrupt:225 Base address:0×4000
远程机:
[root@www www]# uname -a
Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux
[root@www ~]# ifconfig
eth0 Link encap:Ethernet HWaddr 00:0C:29:33:39:61
inet addr:172.16.10.16 Bcast:172.16.10.255 Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:fe33:3961/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:68 errors:0 dropped:0 overruns:0 frame:0
TX packets:100 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:9172 (8.9 KiB) TX bytes:16481 (16.0 KiB)
Base address:0×2000 Memory:d8920000-d8940000
1、确保机器上安装了openssh,如果没有则先安装
[root@www ~]# yum install openssh*
2、在两台机器上创建 RSA 密钥
以下操作要在本地机和远程机上都执行一遍
(1)在 root 用户的主目录内创建.ssh目录并设置正确的权限
[root@www ~]# mkdir ~/.ssh
[root@www ~]# chmod 700 ~/.ssh
(2)使用ssh-keygen命令生成第2版本的SSH协议的RSA密钥
[root@www ~]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
41:e1:f0:64:5e:a8:d7:c9:29:2d:76:04:68:e6:35:d3 root@
(3)启动ssh服务:
[root@www .ssh]# service sshd start
启动 sshd: [确定]
在提示保存私钥( key )和公钥( public key )的位置时,使用默认值。
如果需要私钥密码 ( passphrase ),则输入一个私钥密码(如果使用私钥密码,使用 ssh 执行远程命令时需要输入私钥密码,因此,本次实验中未使用私钥密码),直接回车即可。 、添加密钥到授权密钥文件( authorized key file )中
在本地机上执行:
3
(1)切换到~/.ssh目录:
[root@www ~]# cd ~/.ssh
(2)生成授权密钥文件:
[root@www .ssh]# ssh 172.16.10.189 cat /root/.ssh/id_rsa.pub >> authorized_keys
The authenticity of host ’172.16.10.189 (172.16.10.189)’ can’t be established.
RSA key fingerprint is 47:0e:fe:ba:fa:e6:d1:8f:2b:62:98:37:bf:30:65:33.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ’172.16.10.189′ (RSA) to the list of known hosts.
root@172.16.10.189′s password:
[root@www .ssh]# ssh 172.16.10.16 cat /root/.ssh/id_rsa.pub >> authorized_keys
The authenticity of host ’172.16.10.16 (172.16.10.16)’ can’t be established.
RSA key fingerprint is ac:a8:8a:2c:ec:54:11:f5:41:ef:4a:1a:71:d2:42:0e.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ’172.16.10.16′ (RSA) to the list of known hosts.
root@172.16.10.16′s password:
(3)更改授权密钥文件的权限:
[root@www .ssh]# chmod 600 authorized_keys
(4)拷贝授权密钥文件到远程机:
[root@www .ssh]# scp authorized_keys 172.16.10.16:/root/.ssh/
root@172.16.10.16′s password:
authorized_keys 100% 809 0.8KB/s 00:00
(5)测试
配置完成后,在本地机上执行:
[root@www ~]# ssh 172.16.10.189 date
2011年 04月 25日 星期一 15:04:06 CST
[root@www ~]# ssh 172.16.10.16 date
一 4月 25 15:05:02 CST 2011
不需要输入密码就出现系统日期,说明 SSH 配置成功。
(PS:在远程机上第一次执行时会有警告,直接输入yes就可以了)
Unison 可以在一台主机上使用,同步两个文件夹,也可以在网络上使用。
(一)Unison本地使用
1、使用方法:
unison DIR1 DIR2
2、实例:
在/home/www/文件夹里创建两个文件夹html和src,然后在html文件夹里创建一个text文件,同步html和src文件夹
[root@www www]# mkdir html src
[root@www www]# touch html/text
[root@www www]# unison html/ src/
Contacting server…
Looking for changes
Warning: No archive files were found for these roots, whose canonical names are:
/home/www/html
/home/www/src
This can happen either
because this is the first time you have synchronized these roots,
or because you have upgraded Unison to a new version with a different
archive format.
Update detection may take a while on this run if the replicas are
large.
Unison will assume that the ‘last synchronized state’ of both replicas
was completely empty. This means that any files that are different
will be reported as conflicts, and any files that exist only on one
replica will be judged as new and propagated to the other replica.
If the two replicas are identical, then no changes will be reported.
If you see this message repeatedly, it may be because one of your machines
is getting its address from DHCP, which is causing its host name to change
between synchronizations. See the documentation for the UNISONLOCALHOSTNAME
environment variable for advice on how to correct this.
Donations to the Unison project are gratefully accepted:
~bcpierce/unison
Press return to continue.[
(按enter继续)
Press return to continue.[
html src
file —-> text [f]
(输入f继续)
Proceed with propagating updates? []
(输入y继续)
Propagating updates
UNISON 2.40.63 started propagating changes at 15:25:36.99 on 25 Apr 2011
[BGN] Copying text from /home/www/html to /home/www/src
[END] Copying text
UNISON 2.40.63 finished propagating changes at 15:25:36.99 on 25 Apr 2011
Saving synchronizer state
Synchronization complete at 15:25:36 (1 item transferred, 0 skipped, 0 failed)
3、同步过程分析:
Unison检测到两个文件夹有所不同后给出提示,让用户选择相应的操作。如上例所示 .
表示右边html的文件夹有新的文件,是否同步到左边的src文件夹,f表示force,然后将确认, 进行更新。如果不知道要怎么操作,则可以输入?,会有更详细的介绍。
1、使用方法:
unison < 本地目录 > ssh://username@remotehostname(IP)/< 远程目录的绝对路径 >
2、实例:
(1)本地机上的同步目录:
[root@www www]# pwd
/home/www
[root@www www]#
(2)远程机上的同步目录:
[root@www bakup]# pwd
/home/bakup
(3)注意事项:
由于编译安装后的unison的可执行文件在/root/bin,因此在进行远程同步时需要指定其路径,否则会有如下报错:
[root@www www]# unison /home/www ssh://root@172.16.10.16//home/bakup
Contacting server…
bash: unison: command not found
Fatal error: Lost connection with the server
(4)正确使用方法:
需要加上-servercmd=DIR参数,如下:
[root@www www]# /root/bin/unison -servercmd=/root/bin/unison /home/www ssh://root@172.16.10.16//home/bakup
Contacting server…
……
3、同步过程分析:
本机的目录/home/www和远端主机的/home/bakup进行同步。一般的,需要两台机ssh能连接。
4、注意事项及技巧:
(1)注意事项:[root@www www]# /root/bin/unison -servercmd=/root/bin/unison /home/www ssh://root@172.16.10.16//home/bakup
在主机和目录间是有两个 “/”。
(2)技巧:
在同步的过程中可以知道,unison会很友好的提示操作方法,但是对于熟悉的用户来说,这些步骤太繁琐了,那有没有比较简单的办法呢?
有!在执行的时候加上一些参数即可:
[root@www www]# /root/bin/unison -servercmd=/root/bin/unison -silent -times -perms=-1 -owner=true -group=true -batch /home/www ssh://root@172.16.10.16//home/bakup
至于这些参数说明就看下面的了。
(三)Unison 参数说明
Unison 有很多参数 , 这里只介绍经常使用的几个 , 详细的请参看 unison 手册 :
”
-testserver
测试连通性,连接到服务器即退出。示例 :
[root@bogon ~]#
unison / ssh://root@172.16.10.16/
-servercmd=/root/bin/unison -testserver
如果服务器端 unison 可执行文件不在默认目录下,甚至没有 unison 命令(需要你编译一个上传到服务器),则需要使用 -servercmd参数告诉要执行的服务器 unison 命令位置。
使用 -testserver 参数,则成功链接即退出,也不会去执行目录的比较等后续操作。
”
-servercmd xxx
告诉 unison ,
服务器端的 unison 命令是什么。参见上面的示例。
”
-auto
接受缺省的动作,然后等待用户确认是否执行。
”
-batch
batch mode, 全自动模式,接受缺省动作,并执行。
”
-ignore xxx
增加 xxx 到忽略列表中
”
-ignorecase [true|false|default]
是否忽略文件名大小写
”
-follow xxx
是否支持对符号连接指向内容的同步
”
owner = true ( 保持同步过来的文件属主 )
”
group = true ( 保持同步过来的文件组信息 )
”
perms = -1
( 保持同步过来的文件读写权限 )
”
repeat = 1
( 间隔 1 秒后 , 开始新的一次同步检查 )
”
retry = 3
( 失败重试 )
”
sshargs = -C ( 使用 ssh 的压缩传输方式 )
”
xferbycopying = true不变目录,扫描 时可以忽略
”
-silent
安静模式
”
-times
同步修改时间
”
-path xxx 参数
只同步 -path 参数指定的子目录以及文件,而非整个目录。 -path 可以多次出现 , 例如 五、Unison配置文件的使用
尽管可以完全通过命令行的方式来指定 unison 运行所需要的参数,但还是推荐使用配置文件来进行配置使用unison,原因很简单,看配置文件比看命令行容易理解,而且可管理性更强。
默认的配置文件夹位于 ~currentuser/.unison,由于本次是在root用户下实验,故其配置文件在root用户的主目录下, 即/root/.unison,默认的配置文件名是 default.prf.
运行这样的命令 :
[root@bogon ~]# unison unisonsync
”
-immutable xxx
unison /home/username ssh://remotehost//home/username
-path shared
-path pub
-path .netscape/bookmarks.html
Unison将默认读取/root/.unison/unisonsync.prf 文件里的配置信息 。Unisonsync为自己新建的配置文件,配置信息在/root/.unison/unisonsync.prf, 因此我们可以根据上面参数的介绍 , 把所有的参数配置信息写入到一个 .prf 的文件中 .
下面是本次实验中unison应用实例的配置信息 :
root=/home/promise/html
root=ssh://root@172.16.10.16//home/qqing/bak
servercmd=/root/bin/unison
#ignore = Path as/*
batch = true
#repeat = 5
retry = 30
owner = true
group = true
perms = -1
fastcheck=true
rsync =false
#debug=verbose
sshargs = -C
xferbycopying = true
#log = false
#logfile = /root/.unison/huarong.log
times=true
说明如下 :
两个 root 表示需要同步的文件夹
batch = true, 表示全自动模式,接受缺省动作,并执行 -fastcheck。true 表示同步时使用文件的创建时间来比较两地文件,如果这个选项为 false , unison 则将比较两地文件的内容 . 建议设置为 true
log = true 表示在终端输出运行信息。
logfile 则指定了同时将输出写入 log 文件。
[root@www unison-2.40.63]# make install
mv /root/bin//unison /tmp/unison-12798
mv: 无法 stat “/root/bin//unison”: 没有那个文件或目录
make: [doinstall] 错误 1 (忽略)
cp unison /root/bin/
cp: 无法创建一般文件“/root/bin/”: 是一个目录
make: *** [doinstall] 错误 1
解决办法:
创建/root/bin文件夹
[root@www unison-2.40.63]# mkdir /root/bin
再次执行make install,还是报如下错误:
[root@www unison-2.40.63]# make install
mv /root/bin//unison /tmp/unison-12868
mv: 无法 stat “/root/bin//unison”: 没有那个文件或目录
make: [doinstall] 错误 1 (忽略)
cp unison /root/bin/
cp unison /root/bin/unison-2.40
尝试再次执行make install:
[root@www unison-2.40.63]# make install
mv /root/bin//unison /tmp/unison-12899
cp unison /root/bin/
cp unison /root/bin/unison-2.40
问题解决.
(二)注意事项:
(1)[root@www www]# /root/bin/unison -servercmd=/root/bin/unison /home/www ssh://root@172.16.10.16//home/bakup
在主机和目录间是有两个 “/”。
(2)由于编译安装后的unison的可执行文件在/root/bin,因此在进行远程同步时需要指定其路径,否则会有如下报错:
[root@www www]# unison /home/www ssh://root@172.16.10.16//home/bakup
Contacting server…
bash: unison: command not found
Fatal error: Lost connection with the server
正确使用方法:
需要加上-servercmd=DIR参数,如下:
[root@www www]# /root/bin/unison -servercmd=/root/bin/unison /home/www ssh://root@172.16.10.16//home/bakup
Contacting server…
……
(3)在和远程服务器同步大量数据,上传一部分数据后,超时:
9%
559:15 ETARead from remote host bluehost: Connection reset by peer
Fatal error: Lost connection with the server
实际操作中,最好的方法是 , 第一次先把要上传的文件打成包,用 ftp 上传,然后展开到服务器中,之后执行一次 unison 同步即可。
(三)网上相关资料
1、编译安装报错:
[root@bogon unison-2.40.63]# make UISTYLE=text
val map : (‘a -> ‘b) -> ‘a t -> ‘b t val mapi : (key -> ‘a -> ‘b) -> ‘a t -> ‘b t end
The field `split’ is required but not provided
The field `choose’ is required but not provided
The field `max_binding’ is required but not provided
The field `min_binding’ is required but not provided
The field `bindings’ is required but not provided
The field `cardinal’ is required but not provided
The field `partition’ is required but not provided
The field `filter’ is required but not provided
The field `exists’ is required but not provided
The field `for_all’ is required but not provided
The field `merge’ is required but not provided
The field `singleton’ is required but not provided make: *** [update.cmx] 错误 2
解决方法:
修改 update.mli 把第四行的 module NameMap : Map.S with type key = Name.t 改成 module NameMap : MyMap.S with type key = Name.t
[root@bogon unison-2.40.63]# make UISTYLE=text
成功,不再报错
[root@bogon unison-2.40.63]# mkdir /root/bin
[root@bogon unison-2.40.63]# make install
mv /root/bin//unison /tmp/unison-19913
mv: 无法 stat “/root/bin//unison”: 没有那个文件或目录 make: [doinstall] 错误 1 (忽略)
cp unison /root/bin/
cp unison /root/bin/unison-2.32
解决办法见
六、问题排除及注意事项总结
(一)安装Unison报错
2、SSH超时问题
和远程服务器同步大量数据,上传一部分数据后,超时:
9% 559:15 ETARead from remote host bluehost: Connection reset by peer
Fatal error: Lost connection with the server
该文章提出增大 ssh 超时时间设置:
hello users, I am trying to run unison to sync up 2 different servers.
The directory I am syncing is pretty large… about 3000 subdirs and
each of those subdirs has a couple dozen dirs under them. About 80 GBs
of data total.
I am using the following command:
unison -batch /dir/path ssh://remoteserver//dir/path
I originally used rsync to move the directory over to the remote
server however I now have a need for bidirectional mirroring hence the
reason why I now use unison.
When I run this command, I get the usual warning about no prior index
found since this is the first run.
Unison then begins to list all the dirs and subdirs (building the
index dbase I suppose). When it’s almost finished, I get the following
lines:
Waiting for changes from server
Fatal Error: Lost connection with server
I have used unison before to mirror directories between these two
machines, but none of which were this big with so many subdir branches.
Nothing in any logs to give me any clues. I though it might be some
sort of ssh timeout, so I added “ConnectTimeout 10″ in
/etc/ssh/ssh_config to give it a 10 second timeout, but I dont think
this will help since the ssh session does indeed connect initially.
ulimit -s shows I have an 8MB stack size.
Can anyone shed any light onto this problem?
Thank you
另外一个用户提出将一个任务分解成多个任务的方法:
The connection between the client and the server can remain idle for a
long time when checking for updates. So, if there is a masquerading
router (or a firewall using stateful inspection) between the client
and the server, it may wrongly consider that the connection is dead
after some time.
The usual workaround is to use the path directive to synchronize
smaller parts of the replica at a time. This should necessary only
for the first synchronization, as the subsequent ones are much faster.
You could also try to use the ServerAliveInterval directive of ssh in
order to keep the connection active.
见附件
见附件
见附件