docke swarm mode 集群应用之四-私有仓库harbor设置过程

docker swarm mode 集群下私有仓库harbor设置过程

说明

测试域名：reg.goluk.cn #内网有效
目标：通过访问 可以创建管理私有仓库所有项目；harbor作为内网docker swarm mode 集群的仓库，各个docker node  可以通过docker login reg.goluk.cn 登录，可以push 和pull 映像。

下载在线安装包

wget -c 

解压安装

tar xvf harbor-online-installer-.tgz

修改harbor.cfg

hostname = reg.goluk.cn

uiurlprotocol = http

email_identity = lihui@

emailserver = smtp.mydomain.com emailserverport = 25 emailusername = [e-mail] emailpassword = abc emailfrom = admin [e-mail]> email_ssl = false

harboradminpassword = Harbor12345

authmode = dbauth

ldap_url = ldaps://ldap.mydomain.com

ldap_basedn = ou=people,dc=mydomain,dc=com

ldap_uid = uid

ldap_scope = 3

db_password = mobnote@123

self_registration = on

usecompressedjs = on

maxjobworkers = 3

token_expiration = 30

verifyremotecert = on

customize_crt = on

crtcountry = CN crtstate = State crtlocation = CN crtorganization = goluk crtorganizationalunit = goluk crtcommonname = goluk.cn crt_email = [e-mail]

projectcreationrestriction = everyone

sslcert = /data/cert/server.crt sslcert_key = /data/cert/server.key

运行安装脚本

./install.sh

修改docker-compose.yml 形成docker stack 兼容文件

cat harbor2.yml
version: '3'
services:
log:
 image: vmware/harbor-log:0.5.0
 volumes:
   - /var/log/harbor/:/var/log/docker/
 ports:
   - 1514:514
registry:
  image: library/registry:2.5.0
  volumes:
    - /mnt/cephfs/app/harbor/data/registry:/storage
    - /mnt/cephfs/app/harbor/harbor/common/config/registry/:/etc/registry/
  environment:
    - GODEBUG=netdns=cgo
  command:
    ["serve", "/etc/registry/config.yml"]
  depends_on:
    - log
  logging:
    driver: "syslog"
    options:
      syslog-address: "tcp://127.0.0.1:1514"
      tag: "registry"
mysql:
  image: vmware/harbor-db:0.5.0
    volumes:
     - /mnt/cephfs/app/harbor/data/database:/var/lib/mysql
  environment:
    - MYSQL_ROOT_PASSWORD=mobnote@123
  depends_on:
    - log
  logging:
    driver: "syslog"
  options:
    syslog-address: "tcp://127.0.0.1:1514"
    tag: "mysql"
ui:
  image: vmware/harbor-ui:0.5.0
  environment:
    - MYSQL_HOST=mysql
  - MYSQL_PORT=3306
  - MYSQL_USR=root
  - MYSQL_PWD=mobnote@123
  - REGISTRY_URL=
  - JOB_SERVICE_URL=
  - UI_URL=
  - CONFIG_PATH=/etc/ui/app.conf
  - EXT_REG_URL=reg.goluk.cn
  - HARBOR_ADMIN_PASSWORD=Harbor12345
  - AUTH_MODE=db_auth
  - LDAP_URL=ldaps://ldap.mydomain.com
  - LDAP_SEARCH_DN=
  - LDAP_SEARCH_PWD=
  - LDAP_BASE_DN=ou=people,dc=mydomain,dc=com
  - LDAP_FILTER=
  - LDAP_UID=uid
  - LDAP_SCOPE=3
  - UI_SECRET=YEiVW92oM0szGsWa
  - SECRET_KEY=4tDRVqYEj4YjCdNI
  - SELF_REGISTRATION=on
  - USE_COMPRESSED_JS=on
  - LOG_LEVEL=debug
  - GODEBUG=netdns=cgo
  - EXT_ENDPOINT=
  - TOKEN_ENDPOINT=
  - VERIFY_REMOTE_CERT=on
  - TOKEN_EXPIRATION=30
  - PROJECT_CREATION_RESTRICTION=everyone
volumes:
  - /mnt/cephfs/app/harbor/harbor/common/config/ui/app.conf:/etc/ui/app.conf
  - /mnt/cephfs/app/harbor/harbor/common/config/ui/private_key.pem:/etc/ui/private_key.pem
  - /mnt/cephfs/app/harbor/data:/harbor_storage
depends_on:
  - log
logging:
  driver: "syslog"
  options:
    syslog-address: "tcp://127.0.0.1:1514"
    tag: "ui"
jobservice:
  image: vmware/harbor-jobservice:0.5.0
  environment:
    - MYSQL_HOST=mysql
  - MYSQL_PORT=3306
  - MYSQL_USR=root
  - MYSQL_PWD=mobnote@123
  - UI_SECRET=YEiVW92oM0szGsWa
  - SECRET_KEY=4tDRVqYEj4YjCdNI
  - CONFIG_PATH=/etc/jobservice/app.conf
  - REGISTRY_URL=
  - VERIFY_REMOTE_CERT=on
  - MAX_JOB_WORKERS=3
  - LOG_LEVEL=debug
  - LOG_DIR=/var/log/jobs
  - GODEBUG=netdns=cgo
  - EXT_ENDPOINT=
  - TOKEN_ENDPOINT=
volumes:
  - /mnt/cephfs/app/harbor/data/job_logs:/var/log/jobs
  - /mnt/cephfs/app/harbor/harbor/common/config/jobservice/app.conf:/etc/jobservice/app.conf
depends_on:
  - ui
logging:
  driver: "syslog"
  options:
    syslog-address: "tcp://127.0.0.1:1514"
    tag: "jobservice"
proxy:
  image: nginx:1.11.5
  volumes:
  - /mnt/cephfs/app/harbor/harbor/common/config/nginx:/etc/nginx
ports:
  - 80:80
  - 443:443
depends_on:
  - mysql
  - registry
  - ui
  - log
logging:
  driver: "syslog"
  options:
    syslog-address: "tcp://127.0.0.1:1514"
    tag: "proxy"

运行harbor

docker stack deploy -c harbor2.yml harbor
[root@swarm2 ~]# docker stack ls
NAME    SERVICES
harbor  6

查看运行的services

[root@swarm2 ~]# docker stack services harbor
ID            NAME               MODE        REPLICAS  IMAGE
eyipo6gng5su  harbor_jobservice  replicated  1/1       vmware/harbor-jobservice:0.5.0
mzbeq1oqguqd  harbor_registry    replicated  1/1       library/registry:2.5.0
n3sbohiie3x4  harbor_mysql       replicated  1/1       vmware/harbor-db:0.5.0
prf0jhe0j31x  harbor_ui          replicated  1/1       vmware/harbor-ui:0.5.0
qcmxi1g8x16f  harbor_proxy       replicated  1/1       nginx:1.11.5
ss29zpgdmlut  harbor_log         replicated  1/1       vmware/harbor-log:0.5.0

配置harbor的自签名证书

创建自签名ca证书

openssl req \
  -newkey rsa:4096 -nodes -sha256 -keyout ca.key \
  -x509 -days 365 -out ca.crt
[root@swarm2 ca]# openssl req \
>   -newkey rsa:4096 -nodes -sha256 -keyout ca.key \
>   -x509 -days 365 -out ca.crt
Generating a 4096 bit RSA private key
.....................................................................................................................................................................................................................................................................................................................................................++
........................................................................++
writing new private key to 'ca.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:cn
State or Province Name (full name) []:Beijing
Locality Name (eg, city) [Default City]:Chaoyang
Organization Name (eg, company) [Default Company Ltd]:Goluk
Organizational Unit Name (eg, section) []:
Common Name (eg, your name or your server's hostname) []:reg.goluk.cn
Email Address []:[e-mail]
[root@swarm2 ca]# ls
ca.crt  ca.key

生成证书签名请求

openssl req \
  -newkey rsa:4096 -nodes -keyout reg.goluk.cn.key \
  -out reg.goluk.cn.csr

[root@swarm2 ca]# openssl req -newkey rsa:4096 -nodes  -sha256 -keyout reg.goluk.cn.key -out reg.goluk.cn.csr
Generating a 4096 bit RSA private key
................................................................................................................++
..................................................................................................................................................................++
writing new private key to 'reg.goluk.cn.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:cn
State or Province Name (full name) []:Beijing
Locality Name (eg, city) [Default City]:Chaoyang
Organization Name (eg, company) [Default Company Ltd]:Goluk
Organizational Unit Name (eg, section) []:
Common Name (eg, your name or your server's hostname) []:reg.goluk.cn
Email Address []:[e-mail]

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
[root@swarm2 ca]# ls -al
总用量 11
drwxr-xr-x 1 root root    4 2月  13 16:00 .
drwxr-xr-x 1 root root   15 2月  13 15:54 ..
-rw-r--r-- 1 root root 2037 2月  13 15:55 ca.crt
-rw-r--r-- 1 root root 3272 2月  13 15:55 ca.key
-rw-r--r-- 1 root root 1708 2月  13 16:00 reg.goluk.cn.csr
-rw-r--r-- 1 root root 3272 2月  13 16:00 reg.goluk.cn.key

生成私有仓库主机的证书

openssl x509 -req -days 365 -in reg.goluk.cn.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out reg.goluk.cn.crt
[root@swarm2 ca]# openssl x509 -req -days 365 -in reg.goluk.cn.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out reg.goluk.cn.crt
Signature ok
subject=/C=cn/ST=Beijing/L=Chaoyang/O=Goluk/CN=reg/ emailAddress=[e-mail]
Getting CA Private Key

配置证书到harbor目录

cp reg.goluk.cn.crt /mnt/cephfs/app/harbor/data/cert
cp reg.goluk.cn.key /mnt/cephfs/app/harbor/data/cert

修改harbor.cfg

ui_url_protocol = https
crt_country = cn                            ## 此行至crt_email
行需要和上面自签名证书过程中填写值保持一致
crt_state = Beijing
crt_location = cn
crt_organization = Goluk
crt_organizationalunit =
crt_commonname = reg.goluk.cn
crt_email = [e-mail]         
ssl_cert = /mnt/cephfs/app/harbor/data/cert/reg.goluk.cn.crt  ## 填写生成证书的文件及路径
ssl_cert_key = /mnt/cephfs/app/harbor/data/cert/reg.goluk.cn.key ##同上

运行 prepare.sh

./prepare.sh

确定停止harbor的运行

docker stack rm harbor

重新运行harbor

docker stack deploy -c harbor2.yml

配置每个需要访问harbor的docker主机

将reg.goluk.cn的ca证书放到docker的配置证书的目录下

[root@swarm3 ~]# mkdir -p /etc/docker/certs.d/reg.goluk.cn
[root@swarm3 ~]# cp /mnt/cephfs/app/harbor/harbor/ca/ca.crt /etc/docker/certs.d/reg.goluk.cn/

配置docker主机的系统级别信任自签名证书

[root@swarm3 ~]#cp /mnt/cephfs/app/harbor/data/cert/reg.goluk.cn.crt /etc/pki/ca-trust/source/anchors/
update-ca-trust  ## 更新信任证书列表

docker login 测试

docker login reg.goluk.cn

登录harbor ui 建立一个base 仓库，测试push镜像

docker images 列表

[root@swarm3 ~]# docker images
REPOSITORY                     TAG                 IMAGE ID            CREATED             SIZE
reg.goluk.cn/base/harbor-log   latest              eebc987a891b        2 months ago        190 MB
vmware/harbor-log              0.5.0               eebc987a891b        2 months ago        190 MB
vmware/harbor-jobservice       0.5.0               995368e96860        2 months ago        169 MB
vmware/harbor-ui               0.5.0               232a8664541a        2 months ago        233 MB
vmware/harbor-db               0.5.0               84c4ce8e9b6c        2 months ago        327 MB
nginx                          1.11.5              05a60462f8ba        3 months ago        181 MB
registry                       2.5.0               c6c14b3960bd        6 months ago        33.3 MB

docker tag

[root@swarm3 ~]# docker tag eebc987a891b reg.goluk.cn/base/harbor-log
[root@swarm3 ~]# docker tag 232a8664541a reg.goluk.cn/base/harbor-ui
[root@swarm3 ~]# docker tag 995368e96860 reg.goluk.cn/base/harbor-jobservice
[root@swarm3 ~]# docker tag 84c4ce8e9b6c reg.goluk.cn/base/harbor/db
[root@swarm3 ~]# docker tag 05a60462f8ba reg.goluk.cn/base/nginx
[root@swarm3 ~]# docker tag c6c14b3960bd reg.goluk.cn/base/registry

docker push

[root@swarm3 ~]# docker push reg.goluk.cn/base/registry
The push refers to a repository [reg.goluk.cn/base/registry]
3bb5bc5ad373: Pushed
35039a507f7a: Pushed
d00444e19d65: Pushed
aa3a31ee27f3: Pushed
4fe15f8d0ae6: Pushed
latest: digest: sha256:04cc36f8f72c4272f07325075586b3a0a73db23d3822a7ed1ce34f86f3f410c3 size: 1363
......

登录harbor ui 可以看到已经push的镜像，至此设置过程完成