Appearance

自有测试平台LunarLink部署生产环境实践

geekbing2024-03-02自动化测试测试平台

原理

利用 Jenkins 拉取 GitLab 仓库代码 --> 构建 Docker 镜像 --> 上传 Docker 镜像到自建的 Harbor 镜像仓库 --> 利用 sh 脚本生成 docker-compose.yml --> 将 docker-compose.yml 推送到部署服务器 --> 部署服务器再执行docker-compose up -d命令部署

准备工作

  • 3 台服务器,1 台安装 Docker + Jenkins + Git + Harbor 做持续集成,参照open in new window

  • 1 台部署前端服务(Vue2、Nginx)、后端服务(Django 、定时/异步任务、录制流量代理),需要先安装 Docker

  • 1 台部署 MySQL 数据库&&中间件 RabbitMQ、Redis,需要先安装 Dockeropen in new window

TIP

如果平台访问人数不多,服务器配置足够,可以将 3 台服务器缩减至 1 台,所有这些操作均在 1 台服务器上面执行

操作步骤

Docker

根目录下的build_and_push.sh脚本构建 Docker 镜像并推送到 Harbor 镜像仓库

#!/bin/bash
DOMAIN=$1
VERSION="1.0.0"

# Build the Docker image
docker build -t lunar-link-"$DOMAIN" -f ./deployment/"$DOMAIN"/Dockerfile .

# Tag the Docker image
docker tag lunar-link-"$DOMAIN" ip:port/lunar_link/"$DOMAIN":"$VERSION" # ip:port 为 Harbor 镜像仓库地址

# Login to the Docker registry
docker login -u 账号 -p 密码 ip:port  # 登录镜像仓库

# Push the Docker image
docker push ip:port/lunar_link/"$DOMAIN":"$VERSION"  # 推送到镜像仓库

根目录下的create_compose_file.sh创建 docker-compose.yml 文件脚本

#!/bin/bash

DJANGO_IMAGE_TAG="1.0.0"
WEB_IMAGE_TAG="1.0.0"
CELERY_IMAGE_TAG="1.0.0"
PROXY_IMAGE_TAG="1.0.0"

cat > docker-compose.yml <<EOF
version: "3"
services:
  lunar-link-django:
    image: ip:port/lunar_link/django:${DJANGO_IMAGE_TAG}  # ip:port 为 Harbor 镜像仓库地址
    container_name: lunar-link-django
    working_dir: /backend
    environment:
      PYTHONUNBUFFERED: 1
      TZ: Asia/Shanghai
    expose:
      - 8000
    restart: always

  lunar-link-web:
    image: ip:port/lunar_link/web:${WEB_IMAGE_TAG}
    container_name: lunar-link-web
    depends_on:
      - lunar-link-django
    ports:
      - "8081:8081"
    expose:
      - 8081
    environment:
      TZ: Asia/Shanghai
    restart: always

  lunar-link-celery:
    image: ip:port/lunar_link/celery:${CELERY_IMAGE_TAG}
    container_name: lunar-link-celery
    depends_on:
      - lunar-link-django
    environment:
      PYTHONUNBUFFERED: 1
      TZ: Asia/Shanghai
    restart: always

  lunar-link-proxy:
    image: ip:port/lunar_link/proxy:${PROXY_IMAGE_TAG}
    container_name: lunar-link-proxy
    depends_on:
      - lunar-link-django
    environment:
      PYTHONUNBUFFERED: 1
      TZ: Asia/Shanghai
    ports:
      - "7778:7778"
    expose:
      - 7778
    restart: always
EOF

Jenkins

  • 安装 Publish over SSH 插件,用于连接部署服务器并推送 docker-compose.yml 文件

    deploy2

  • 在 Jenkins 所在的机器上生成秘钥,一路按下 Enter 采用默认值

    # 生成秘钥,一路按回车键
    ssh-keygen
    
    # 查看公钥,验证是否创建成功
    cat ~/.ssh/id_rsa.pub
    
  • 将 Jenkins 所在服务器的公钥拷贝到远服务器,如当前远程服务器的 ip 为 192.168.3.84

    ssh-copy-id 192.168.3.84
    

    deploy4

  • 进入 Manage Jenkins --> System 拉到底部的Publish over SSH 区域填入 Jenkins 所在服务器的私钥

    # 查看私钥
    cat id_rsa
    

    deploy3

  • 配置 SSH Servers 连接远程服务器

    deploy5

  • 选择 New Item 创建 Pipeline ,名称为 LunarLink。

  • 进入Configure,勾选 This project is parameterized,选择 Boolean Parameter

    deploy1

  • 进入Pipeline Syntax 生成连接远程服务器的Pipeline Script

    deploy6

    deploy7

    deploy8

  • 配置Pipeline Script。

    deploy9

    // Pipeline script 详细代码如下
    pipeline {
        agent {
            label 'slave3'  // Jenkins 节点 slave3 执行构建任务
        }
        
        parameters {
            booleanParam(defaultValue: true, description: 'Do you want to deploy lunar-link-proxy?', name: 'DEPLOY_PROXY')
        }
    
        stages {
            stage('从gitlab中拉取代码') {
                steps {
                    git credentialsId: 'git凭证', url: '代码仓库地址'  // 代码仓库地址为http开头
                }
            }
            stage('打包构建docker镜像&&镜像打标签并上传') {
                steps {
                    sh 'chmod +x ./build_and_push.sh'
                    script {
                        if (params.DEPLOY_PROXY) {
                            sh './build_and_push.sh proxy'  // 根据 DEPLOY_PROXY 变量判断是否构建代理Docker镜像
                        }
                    }
                    sh './build_and_push.sh web'
                    sh './build_and_push.sh django'
                    sh './build_and_push.sh celery'
                }
            }
            stage('创建docker-compose.yml') {
                steps {
                    sh 'chmod +x ./create_compose_file.sh'
                    sh './create_compose_file.sh'
                }
            }  
            stage('远程部署应用') {
                steps {
                    script {
                        def deployActions = ""
    
                        if (params.DEPLOY_PROXY) {  
                            deployActions = "sudo docker-compose down && sudo docker-compose pull && sudo docker-compose up -d"
                        } else {
                            deployActions = "sudo docker-compose stop lunar-link-django lunar-link-web lunar-link-celery && " +
                                            "sudo docker-compose pull lunar-link-django lunar-link-web lunar-link-celery && " +
                                            "sudo docker-compose up -d lunar-link-django lunar-link-web lunar-link-celery"
                        }
    
                        // 命令组合
                        def execCommand = "cd /home/jdkapp/tester && ${deployActions}"
                       // 推送 docker-compose.yml 到部署服务器并执行 execCommand 命令
                        sshPublisher(publishers: [sshPublisherDesc(configName: 'product_server',
                        transfers: [sshTransfer(cleanRemote: false, excludes: '',
                        execCommand: execCommand,
                        execTimeout: 120000, flatten: false, makeEmptyDirs: false, noDefaultExcludes: false,
                        patternSeparator: '[, ]+', remoteDirectory: '', remoteDirectorySDF: false,
                        removePrefix: '', sourceFiles: 'docker-compose.yml')], usePromotionTimestamp: false,
                        useWorkspaceInPromotion: false, verbose: false)])
                    }
                }
            }
            stage('企业微信通知') {
                steps {
                    script {
                        wrap([$class: 'BuildUser']){
                            echo "full name is $BUILD_USER"
                            echo "user id is $BUILD_USER_ID"
                            echo "user email is ${env.BUILD_USER_EMAIL}"
                            env.BUILD_USER = "${BUILD_USER}"
                        }
                    }                
                }            
            }  
        }
        post {  // 构建成功失败均发送企微同志
            success {
                script {
                    WeiXinSuccess()
                }            
            }
            failure {
                script {
                    WeiXinFailure()
                }
            }
            unstable{
                script {
                    WeiXinSuccess()
                }             
            }
        }
    }
    
    def WeiXinSuccess(){
            sh """
                curl --location --request POST 'https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=企微机器人Webhook地址' \
                    --header 'Content-Type: application/json' \
                    --data '{
                        "msgtype": "markdown",
                        "markdown": {
                            "content": "<font color=\'warning\'>**Jenkins任务通知**</font> \n
                            >构建人: <font color=\'comment\'>${env.BUILD_USER}</font>
                            >构建时间: <font color=\'comment\'>${BUILD_TIMESTAMP}</font>
                            >运行时长: <font color=\'comment\'>${currentBuild.durationString}</font>
                            >任务名称: <font color=\'comment\'>${JOB_BASE_NAME}</font>
                            >构建日志: <font color=\'comment\'>[点击查看](${BUILD_URL}/console)</font>
                            >构建状态: <font color=\'info\'>**Success**</font>"                                     
                        }
                    }'
            """
    }
    
    def WeiXinFailure(){
            sh """
                curl --location --request POST 'https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=企微机器人Webhook地址' \
                    --header 'Content-Type: application/json' \
                    --data '{
                        "msgtype": "markdown",
                        "markdown": {
                            "content": "<font color=\'warning\'>**Jenkins任务通知**</font> \n
                            >构建人: <font color=\'comment\'>${env.BUILD_USER}</font>
                            >构建时间: <font color=\'comment\'>${BUILD_TIMESTAMP}</font>
                            >运行时长: <font color=\'comment\'>${currentBuild.durationString}</font>
                            >任务名称: <font color=\'comment\'>${JOB_BASE_NAME}</font>
                            >构建日志: <font color=\'comment\'>[点击查看](${BUILD_URL}/console)</font>
                            >构建状态: <font color=\'comment\'>**Failure**</font>"      
                        }
                    }'
            """
    }
    

Nginx

LunarLink/deployment/web/nginx.conf

server {
    listen 8081;
    server_name 192.168.3.84;  # 替换成要部署的服务器ip
    client_max_body_size 100M;

    # 开启gzip
    gzip on;
    # https://blog.csdn.net/fxss5201/article/details/106535475
    gzip_static on;
    gzip_proxied any;
    # 低于1kb的资源不压缩
    gzip_min_length 1k;
    gzip_buffers 4 16k;
    gzip_comp_level 6;
    # 需要压缩的类型
    gzip_types text/plain application/javascript text/css application/xml text/javascript application/json;
    # 是否添加“Vary: Accept-Encoding”响应头
    gzip_vary on;

    location /django_static/ {
        alias /www/LunarLink/static/;
    }

    location /api/ {
      # 接口转发
        proxy_pass http://lunar-link-django:8000/api/;
        proxy_set_header Host $host;
        proxy_set_header Cookie $http_cookie;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_redirect default;
        add_header Access-Control-Allow-Origin *;
        add_header Access-Control-Allow-Headers X-Requested-With;
        add_header Access-Control-Allow-Methods GET,POST,OPTIONS;
    }

    location /admin/ {
        proxy_pass http://lunar-link-django:8000/admin/;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $host;
        proxy_redirect off;
    }

    location / {
        root /usr/share/nginx/html;
        index  index.html index.htm;
        try_files $uri $uri/ /index.html;
    }
}

前端

LunarLink/frontend/config/prod.env.js

"use strict";
// 生产环境的环境变量配置
const LunarLink = process.env.LUNAR_LINK || "LunarLink";
module.exports = {
    NODE_ENV: '"production"',
    DOCS_URL: '"http://192.168.3.84:8888/docs/guide/introduce.html"',  // 测试平台指南
    LUNAR_LINK: "'" + LunarLink + "'",
    RECORD_CASE_DOCS_URL:
        '"http://192.168.3.84:8888/docs/guide/test_case.html"',  // 测试平台操作手册
    VUE_APP_BASE_URL: '""'  // 接口地址 Nginx 会自动转发到后端,保持为空即可
};

后端

LunarLink/backend/conf/docker.py

# -*- coding: utf-8 -*-
"""
@File    : docker.py
@Time    : 
@Author  : 
"""
import sentry_sdk
from sentry_sdk.integrations.django import DjangoIntegration

# ================================================= #
# ************** mysql数据库 配置  ************** #
# ================================================= #
# 数据库地址
DATABASE_HOST = ""
# 数据库端口
DATABASE_PORT = 3308
# 数据库用户名
DATABASE_USER = ""
# 数据库密码
DATABASE_PASSWORD = ""
# 数据库名
DATABASE_NAME = "lunar_prod"

# ================================================= #
# ************** RabbitMQ配置 ************** #
# ================================================= #
MQ_USER = "guest"
MQ_PASSWORD = ""
MQ_HOST = ""
MQ_URL = f"amqp://{MQ_USER}:{MQ_PASSWORD}@{MQ_HOST}:5672//"

# ================================================= #
# ************** Redis配置 ************** #
# ================================================= #
REDIS_ON = True
REDIS_HOST = ""
REDIS_PASSWORD = ""
REDIS_PORT = 6379
REDIS_DB = 0

# ================================================= #
# ************** Sentry监控配置  ************** #
# ================================================= #
sentry_sdk.init(
    dsn="",  # Sentry监控DSN,没有可不填
    integrations=[DjangoIntegration()],
    # Set traces_sample_rate to 1.0 to capture 100%
    # of transactions for performance monitoring.
    # We recommend adjusting this value in production.
    traces_sample_rate=1.0,
    # If you wish to associate users to errors (assuming you are using
    # django.contrib.auth) you may enable sending PII data.
    send_default_pii=True,
)

# ================================================= #
# ************** 其他 配置  ************** #
# ================================================= #
DEBUG = False  # 线上环境请设置为False
# 启动登录日志记录(通过调用api获取ip详细地址。如果是内网,关闭即可)
ENABLE_LOGIN_ANALYSIS_LOG = True
ALLOWED_HOSTS = ["*"]
BASE_REPORT_URL = "http://47.119.28.171:8081/api/lunarlink/reports"  # 替换成部署的服务器地址或域名
IM_REPORT_SETTING = {
    "base_url": "http://47.119.28.171",
    "port": 8081,
    "report_title": "自动化测试报告",
}

# 设置请求体的最大大小
DATA_UPLOAD_MAX_MEMORY_SIZE = 52428800  # 50M

# ================================================= #
# ************** 监控告警企微机器人配置  ************** #
# ================================================= #
QY_WEB_HOOK = ""

# ================================================= #
# ************** 发送邮件配置  ************** #
# ================================================= #
# 使用 SMTP 服务器发送邮件
EMAIL_BACKEND = "django.core.mail.backends.smtp.EmailBackend"
# SMTP 服务器地址
EMAIL_HOST = "smtphz.qiye.163.com"
# SMTP 服务器端口
EMAIL_PORT = 465
# 发件人邮箱账号
EMAIL_HOST_USER = ""
DEFAULT_FROM_EMAIL = ""
# 发件人邮箱密码
EMAIL_HOST_PASSWORD = ""
# 是否使用 TLS
EMAIL_USE_TLS = False
# 是否使用 SSL
EMAIL_USE_SSL = True

# ================================================= #
# ************** 录制流量代理配置  ************** #
# ================================================= #
# PROXY Server
PROXY_ON = True  # 是否开启代理
PROXY_PORT = 7778

Jenkins构建部署

deploy10

deploy11

上次更新 3/25/2024, 10:43:07 AM