- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] sparrow/install-server: use "start-only" instead of "run-only"
by Liu Yinsi 09 Mar '21

09 Mar '21

[why] when locally deploy compass-ci server, no need to check docker images exist, just start container. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/install-server | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/install-server b/sparrow/install-server index c532956..f10040a 100755 --- a/sparrow/install-server +++ b/sparrow/install-server @@ -16,5 +16,5 @@ cd $CCI_SRC/sparrow || exit 2-network/cifs 3-code/dev-env . /etc/profile.d/compass.sh -4-docker/buildall run-only +4-docker/buildall start-only 7-systemd/systemd-setup -- 2.23.0

1 0

[PATCH compass-ci] sparrow/README.md: delete unnecessary steps
by Liu Yinsi 09 Mar '21

09 Mar '21

1. some steps already finish in sparrow/install-tiny, so delete unnecessary steps. 2. submit iperf.yaml, the default os_mount is container, so change the contents printed by the terminal of the command 3. add FAQ Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/README.md | 64 ++++++++++++++++++++--------------------------- 1 file changed, 27 insertions(+), 37 deletions(-) diff --git a/sparrow/README.md b/sparrow/README.md index 703e8f9..d7b8e81 100644 --- a/sparrow/README.md +++ b/sparrow/README.md @@ -29,26 +29,28 @@ 1. 登录 openEuler 系统 -2. 配置git账号 +2. 设置文件权限 ```bash - git config --global user.name "xxx" - git config --global user.email "xxx(a)xxx.com" - ``` - -3. 设置文件权限并关闭SELINUX - ```bash - umask 002 && setenforce 0 + umask 002 ``` >**说明：** - >setenforce 0 只是暂时禁用SELINUX，需要修改/etc/selinux/config中的SELINUX=enforcing改为SELINUX=permissive或disabled才能长期有效 - >umask 002 只是暂时设置umask值，需要修改/etc/bashrc中的umask值为002才能长期有效 + >umask 002 只是暂时设置umask值，需要修改/etc/bashrc中的umask值为002才能长期有效 + >执行下个步骤前请确保当前umask值为002，否则因文件权限问题将导致部分服务无法正常运行。 -4. 创建工作目录并克隆 compass-ci 项目代码 +3. 创建工作目录并克隆 compass-ci 项目代码 ```bash mkdir /c/ && ln -s /c/compass-ci /c/cci git clone https://gitee.com/wu_fengguang/compass-ci.git ``` + +4. 编辑setup.yaml配置用户名和邮箱 + ```bash + vi /c/compass-ci/sparrow/setup.yaml + ``` + + >**说明：** + >只需填写my_name, my_email, 且my_name, my_email冒号后面必须有1个空格。 5. 执行一键部署脚本 install-tiny ```bash @@ -57,38 +59,18 @@ #### 提交测试任务前的准备 -1. 生成lkp-aarch64.cgz压缩包 - ```bash - cd /c/compass-ci/container/lkp-initrd && ./run - ``` -2. 验证账号 +1. 测试环境是否可以提交job测试 ```bash - cd /c/compass-ci/sbin && ./build-my-info.rb - ``` - -3. 测试环境是否可以提交job测试 - ```bash - submit iperf.yaml testbox=vm-2p8g + submit iperf.yaml ``` 执行上述命令正常情况下会提示信息如下: ``` - submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory - submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory - ``` - compass-ci搭建完毕，执行步骤4下载所需要的rootfs文件就可以开始进行测试了。 - -4. 下载rootfs文件（根据所需要的rootfs在[该目录](http://124.160.11.58:11300/os/)下获取对应版本的cgz文件） - ```bash - mkdir -p /srv/os/openeuler/aarch64/20.03 - cd /srv/os/openeuler/aarch64/20.03 - wget http://api.compass-ci.openeuler.org:11300/os/openeuler/aarch64/20.03.cgz - ``` - -5. 解压rootfs cgz 文件 - ```bash - gzip -dc 20.03.cgz | cpio -idv + submit_id=bf5e7ad7-839d-48ec-a033-23281323c750 + submit /c/lkp-tests/jobs/iperf.yaml, got job id=nolab.1 + submit /c/lkp-tests/jobs/iperf.yaml, got job id=nolab.1 ``` + compass-ci搭建完毕，下面就可以开始进行测试了。 #### 提交测试任务到本地compass-ci 本文以测试用例iperf.yaml为例 @@ -117,3 +99,11 @@ >**说明：** >[登陆测试环境调测任务](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… + +# FAQ + +* 选择下载os rootfs + + 启动qemu测试机需要[os rootfs文件](http://api.compass-ci.openeuler.org:11304/os/)， + 一键部署默认下载os rootfs为openeuler aarch64 20.03，当部署完毕后，可在该目录下查看: /srv/os/openeuler/aarch64/20.03 + 如需要其他os rootfs，可使用该脚本下载: /c/compass-ci/sbin/download-rootfs -- 2.23.0

1 0

[PATCH compass-ci] doc/manual: add chinese document
by Luan Shengde 09 Mar '21

09 Mar '21

Add chinese document for submit container. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/manual/build-lkp-tests-container.zh.md | 63 ++++++++++++++++++++++ 1 file changed, 63 insertions(+) create mode 100644 doc/manual/build-lkp-tests-container.zh.md diff --git a/doc/manual/build-lkp-tests-container.zh.md b/doc/manual/build-lkp-tests-container.zh.md new file mode 100644 index 0000000..10a1611 --- /dev/null +++ b/doc/manual/build-lkp-tests-container.zh.md @@ -0,0 +1,63 @@ +# 前言 + +为了适应在各个版本的Linux操作系统上运行，我们提供了lkp-tests的容器化解决方案。 +你无需在您的本地服务上安装lkp-tests，也将避免缺乏依赖包而导致的安装失败。 + +# 前置准备 + + 请在您的本地服务器上完成下面步骤： + + - 安装docker + - 申请账号并配置默认的yaml文件 + - 生成本地的公私钥 + +# 构建容器 + +## 1. 下载资源 + + 本地下载lkp-tests和compass-ci + + 下载资源： + + git clone https://gitee.com/wu_fengguang/compass-ci.git + git clone https://gitee.com/wu_fengguang/lkp-tests.git + +## 2. 添加环境变量 + + 配置环境变量： + + echo "export LKP_SRC=$PWD/lkp-tests" >> ~/.${SHELL##*/}rc + echo "export CCI_SRC=$PWD/compass-ci" >> ~/.${SHELL##*/}rc + source ~/.${SHELL##*/}rc + +## 3. 构建镜像 + + 镜像构建： + + cd compass-ci/container/submit + ./build + +## 4. 添加可执行文件 + + 创建可执行文件： + + ln -s $CCI_SRC/container/submit/submit /usr/bin/submit + +# 开始第一个任务 + + 说明： + + 和在你的本地服务器上安装lkp-tests一样，您可以直接使用‘submit’命令提交任务。 + 每一个任务都会生成一个一次性的容器来执行。 + 可执行文件submit会将您下载的lkp-tests挂载到容器内的lkp-tests路径上，这样您在本地编辑lkp-tests文件，在容器内也会生效。 + + + + 提交任务命令示例： + + + submit -c -m testbox=vm-2p8g borrow-1h.yaml + + submit命令： + + 了解更多submit命令的使用方法，请参考：[submit user manual](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/su… -- 2.23.0

1 0

[PATCH compass-ci] scheduler/pp_params.cr: fix get pp_params
by Wei Jihui 09 Mar '21

09 Mar '21

[error] if pp params only have one value and this value's length is bigger than 40, pp_params will be empty. [how] add all values to pp_params. if one value's length + pp_params's length bigger then 40, then cut value. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- src/scheduler/pp_params.cr | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/scheduler/pp_params.cr b/src/scheduler/pp_params.cr index 03bf0b8..dc24938 100644 --- a/src/scheduler/pp_params.cr +++ b/src/scheduler/pp_params.cr @@ -37,7 +37,8 @@ class Job value = format_string(hash[key].to_s) next if 0 == value.size - if 40 > value.size + items_size + values.size + if items_size < 40 + value = value[0...(40-items_size)] if items_size + value.size > 40 values << value items_size += value.size else -- 2.23.0

2 1

[PATCH v2 compass-ci] /src/extract-stats: rescue when the task-queue disconnect
by Lu Weitao 09 Mar '21

09 Mar '21

[Why] the connection between stats_worker and task-queue may be interrupted, and the old connection will be disable, so stats_worker need create a new connection with task-queue Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- src/extract-stats/stats_worker.cr | 3 +++ 1 file changed, 3 insertions(+) diff --git a/src/extract-stats/stats_worker.cr b/src/extract-stats/stats_worker.cr index 836d1f8..dbb3fe2 100644 --- a/src/extract-stats/stats_worker.cr +++ b/src/extract-stats/stats_worker.cr @@ -27,8 +27,11 @@ class StatsWorker end rescue e STDERR.puts e.message + error_message = e.message + # incase of many error message when task-queue, ES does not work sleep(10) + @tq = TaskQueueAPI.new if error_message && error_message.includes?("3060': Connection refused") end end end -- 2.23.0

1 0

[PATCH compass-ci] container/rabbitmq: support data persistence to hard disks
by Wu Zhende 09 Mar '21

09 Mar '21

If data persistence is not performed, all data in the queue will be cleared when the service is restarted. However, some queues need to be restarted to restore data. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/rabbitmq/start | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/container/rabbitmq/start b/container/rabbitmq/start index 6b875f3..cdf5dee 100755 --- a/container/rabbitmq/start +++ b/container/rabbitmq/start @@ -4,16 +4,21 @@ . $CCI_SRC/container/defconfig.sh -docker_rm rabbitmq +docker_name=rabbitmq + +docker_rm $docker_name cmd=( docker run --restart=always -d - --name rabbitmq + --name $docker_name + --hostname $docker_name + -u 1090 -p 5672:5672 -p 15672:15672 -v /etc/localtime:/etc/localtime:ro + -v /srv/rabbitmq:/var/lib/rabbitmq rabbitmq:3-management ) -- 2.23.0

1 0

[PATCH compass-ci] container/dracut-initrd: judge local before handle 90lkp
by Xu Xijian 09 Mar '21

09 Mar '21

[why] 90lkp is only used for local mount to replace 90overlay-root, don't use 90lkp with other mount, the errlog occured with cifs. [errlog] Starting dracut pre-pivot and cleanup hook... [ 7.812554] dracut-pre-pivot[626]: cp: cannot create directory '/sysroot/lkp': Read-only file system [ 7.816968] dracut-pre-pivot[626]: cp: cannot create symbolic link '/sysroot/etc/X11/X': Read-only file system Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/modules.d/90lkp/lkp-deploy.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/container/dracut-initrd/modules.d/90lkp/lkp-deploy.sh b/container/dracut-initrd/modules.d/90lkp/lkp-deploy.sh index 8be927c..44f994e 100755 --- a/container/dracut-initrd/modules.d/90lkp/lkp-deploy.sh +++ b/container/dracut-initrd/modules.d/90lkp/lkp-deploy.sh @@ -3,6 +3,10 @@ . /lib/dracut-lib.sh +if ! getargbool 0 local; then + return +fi + # transfer LKP dirs [ -d /lkp ] || return 0 -- 2.23.0

1 0

[PATCH compass-ci] lib/git_mirror.rb: github download workaround
by Li Yuanchao 09 Mar '21

09 Mar '21

There is a problem to download from github by http or https. fatal: unable to access 'https://github.com/liyc-github/AvxToNeon/': Failed to connect to github.com port 443: Connection timed out When it occurs, change url to "git://xxx". Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 53c9e59..1fa8c9e 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -42,16 +42,20 @@ class GitMirror def git_clone(url, mirror_dir) url = get_url(Array(url)[0]) 10.times do - %x(git clone --mirror --depth 1 #{url} #{mirror_dir} 2>&1) + stderr = %x(git clone --mirror --depth 1 #{url} #{mirror_dir} 2>&1) return true if File.directory?(mirror_dir) && File.exist?("#{mirror_dir}/config") + + if stderr.include?('Failed to connect to github.com port 443') + url = "git://#{url.split('://')[1]}" + end end return false end def git_fetch(mirror_dir) if File.exist?("#{mirror_dir}/shallow") + FileUtils.rm("#{mirror_dir}/shallow.lock") if File.exist?("#{mirror_dir}/shallow.lock") %x(git -C #{mirror_dir} fetch --unshallow 2>&1) - return false end fetch_info = %x(git -C #{mirror_dir} fetch 2>&1) @@ -59,6 +63,11 @@ class GitMirror if fetch_info.include?(ERR_MESSAGE) && Dir.empty?(mirror_dir) FileUtils.rmdir(mirror_dir) end + + if fetch_info.include?('Failed to connect to github.com port 443') + url = %x(git -C #{mirror_dir} ls-remote --get-url origin).chomp + %x(git -C #{mirror_dir} remote set-url origin git://#{url.split('://')[1]}) + end return fetch_info.include? '->' end @@ -66,7 +75,7 @@ class GitMirror url = get_url(Array(url)[0]) git_url = %x(git -C #{mirror_dir} ls-remote --get-url origin).chomp - return true if url == git_url + return true if git_url == url || git_url.include?(url.split('://')[1]) return false end -- 2.23.0

2 1

[PATCH lkp-tests] sbin/submit: fix input docker_image value by cmdline
by Wei Jihui 09 Mar '21

09 Mar '21

[error] input: submit docker_image=centos:7 job.yaml output: docker_image: centos [why] submit split centos:7 to [centos, 7] [how] -s "docker_image: centos:7" ==> docker_image: centos:7 docker_image=centos:7 ==> docker_image: centos:7 Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- sbin/submit | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/sbin/submit b/sbin/submit index e5e41828..feb65d00 100755 --- a/sbin/submit +++ b/sbin/submit @@ -28,7 +28,7 @@ options = OptionParser.new do |opts| opts.separator 'options:' opts.on("-s 'KEY: VALUE'", "--set 'KEY: VALUE'", 'add YAML hash to job') do |key_value| - k, v = key_value.sub(' ', '').split(':') + k, v = key_value.sub(' ', '').split(':', 2) opt_set_key_value[k] = v end @@ -59,7 +59,7 @@ options = OptionParser.new do |opts| opts.on('-m', '--monitor', "monitor job status: use -m 'KEY: VALUE' to add rule") do opt_monitor = true - k, v = ARGV[0].sub(' ', '').split(':') if ARGV[0] + k, v = ARGV[0].sub(' ', '').split(':', 2) if ARGV[0] if (k && !k.empty?) && (v && !v.empty?) opt_monitor_query[k] = v ARGV.shift @@ -78,7 +78,7 @@ options.parse!(ARGV) seen_job_yaml = false ARGV.delete_if do |arg| if arg.index '=' - k, v = arg.sub(' ', '').sub(/=/, ':').split(':') + k, v = arg.sub(' ', '').split('=', 2) if seen_job_yaml opt_set_key_value[k] = v else -- 2.23.0

1 0

[PATCH compass-ci 2/2] sparrow/install-tiny: call 5-build/register-account
by Liu Yinsi 09 Mar '21

09 Mar '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/install-tiny | 1 + 1 file changed, 1 insertion(+) diff --git a/sparrow/install-tiny b/sparrow/install-tiny index 9e94e39..e48bd25 100755 --- a/sparrow/install-tiny +++ b/sparrow/install-tiny @@ -21,4 +21,5 @@ cd $CCI_SRC/sparrow || exit 4-docker/buildall 5-build/ipxe 5-build/os-ready +5-build/register-account 7-systemd/systemd-setup -- 2.23.0

1 0