- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] container/assist-result: add check_job_credible function
by Cao Xueliang 21 Jan '21

21 Jan '21

The container assist-result mount "/srv/result" dir, the function credible? need "/srv/result" dir, so it can be implemented in the container. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assist-result/Dockerfile | 2 +- container/assist-result/routes.rb | 12 ++++++++++++ .../assist-result/views/check_job_credible.rb | 9 +++++++++ lib/assist_result_client.rb | 15 +++++++++++++++ 4 files changed, 37 insertions(+), 1 deletion(-) create mode 100644 container/assist-result/views/check_job_credible.rb diff --git a/container/assist-result/Dockerfile b/container/assist-result/Dockerfile index 47ef92f..60a2fbf 100644 --- a/container/assist-result/Dockerfile +++ b/container/assist-result/Dockerfile @@ -12,4 +12,4 @@ RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweiclou RUN umask 002 && \ echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ - gem install rest-client activesupport json yaml elasticsearch sinatra puma + gem install rest-client activesupport json yaml elasticsearch sinatra puma terminal-table diff --git a/container/assist-result/routes.rb b/container/assist-result/routes.rb index 1013643..077a427 100755 --- a/container/assist-result/routes.rb +++ b/container/assist-result/routes.rb @@ -7,6 +7,7 @@ require 'json' require 'sinatra' require_relative './views/get_job_yaml' +require_relative './views/check_job_credible' configure do set :bind, '0.0.0.0' @@ -22,3 +23,14 @@ get '/get_job_yaml/:job_id' do return [200, result.to_json] end + +post '/check_job_credible' do + begin + data = JSON.parse(Base64.decode64(request.body.read)) + result = check_job_credible(data['pre_job_id'], data['cur_job_id'], data['error_id']) + rescue StandardError => e + return [400, e.backtrace.inspect] + end + + return [200, {'credible' => result}.to_json] +end diff --git a/container/assist-result/views/check_job_credible.rb b/container/assist-result/views/check_job_credible.rb new file mode 100644 index 0000000..a8d387f --- /dev/null +++ b/container/assist-result/views/check_job_credible.rb @@ -0,0 +1,9 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative "#{ENV['CCI_SRC']}/lib/compare_error_messages" + +def check_job_credible(pre_job_id, cur_job_id, error_id) + return credible?(pre_job_id, cur_job_id, error_id) +end diff --git a/lib/assist_result_client.rb b/lib/assist_result_client.rb index 042b189..fcf5d9d 100644 --- a/lib/assist_result_client.rb +++ b/lib/assist_result_client.rb @@ -3,6 +3,8 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true +require 'json' +require 'base64' require 'rest-client' require_relative 'constants' @@ -19,4 +21,17 @@ class AssistResult return response.body end + + def check_job_credible(pre_job_id, cur_job_id, error_id) + data = { + 'pre_job_id' => pre_job_id, + 'cur_job_id' => cur_job_id, + 'error_id' => error_id + } + resource = RestClient::Resource.new("http://#{@host}:#{@port}/check_job_credible") + response = resource.post(Base64.encode64(data.to_json)) + return nil unless response.code == 200 + + return JSON.parse(response.body) + end end -- 2.23.0

1 0

[PATCH compass-ci] /container/dracut-initrd/bin: refactor the code
by Xu Xijian 21 Jan '21

21 Jan '21

1. delete the redundant space before "-d" of "[ -d xx ]" 2. add blank line after "continue" cmd 3. turn some 2 line code block into one line to be more compact Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/bin/overlay-lkp.sh | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/container/dracut-initrd/bin/overlay-lkp.sh b/container/dracut-initrd/bin/overlay-lkp.sh index 1f5819b..9a2800e 100644 --- a/container/dracut-initrd/bin/overlay-lkp.sh +++ b/container/dracut-initrd/bin/overlay-lkp.sh @@ -2,7 +2,7 @@ # SPDX-License-Identifier: GPL-2.0 # transfer LKP dirs -[ -d /lkp ] || return 0 +[ -d /lkp ] || return 0 cp -a /lkp "$NEWROOT"/ @@ -11,8 +11,8 @@ do dir=$(basename "$i") [ "$i" != "${i%/\*}" ] && continue # skip: i='dir/*' - [ -d "$NEWROOT/$dir" ] || - mkdir -p "$NEWROOT/$dir" + + [ -d "$NEWROOT/$dir" ] || mkdir -p "$NEWROOT/$dir" for j in "$i"/* "$i"/.??* do @@ -25,8 +25,7 @@ do subdir=$(basename "$j") - [ -d "$NEWROOT/$dir/$subdir" ] || - mkdir -p "$NEWROOT/$dir/$subdir" + [ -d "$NEWROOT/$dir/$subdir" ] || mkdir -p "$NEWROOT/$dir/$subdir" for k in "$j"/* do @@ -37,7 +36,7 @@ do done done -[ -d /usr/src ] && +[ -d /usr/src ] && cp -a /usr/src "$NEWROOT"/usr/ kmdir=/lib/modules/$(uname -r) -- 2.23.0

2 1

[PATCH compass-ci] sparrow: use nokogiri to parse xml file
by Xiao Shenwei 21 Jan '21

21 Jan '21

[why] nokogiri dependent zlib-devel Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- sparrow/0-package/common | 2 +- sparrow/0-package/os/openEuler | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 2d7c9e3..1eb175c 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -2,7 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -gem install --no-user-install git activesupport rest-client cucumber json faye-websocket elasticsearch +gem install --no-user-install git activesupport rest-client cucumber json faye-websocket elasticsearch nokogiri grep -q '^lkp:' /etc/passwd || useradd -u 1090 lkp grep -q '^mailer:' /etc/passwd || useradd -u 1144 mailer diff --git a/sparrow/0-package/os/openEuler b/sparrow/0-package/os/openEuler index 61a37c4..e815fd1 100755 --- a/sparrow/0-package/os/openEuler +++ b/sparrow/0-package/os/openEuler @@ -24,6 +24,7 @@ pkgs=( libvirt curl sudo + zlib-devel ) yum install -y --skip-broken "${pkgs[@]}" -- 2.23.0

1 0

[PATCH v5 compass-ci] create-image: disable selinux
by Wang Chenglong 21 Jan '21

21 Jan '21

[Error] [ 5.786497] Run /sbin/init as init process SELinux: Could not open policy file <= /etc/selinux/targeted/policy/policy.32: No such file or directory Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/osimage/openeuler/create-image | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/container/osimage/openeuler/create-image b/container/osimage/openeuler/create-image index 631bb14..7f3d982 100755 --- a/container/osimage/openeuler/create-image +++ b/container/osimage/openeuler/create-image @@ -13,6 +13,16 @@ setup_rootfs() ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime rm -f /etc/machine-id /var/lib/dbus/machine-id systemd-machine-id-setup + + echo "Disable selinux" + if [ -f /etc/selinux/config ]; then + sed -i -r 's/^.*SELINUX=enforcing.*$/SELINUX=disabled/g' "/etc/selinux/config" + else + cat > "/etc/selinux/config" <<-EOF + SELINUX=disabled + SELINUXTYPE=targeted + EOF + fi } setup_login() { -- 2.23.0

1 0

[PATCH compass-ci] upload-libvirt/nginx.conf: nginx.conf in upload-libvirt container
by cuiyili 21 Jan '21

21 Jan '21

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/upload-libvirt/nginx.conf | 43 +++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) create mode 100644 container/upload-libvirt/nginx.conf diff --git a/container/upload-libvirt/nginx.conf b/container/upload-libvirt/nginx.conf new file mode 100644 index 0000000..6dea3d3 --- /dev/null +++ b/container/upload-libvirt/nginx.conf @@ -0,0 +1,43 @@ +# For more information on configuration, see: +# * Official English Documentation: http://nginx.org/en/docs/ + +user lkp; +worker_processes auto; +pid /run/nginx.pid; + +events {} + +http { + server { + listen 3081; + server_name "upload-libvirt"; + server_tokens off; + client_max_body_size 50m; + access_log /tmp/access.log; + error_log /tmp/error.log; + + location / { + allow all; + root /srv/cci/libvirt-xml/; + + autoindex on; + create_full_put_path on; + dav_methods PUT MKCOL; + dav_access user:rw group:rw all:rw; + content_by_lua_block { + ngx.req.read_body() + local filename = string.gsub(ngx.var.request_uri, "/", "") + if string.match(filename, "([^/]+).xml$") then + add_path = string.match(filename, "([^/]+).xml$") + local data = ngx.req.get_body_data() + local file = io.open("/srv/cci/libvirt-xml/"..filename, "w") + file:write(data) + file:close() + file_md5 = io.popen("cd /srv/cci/libvirt-xml/ && file_md5=$(md5sum "..filename.." | awk '{print $1}') && mkdir -p "..add_path.." && mv "..filename.." "..add_path.."/${file_md5}.xml && echo ${file_md5}"):read() + ngx.say(add_path.."/"..tostring(file_md5)..".xml") + end + } + } + + } +} -- 2.23.0

2 1

[PATCH v4 compass-ci] create-image: disable selinux
by Wang Chenglong 21 Jan '21

21 Jan '21

[Error] [ 5.786497] Run /sbin/init as init process SELinux: Could not open policy file <= /etc/selinux/targeted/policy/policy.32: No such file or directory Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/osimage/openeuler/create-image | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/container/osimage/openeuler/create-image b/container/osimage/openeuler/create-image index 631bb14..411c130 100755 --- a/container/osimage/openeuler/create-image +++ b/container/osimage/openeuler/create-image @@ -13,6 +13,16 @@ setup_rootfs() ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime rm -f /etc/machine-id /var/lib/dbus/machine-id systemd-machine-id-setup + + echo "Disable selinux" + if [ -f /etc/selinux/config ]; then + sed -i -r 's/^.*SELINUX=enforcing.*$/SELINUX=disabled/g' "/etc/selinux/config" + else + cat > "/etc/selinux/config" <<-EOF + SELINUX=disabled + SELINUXTYPE=targeted + EOF + fi } setup_login() { -- 2.23.0

1 0

[PATCH v3 compass-ci] create-image: disable selinux
by Wang Chenglong 21 Jan '21

21 Jan '21

[Error] [ 5.786497] Run /sbin/init as init process SELinux: Could not open policy file <= /etc/selinux/targeted/policy/policy.32: No such file or directory Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/osimage/openeuler/create-image | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/container/osimage/openeuler/create-image b/container/osimage/openeuler/create-image index 631bb14..e161b17 100755 --- a/container/osimage/openeuler/create-image +++ b/container/osimage/openeuler/create-image @@ -13,6 +13,17 @@ setup_rootfs() ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime rm -f /etc/machine-id /var/lib/dbus/machine-id systemd-machine-id-setup + + # disable selinux + echo "Disable selinux" + if [ -f /etc/selinux/config ]; then + sed -i -r 's/^.*SELINUX=enforcing.*$/SELINUX=disabled/g' "/etc/selinux/config" + else + cat > "/etc/selinux/config" <<-EOF + SELINUX=disabled + SELINUXTYPE=targeted + EOF + fi } setup_login() { -- 2.23.0

1 1

[PATCH v2 lkp-tests] pkg: add PKGBUILD for openeuler docker test
by Bai Jing 20 Jan '21

20 Jan '21

[why] The source file is missing in the openEuler image, So we need to add source files for it. Signed-off-by: Bai Jing <799286817(a)qq.com> --- pkg/openeuler_docker/PKGBUILD | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) create mode 100644 pkg/openeuler_docker/PKGBUILD diff --git a/pkg/openeuler_docker/PKGBUILD b/pkg/openeuler_docker/PKGBUILD new file mode 100644 index 000000000..6a1190718 --- /dev/null +++ b/pkg/openeuler_docker/PKGBUILD @@ -0,0 +1,27 @@ +pkgname=openeuler_docker +pkgver=1.0 +pkgrel=1 +pkgdesc="openeuler docker image" +arch=('i386' 'x86_64') +url="https://repo.openeuler.org/openEuler-20.03-LTS/docker_img" +license=('GPL') +source=("https://repo.openeuler.org/openEuler-20.03-LTS/docker_img/aarch64/openEuler…") +md5sums=('6aa300910c9f9040b725939fb80e8121') + +handle_docker_image() +{ + docker load < openEuler-docker.aarch64.tar.xz + docker run -d --name my-openeuler openeuler-20.03-lts:latest + container_id=$(docker ps -aqf 'name=my-openeuler') + docker cp /etc/yum.repos.d/openEuler.repo $container_id:/etc/yum.repos.d/ + docker commit $container_id openeuler-20.03-lts:v1 + docker save -o openeuler-20.03-lts-v1.tar openeuler-20.03-lts:v1 + docker rm -f $container_id +} + +package() +{ + handle_docker_image + mkdir -p "$pkgdir/lkp/benchmarks/${pkgname}" + cp "$srcdir/openeuler-20.03-lts-v1.tar" "$pkgdir/lkp/benchmarks/${pkgname}" +} -- 2.23.0

1 0

[PATCH v4 compass-ci] git_mirror.rb: fix that es-states clean when git-mirror restart
by Li Yuanchao 20 Jan '21

20 Jan '21

Everytime git-mirror get new_refs or just git fetch, these actions and the time will be recorded in hash fork_stat{}. But when container git-mirror restart, fork_stat{} will be initialized. So historical data can't be kept. Now, git-mirror will get historical data from es when initialize hash fork_stat{}. So historical data will be kept. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 30 +++++++++++++++++++++--------- 1 file changed, 21 insertions(+), 9 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 86f02e5..4f74751 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -109,15 +109,8 @@ class MirrorMain @webhook_queue = channel.queue('web_hook') end - def fork_stat_init(stat_key) - @fork_stat[stat_key] = { - queued: false, - priority: 0, - fetch_time: [], - offset_fetch: 0, - new_refs_time: [], - offset_new_refs: 0 - } + def fork_stat_init(git_repo) + @fork_stat[git_repo] = get_fork_stat(git_repo) end def load_defaults(repodir) @@ -372,4 +365,23 @@ class MirrorMain handle_submodule(submodule) end + + def get_fork_stat(git_repo) + fork_stat = { + queued: false, + priority: 0, + fetch_time: [], + offset_fetch: 0, + new_refs_time: [], + offset_new_refs: 0 + } + query = { query: { match: { git_repo: git_repo } } } + result = @es_client.search(index: 'repo', body: query)['hits'] + return fork_stat unless result['total'].positive? + + fork_stat.each_key do |key| + fork_stat[key] = result['hits'][0]['_source'][key.to_s] + end + return fork_stat + end end -- 2.23.0

1 0

[PATCH v3 compass-ci] git_mirror.rb: fix that es-states clean when git-mirror restart
by Li Yuanchao 20 Jan '21

20 Jan '21

Everytime git-mirror get new_refs or just git fetch, these actions and the time will be recorded in hash fork_stat{}. But when container git-mirror restart, fork_stat{} will be initialized. So historical data can't be kept. Now, git-mirror will get historical data from es when initialize hash fork_stat{}. So historical data will be kept. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 35 ++++++++++++++++++++++++++--------- 1 file changed, 26 insertions(+), 9 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 86f02e5..6bb474a 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -109,15 +109,8 @@ class MirrorMain @webhook_queue = channel.queue('web_hook') end - def fork_stat_init(stat_key) - @fork_stat[stat_key] = { - queued: false, - priority: 0, - fetch_time: [], - offset_fetch: 0, - new_refs_time: [], - offset_new_refs: 0 - } + def fork_stat_init(git_repo) + @fork_stat[git_repo] = get_fork_stat(git_repo) end def load_defaults(repodir) @@ -372,4 +365,28 @@ class MirrorMain handle_submodule(submodule) end + + def copy_from_es(es_data) + fork_stat = {} + es_data.each do |key, value| + fork_stat[key.to_sym] = value + end + return fork_stat + end + + def get_fork_stat(git_repo) + query = { query: { match: { git_repo: git_repo } } } + result = @es_client.search(index: 'repo', body: query)['hits'] + return copy_from_es(result['hits'][0]['_source']) if result['total'].positive? + + fork_stat = { + queued: false, + priority: 0, + fetch_time: [], + offset_fetch: 0, + new_refs_time: [], + offset_new_refs: 0 + } + return fork_stat + end end -- 2.23.0

1 0