- Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci] /src/extract-stats: rescue when the task-queue disconnect
by Lu Weitao 09 Mar '21

09 Mar '21

[Why] the connection between stats_worker and task-queue may be interrupted, and the old connection will be disable, so stats_worker need create a new connection with task-queue Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- src/extract-stats/stats_worker.cr | 3 +++ 1 file changed, 3 insertions(+) diff --git a/src/extract-stats/stats_worker.cr b/src/extract-stats/stats_worker.cr index 836d1f8..dbb3fe2 100644 --- a/src/extract-stats/stats_worker.cr +++ b/src/extract-stats/stats_worker.cr @@ -27,8 +27,11 @@ class StatsWorker end rescue e STDERR.puts e.message + error_message = e.message + # incase of many error message when task-queue, ES does not work sleep(10) + @tq = TaskQueueAPI.new if error_message && error_message.includes?("3060': Connection refused") end end end -- 2.23.0

1 0

[PATCH compass-ci] container/rabbitmq: support data persistence to hard disks
by Wu Zhende 09 Mar '21

09 Mar '21

If data persistence is not performed, all data in the queue will be cleared when the service is restarted. However, some queues need to be restarted to restore data. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/rabbitmq/start | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/container/rabbitmq/start b/container/rabbitmq/start index 6b875f3..cdf5dee 100755 --- a/container/rabbitmq/start +++ b/container/rabbitmq/start @@ -4,16 +4,21 @@ . $CCI_SRC/container/defconfig.sh -docker_rm rabbitmq +docker_name=rabbitmq + +docker_rm $docker_name cmd=( docker run --restart=always -d - --name rabbitmq + --name $docker_name + --hostname $docker_name + -u 1090 -p 5672:5672 -p 15672:15672 -v /etc/localtime:/etc/localtime:ro + -v /srv/rabbitmq:/var/lib/rabbitmq rabbitmq:3-management ) -- 2.23.0

1 0

[PATCH compass-ci] container/dracut-initrd: judge local before handle 90lkp
by Xu Xijian 09 Mar '21

09 Mar '21

[why] 90lkp is only used for local mount to replace 90overlay-root, don't use 90lkp with other mount, the errlog occured with cifs. [errlog] Starting dracut pre-pivot and cleanup hook... [ 7.812554] dracut-pre-pivot[626]: cp: cannot create directory '/sysroot/lkp': Read-only file system [ 7.816968] dracut-pre-pivot[626]: cp: cannot create symbolic link '/sysroot/etc/X11/X': Read-only file system Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/modules.d/90lkp/lkp-deploy.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/container/dracut-initrd/modules.d/90lkp/lkp-deploy.sh b/container/dracut-initrd/modules.d/90lkp/lkp-deploy.sh index 8be927c..44f994e 100755 --- a/container/dracut-initrd/modules.d/90lkp/lkp-deploy.sh +++ b/container/dracut-initrd/modules.d/90lkp/lkp-deploy.sh @@ -3,6 +3,10 @@ . /lib/dracut-lib.sh +if ! getargbool 0 local; then + return +fi + # transfer LKP dirs [ -d /lkp ] || return 0 -- 2.23.0

1 0

[PATCH compass-ci] lib/git_mirror.rb: github download workaround
by Li Yuanchao 09 Mar '21

09 Mar '21

There is a problem to download from github by http or https. fatal: unable to access 'https://github.com/liyc-github/AvxToNeon/': Failed to connect to github.com port 443: Connection timed out When it occurs, change url to "git://xxx". Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 53c9e59..1fa8c9e 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -42,16 +42,20 @@ class GitMirror def git_clone(url, mirror_dir) url = get_url(Array(url)[0]) 10.times do - %x(git clone --mirror --depth 1 #{url} #{mirror_dir} 2>&1) + stderr = %x(git clone --mirror --depth 1 #{url} #{mirror_dir} 2>&1) return true if File.directory?(mirror_dir) && File.exist?("#{mirror_dir}/config") + + if stderr.include?('Failed to connect to github.com port 443') + url = "git://#{url.split('://')[1]}" + end end return false end def git_fetch(mirror_dir) if File.exist?("#{mirror_dir}/shallow") + FileUtils.rm("#{mirror_dir}/shallow.lock") if File.exist?("#{mirror_dir}/shallow.lock") %x(git -C #{mirror_dir} fetch --unshallow 2>&1) - return false end fetch_info = %x(git -C #{mirror_dir} fetch 2>&1) @@ -59,6 +63,11 @@ class GitMirror if fetch_info.include?(ERR_MESSAGE) && Dir.empty?(mirror_dir) FileUtils.rmdir(mirror_dir) end + + if fetch_info.include?('Failed to connect to github.com port 443') + url = %x(git -C #{mirror_dir} ls-remote --get-url origin).chomp + %x(git -C #{mirror_dir} remote set-url origin git://#{url.split('://')[1]}) + end return fetch_info.include? '->' end @@ -66,7 +75,7 @@ class GitMirror url = get_url(Array(url)[0]) git_url = %x(git -C #{mirror_dir} ls-remote --get-url origin).chomp - return true if url == git_url + return true if git_url == url || git_url.include?(url.split('://')[1]) return false end -- 2.23.0

2 1

[PATCH lkp-tests] sbin/submit: fix input docker_image value by cmdline
by Wei Jihui 09 Mar '21

09 Mar '21

[error] input: submit docker_image=centos:7 job.yaml output: docker_image: centos [why] submit split centos:7 to [centos, 7] [how] -s "docker_image: centos:7" ==> docker_image: centos:7 docker_image=centos:7 ==> docker_image: centos:7 Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- sbin/submit | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/sbin/submit b/sbin/submit index e5e41828..feb65d00 100755 --- a/sbin/submit +++ b/sbin/submit @@ -28,7 +28,7 @@ options = OptionParser.new do |opts| opts.separator 'options:' opts.on("-s 'KEY: VALUE'", "--set 'KEY: VALUE'", 'add YAML hash to job') do |key_value| - k, v = key_value.sub(' ', '').split(':') + k, v = key_value.sub(' ', '').split(':', 2) opt_set_key_value[k] = v end @@ -59,7 +59,7 @@ options = OptionParser.new do |opts| opts.on('-m', '--monitor', "monitor job status: use -m 'KEY: VALUE' to add rule") do opt_monitor = true - k, v = ARGV[0].sub(' ', '').split(':') if ARGV[0] + k, v = ARGV[0].sub(' ', '').split(':', 2) if ARGV[0] if (k && !k.empty?) && (v && !v.empty?) opt_monitor_query[k] = v ARGV.shift @@ -78,7 +78,7 @@ options.parse!(ARGV) seen_job_yaml = false ARGV.delete_if do |arg| if arg.index '=' - k, v = arg.sub(' ', '').sub(/=/, ':').split(':') + k, v = arg.sub(' ', '').split('=', 2) if seen_job_yaml opt_set_key_value[k] = v else -- 2.23.0

1 0

[PATCH compass-ci 2/2] sparrow/install-tiny: call 5-build/register-account
by Liu Yinsi 09 Mar '21

09 Mar '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/install-tiny | 1 + 1 file changed, 1 insertion(+) diff --git a/sparrow/install-tiny b/sparrow/install-tiny index 9e94e39..e48bd25 100755 --- a/sparrow/install-tiny +++ b/sparrow/install-tiny @@ -21,4 +21,5 @@ cd $CCI_SRC/sparrow || exit 4-docker/buildall 5-build/ipxe 5-build/os-ready +5-build/register-account 7-systemd/systemd-setup -- 2.23.0

1 0

[PATCH compass-ci 1/2] sparrow/install-tiny: call 0-package/read-config
by Liu Yinsi 09 Mar '21

09 Mar '21

for users locally deploy compass-ci, call 0-package/read-config, export my_name, my_email to register account. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/install-tiny | 1 + 1 file changed, 1 insertion(+) diff --git a/sparrow/install-tiny b/sparrow/install-tiny index 5f026e5..9e94e39 100755 --- a/sparrow/install-tiny +++ b/sparrow/install-tiny @@ -7,6 +7,7 @@ [[ $CCI_SRC ]] || export CCI_SRC=$(cd $(dirname $(realpath $0)); git rev-parse --show-toplevel) cd $CCI_SRC/sparrow || exit +. 0-package/read-config 0-package/install 1-storage/tiny 1-storage/permission -- 2.23.0

1 0

[PATCH v3 lkp-tests] setup/simplify-ci: call $CCI_SRC/sparrow/install-client
by Liu Yinsi 09 Mar '21

09 Mar '21

[why] for simplicy-ci job test, if there is $SCHED_HOST, and export server_ip=$SCHED_HOST, when call $CCI_SRC/sparrow/install-client, config sparrow/setup.yaml and execute sparrow/0-package/read-config(export server_ip) is unnecessary. for locally deploy compass-ci client, must config sparrow/setup.yaml, and execute sparrow/0-package/read-config(export server_ip), if server_ip not exists, will call log_error() and exit. [how] in $LKP_SRC/setup/simplify-ci, just call $CCI_SRC/sparrow/install-client script, make code more reusablely. refer patch: Subject: [PATCH v5 compass-ci 1/2] sparrow/install-client: add deploy compass-ci client script Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- setup/simplify-ci | 61 ++++------------------------------------------- 1 file changed, 5 insertions(+), 56 deletions(-) diff --git a/setup/simplify-ci b/setup/simplify-ci index 5aede9cdb..83e20b5f8 100755 --- a/setup/simplify-ci +++ b/setup/simplify-ci @@ -2,8 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -: ${SCHED_HOST:=172.17.0.1} -: ${SCHED_PORT:=3000} +export server_ip=$SCHED_HOST git_ci() { @@ -17,61 +16,11 @@ git_ci() git clone https://gitee.com/wu_fengguang/compass-ci.git /c/compass-ci || return 1 } -dev_env() +deploy() { - export sched_host=$SCHED_HOST - export sched_port=$SCHED_PORT - 3-code/dev-env + cd /c/compass-ci/sparrow && ./install-client } -install_env() -{ - cd /c/compass-ci/sparrow || return - 0-package/install - 1-storage/tiny - 5-build/ipxe & - 1-storage/permission - 2-network/br0 - 2-network/iptables - 3-code/git - dev_env - . /etc/profile.d/compass.sh -} - -boot_ipxe() -{ - sed -i "s%172.17.0.1%$SCHED_HOST%g" /tftpboot/boot.ipxe - sed -i "s%3000%$SCHED_PORT%g" /tftpboot/boot.ipxe -} - -run_service() -{ - ( - cd $CCI_SRC/container/dnsmasq || return - ./build - ./start - boot_ipxe - )& - ( - cd $CCI_SRC/container/qemu-efi || return - ./build - ./install - )& - ( - cd $CCI_SRC/container/fluentd-base || return - ./build - cd $CCI_SRC/container/sub-fluentd || return - ./build - ./start - )& -} - -main() -{ - git_ci || return 1 - install_env - run_service -} +git_ci || exit 1 +deploy -main -wait -- 2.23.0

1 0

[PATCH lkp-tests] tests/rpmbuild-pkg: delete os_mount field in sync_dest dir.
by Li Ping 09 Mar '21

09 Mar '21

[why] rpm has no much to do with os_mount Signed-off-by: Li Ping <1477412247(a)qq.com> --- tests/rpmbuild-pkg | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index 38de16e66..b641e97d1 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -6,8 +6,7 @@ [ -n "$rpm_repo" ] || die "rpm_repo is empty" package_name=${rpm_repo##*/} -[[ "$os_mount" = "cifs" ]] && os_mount="nfs" -sync_dest="/initrd/rpmbuild-pkg/${os_mount}/${os}/${os_arch}/${os_version}/${package_name}" +sync_dest="/initrd/rpmbuild-pkg/${os}/${os_arch}/${os_version}/${package_name}" init_workspace() { -- 2.23.0

1 0

[PATCH v2 compass-ci] src/lib/job.cr: optimize the get_pkg_common_dir function
by Li Ping 09 Mar '21

09 Mar '21

[why] 1. rpm has no much to do with os_mount. add a judgment 2. optimize the code style Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/job.cr | 29 +++++++++++++++++------------ 1 file changed, 17 insertions(+), 12 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 0b4bee4..fe1b7c1 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -258,23 +258,28 @@ class Job end def get_pkg_common_dir - pkg_style = nil + pkg_style = Hash(String, JSON::Any).new {|h, k| h[k] = JSON::Any.new(nil)} ["cci-makepkg", "cci-depends", "build-pkg", "rpmbuild-pkg"].each do |item| - pkg_style = @hash[item]? - break if pkg_style - end - return nil unless pkg_style + next unless @hash[item]? - pkg_style = JSON.parse("{}") if pkg_style == nil + pkg_style.merge!((a)hash[item].as_h) + break + end + return nil if pkg_style.empty? - tmp_os = pkg_style["os"]? == nil ? "#{os}" : pkg_style["os"] - tmp_os_arch = pkg_style["os_arch"]? == nil ? "#{os_arch}" : pkg_style["os_arch"] - tmp_os_version = pkg_style["os_version"]? == nil ? "#{os_version}" : pkg_style["os_version"] + tmp_os = pkg_style["os"].as_s? || "#{os}" + tmp_os_arch = pkg_style["os_arch"].as_s? || "#{os_arch}" + tmp_os_version = pkg_style["os_version"].as_s? || "#{os_version}" - tmp_os_mount = pkg_style["os_mount"]? == nil ? "#{os_mount}" : pkg_style["os_mount"] - mount_type = tmp_os_mount == "cifs" ? "nfs" : tmp_os_mount.dup + mount_type = pkg_style["os_mount"].as_s? || "#{os_mount}" + # same usage for client + mount_type = "nfs" if mount_type == "cifs" - common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" + if @hash["rpmbuild-pkg"]? + common_dir = "#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" + else + common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" + end return common_dir end -- 2.23.0

1 0