- Compass-ci - mailweb.openeuler.org

[PATCH v4 compass-ci] container/scheduler-https: raise warning if missing CA
by Wu Zhende 12 Nov '20

12 Nov '20

Not all environments require https scheduler. So start it only if CA certificate exists. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/scheduler-https/start | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/container/scheduler-https/start b/container/scheduler-https/start index 38c4892..9898c1a 100755 --- a/container/scheduler-https/start +++ b/container/scheduler-https/start @@ -6,6 +6,13 @@ require 'set' require_relative '../defconfig.rb' +ca_path = '/etc/compass-ci/ca' +crt_path = "#{ca_path}/server.crt" +key_path = "#{ca_path}/server.key" + +warn_msg = 'CA certificate is missing. Ignore if the HTTPS scheduler is not required' +raise warn_msg unless File.exist?(crt_path) && File.exist?(key_path) + names = Set.new %w[ SCHED_HOST SCHED_PORT -- 2.23.0

1 0

[PATCH lkp-tests 1/3] iso2rootfs: support not submit test job
by Yu Chuan 12 Nov '20

12 Nov '20

[Why] If user only want generate a rootfs without testing, he can not write submit related fields in the job.yaml. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/iso2rootfs.yaml | 2 ++ tests/iso2rootfs | 22 +++++++++++++++------- 2 files changed, 17 insertions(+), 7 deletions(-) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index 644c64be06b4..d4f0ffc0e80c 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -11,6 +11,8 @@ iso2rootfs: rootfs_path: initrd_http_host: initrd_http_port: + + # submit test yaml related fields test_yaml: test_os: test_os_arch: diff --git a/tests/iso2rootfs b/tests/iso2rootfs index e8a027b0dad2..f864188a8b02 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -53,12 +53,6 @@ check_yaml_vars() "rootfs_path" "initrd_http_host" "initrd_http_port" - "test_yaml" - "test_os" - "test_os_arch" - "test_os_version" - "test_os_mount" - "test_testbox" ) local yaml_t @@ -443,6 +437,20 @@ submit_test_job_yaml() [ -d "${LKP_SRC}" ] || die "cannot find value of var: LKP_SRC." + local test_yaml_vars=( + "test_os" + "test_os_arch" + "test_os_version" + "test_os_mount" + "test_testbox" + ) + + local yaml_t + for yaml_t in "${test_yaml_vars[@]}" + do + [ -n "$(eval echo "\$${yaml_t}")" ] || die "cannot fine value of var: ${yaml_t}." + done + cd "${LKP_SRC}" && { chmod +x ./sbin/install-dependencies.sh ./sbin/install-dependencies.sh @@ -492,7 +500,7 @@ main() run_qcow2rootfs test_rootfs - submit_test_job_yaml + [ -z ${test_yaml} ] || submit_test_job_yaml post_works } -- 2.23.0

3 6

[PATCH v3 compass-ci] sched: simplify logic and code
by Ren Wen 12 Nov '20

12 Nov '20

Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/job.cr | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 2418c82..1815b93 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -289,9 +289,7 @@ class Job "SCHED_PORT"] initialized_keys.each do |key| - if @hash.has_key?(key) == false - return false - end + return false unless @hash.has_key?(key) end return false if "#{@hash["id"]}" == "" -- 2.23.0

1 0

[PATCH v2 compass-ci] sched: simplify logic and code
by Ren Wen 12 Nov '20

12 Nov '20

def has_key?(key) : Returns true when key given by key exists, otherwise false. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/job.cr | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 2418c82..1815b93 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -289,9 +289,7 @@ class Job "SCHED_PORT"] initialized_keys.each do |key| - if @hash.has_key?(key) == false - return false - end + return false unless @hash.has_key?(key) end return false if "#{@hash["id"]}" == "" -- 2.23.0

2 2

[PATCH lkp-tests] stats: add hackbench-git result parsing script.
by Zhang Yale 12 Nov '20

12 Nov '20

Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- stats/hackbench-git | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100755 stats/hackbench-git diff --git a/stats/hackbench-git b/stats/hackbench-git new file mode 100755 index 00000000..435948ea --- /dev/null +++ b/stats/hackbench-git @@ -0,0 +1,19 @@ +#!/usr/bin/env ruby + +LKP_SRC = ENV['LKP_SRC'] || File.dirname(File.dirname(File.realpath($PROGRAM_NAME))) + +require "#{LKP_SRC}/lib/statistics" + +time = [] +while (line = gets) + case line + when /^Running with .* $== (\d+)$ tasks/ + tasks = $1.to_i + when /^Time:/ + _name, seconds = line.split + time << seconds.to_f + end +end + +puts "tasks: #{tasks}" +puts "Time: #{time.sum / time.size}" -- 2.23.0

1 1

[PATCH v2 compass-ci 2/2] sched: simplify logic
by Ren Wen 12 Nov '20

12 Nov '20

def has_key?(key) : Returns true when key given by key exists, otherwise false. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/job.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 2418c82..e3d47de 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -289,7 +289,7 @@ class Job "SCHED_PORT"] initialized_keys.each do |key| - if @hash.has_key?(key) == false + unless @hash.has_key?(key) return false end end -- 2.23.0

2 3

[PATCH v2 lkp-tests] test/cci-makepkg: fix a softlink will be established if the package fails
by Sun Yukui 12 Nov '20

12 Nov '20

[why] After packaging fails, a softlink will be established, but this is wrong. If the packaging fails, the soft connection should not be established. Signed-off-by: Sun Yukui <sun.yukui(a)foxmail.com> --- tests/cci-makepkg | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index a105f189..c6fce675 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -92,10 +92,6 @@ cgz_name="${pkgver:-0}-${pkgrel:-0}.cgz" setup_proxy -echo $LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf -PACMAN=true BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" \ - $LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf - update_softlink() { [ -e "$sync_dest/$cgz_name" ] || return @@ -109,4 +105,13 @@ update_softlink() rm $sync_dest/$cgz_name } -update_softlink && echo "Makepkg finished successfully" +build_source_pkg() +{ + echo $LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf + PACMAN=true BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" \ + $LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf + + [ "$?" == 0 ] || update_softlink +} + +build_source_pkg -- 2.23.0

2 1

[PATCH v3 compass-ci 3/3] git_mirror: use "gitclone.com" to make "git clone" faster
by Li Yuanchao 12 Nov '20

12 Nov '20

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 99534f6..29d1390 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -31,12 +31,18 @@ class GitMirror @feedback_queue.push(@feedback_info) end - def git_clone(url, mirror_dir) - ret = false - url = Array(url)[0] + def get_url(url) if url.include?('gitee.com/') && File.exist?("/srv/git/#{url.delete_prefix('https://')}") url = "/srv/git/#{url.delete_prefix('https://')}" + elsif url.include?('://github.com') + url = "https://gitclone.com/#{url.split('://')[1]}" end + return url + end + + def git_clone(url, mirror_dir) + ret = false + url = get_url(Array(url)[0]) 10.times do stderr = %x(git clone --mirror #{url} #{mirror_dir} 2>&1) ret = !stderr.include?('fatal') -- 2.23.0

1 0

[PATCH v3 compass-ci 2/3] git_mirror: fix that worker thread die after system(git clone)
by Li Yuanchao 12 Nov '20

12 Nov '20

[why] As Signal.trap(:SIGCHLD, 'SIG_IGN') has been set, the init process will handle the return status of child process, so ret = system(git clone xxx) can't get return value, which cause worker thread die. Use %x() instead of system(), and use stderr to check if 'git clone' succeed. So return value is not needed. [errmsg] in `system': No child process - Another thread waited the process started by system(). (Errno::ECHILD) Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 6e6664a..99534f6 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -38,7 +38,8 @@ class GitMirror url = "/srv/git/#{url.delete_prefix('https://')}" end 10.times do - ret = system("git clone --mirror #{url} #{mirror_dir}") + stderr = %x(git clone --mirror #{url} #{mirror_dir} 2>&1) + ret = !stderr.include?('fatal') break if ret end FileUtils.rm_r(mirror_dir) unless ret -- 2.23.0

1 0

[PATCH v3 compass-ci 1/3] git_mirror: pull submodule if repositories have
by Li Yuanchao 12 Nov '20

12 Nov '20

[why] Some repositories have submodules. Without these submodules, they can not be built successfully. Submodules of repositories are repositories too. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 33 ++++++++++++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 9d6a45d..6e6664a 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -56,7 +56,8 @@ class GitMirror def mirror_sync fork_info = @queue.pop - mirror_dir = "/srv/git/#{fork_info['git_repo']}.git" + mirror_dir = "/srv/git/#{fork_info['git_repo']}" + mirror_dir = "#{mirror_dir}.git" unless fork_info['is_submodule'] possible_new_refs = false if File.directory?(mirror_dir) possible_new_refs = git_fetch(mirror_dir) @@ -159,6 +160,8 @@ class MirrorMain feedback_info = @feedback_queue.pop(true) git_repo = feedback_info[:git_repo] + return if check_submodule(git_repo) + update_fork_stat(git_repo, feedback_info[:possible_new_refs]) return unless feedback_info[:possible_new_refs] @@ -219,6 +222,8 @@ class MirrorMain end def get_cur_refs(git_repo) + return if @git_info[git_repo]['is_submodule'] + mirror_dir = "/srv/git/#{git_repo}.git" show_ref_out = %x(git -C #{mirror_dir} show-ref --heads) cur_refs = { heads: {} } @@ -328,4 +333,30 @@ class MirrorMain end end end + + def handle_submodule(submodule, parent_project) + submodule.each_line do |line| + next unless line.include?('url = ') + + url = line.split(' = ')[1].chomp + git_repo = "submodules/#{parent_project[0]}/#{parent_project}/#{File.basename(url)}" + @git_info[git_repo] = { 'url' => url, 'git_repo' => git_repo, 'is_submodule' => true } + fork_stat_init(git_repo) + @priority_queue.push git_repo, @priority + @priority += 1 + end + end + + def check_submodule(git_repo) + if @git_info[git_repo]['is_submodule'] + @fork_stat[git_repo][:queued] = false + return true + end + + mirror_dir = "/srv/git/#{git_repo}.git" + submodule = %x(git -C #{mirror_dir} show HEAD:.gitmodules) + return if submodule.empty? + + handle_submodule(submodule, File.basename(git_repo)) + end end -- 2.23.0

1 0