November 2020 - Compass-ci - mailweb.openeuler.org

[PATCH compass-ci 2/2] lib/job.cr: check the account
by Wu Zhende 23 Nov '20

23 Nov '20

When a job submit, check it's account info. Must key:my_email, my_name, my_uuid Check whether the information is the same as that stored in the ES database Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 4bef329..05c5462 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -9,6 +9,7 @@ require "scheduler/constants.cr" require "scheduler/jobfile_operate.cr" require "scheduler/kernel_params.cr" require "scheduler/pp_params.cr" +require "../scheduler/elasticsearch_client" struct JSON::Any def []=(key : String, value : String) @@ -58,8 +59,10 @@ class Job return if @hash["id"] == "#{id}" end + @es = Elasticsearch::Client.new @hash["id"] = JSON::Any.new("#{id}") check_required_keys() + check_account_info() set_defaults() end @@ -271,6 +274,9 @@ class Job id suite testbox + my_email + my_uuid + my_name ] private def check_required_keys @@ -281,6 +287,17 @@ class Job end end + private def check_account_info + error_msg = "Failed to verify the account. Please check your configuration" + account_info = @es.get_account(self["my_email"]) + + raise error_msg if account_info["found"]? == false + raise error_msg unless self["my_name"] == account_info["my_name"].to_s + raise error_msg unless self["my_uuid"] == account_info["my_uuid"] + + @hash.delete("my_uuid") + end + private def initialized? initialized_keys = [] of String @@ -302,6 +319,8 @@ class Job "SCHED_HOST", "SCHED_PORT"] + initialized_keys -= ["my_uuid"] + initialized_keys.each do |key| return false unless @hash.has_key?(key) end @@ -441,7 +460,7 @@ class Job else program_version = "latest" end - + deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}/#{program}.cgz" pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}/#{program_version}.cgz" -- 2.23.0

2 1

[PATCH v3 compass-ci 1/3] git_mirror: fix that README.md in upstream-repos cause a load error
by Li Yuanchao 23 Nov '20

23 Nov '20

[why] As README.md or other docs in upstream-repos, and they are not yaml file. But git-mirror load all files in upstream-repos as yaml file, so there comes out a load error. Now files whose dirname is 'xxx/upstream-repos' will not be loaded. [errmsg] ...in `load_repo_file': undefined method `[]=' for x:Integer (NoMethodError) Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 9648063..72ca328 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -135,6 +135,8 @@ class MirrorMain traverse_repodir("#{repodir}/#{entry}") end else + return if File.dirname(repodir) == REPO_DIR + project = File.dirname(repodir).delete_prefix("#{REPO_DIR}/") fork_name = File.basename(repodir) load_repo_file(repodir, project, fork_name) -- 2.23.0

2 1

[PATCH v3 compass-ci 3/3] container/git-mirror: fix that can't get group id of committer in z9
by Li Yuanchao 23 Nov '20

23 Nov '20

In z9, 'committer' is only a group, not a user, so command 'id -g committer' can't get id of committer group Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/git-mirror/start | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/git-mirror/start b/container/git-mirror/start index e1d7d8f..a618ec3 100755 --- a/container/git-mirror/start +++ b/container/git-mirror/start @@ -10,13 +10,13 @@ DOCKER_REPO = '/c/upstream-repos' docker_rm "git-mirror" lkp = %x(id -u lkp) -committer = %x(id -g committer) +group_id = %x(cat /etc/group | grep '^committer' | awk -F ':' '{print $3}') cmd = %W[ docker run --restart=always --name git-mirror - -u #{lkp.to_i}:#{committer.to_i} + -u #{lkp.to_i}:#{group_id.to_i} -d -e CCI_SRC=#{DOCKER_CCI} -e REPO_SRC=#{DOCKER_REPO} -- 2.23.0

2 1

[PATCH lkp-tests 2/2] sbin/my-submit: submit a task to personal queue
by Xiao Shenwei 23 Nov '20

23 Nov '20

usage: my-submit testbox=vm-2p8g jobs/borrow-1h.yaml use this script to submit job, your job will be add to persoanl queue which is $tbox_group~$USER Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- sbin/my-submit | 3 +++ 1 file changed, 3 insertions(+) create mode 100755 sbin/my-submit diff --git a/sbin/my-submit b/sbin/my-submit new file mode 100755 index 00000000..57489948 --- /dev/null +++ b/sbin/my-submit @@ -0,0 +1,3 @@ +#!/bin/bash + +submit --my-queue $@ -- 2.23.0

2 2

[PATCH v3 compass-ci 2/3] git_mirror: fix that too few repos in upstream-repos cause memory leak
by Li Yuanchao 21 Nov '20

21 Nov '20

When there are only a few repo files in upstream-repos, that means the same repo will be called very frequently. As the main loop is too fast, it seems the new flow would be called before the last flow of the same repo end completely. That cause a problem. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 72ca328..5b6b889 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -203,6 +203,7 @@ class MirrorMain push_git_queue handle_feedback Signal.trap(:SIGCHLD, 'SIG_IGN') + sleep(0.1) end end end -- 2.23.0

1 0

[PATCH compass-ci 1/2] scheduler/elasticsearch_client.cr: add function get_account
by Wu Zhende 21 Nov '20

21 Nov '20

Add a new function "get_account" to get account info by "my_email". Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/elasticsearch_client.cr | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/src/scheduler/elasticsearch_client.cr b/src/scheduler/elasticsearch_client.cr index ec6de0d..6ea8e2d 100644 --- a/src/scheduler/elasticsearch_client.cr +++ b/src/scheduler/elasticsearch_client.cr @@ -69,6 +69,16 @@ class Elasticsearch::Client return job end + def get_account(my_email : String) + if @client.exists({:index => "accounts", :type => "_doc", :id => my_email}) + response = @client.get_source({:index => "accounts", :type => "_doc", :id => my_email}) + else + response = {"_id" => my_email, "found" => false} + end + + return response + end + private def create(job_content : JSON::Any, job_id : String) return @client.create( { -- 2.23.0

1 0

[PATCH lkp-tests] lkp-tests/*: add archlinux install pp support
by Wang Yong 21 Nov '20

21 Nov '20

archlinux use pacman to install pp Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- bin/lkp-setup-rootfs | 1 + lib/bootstrap.sh | 7 +++++++ 2 files changed, 8 insertions(+) diff --git a/bin/lkp-setup-rootfs b/bin/lkp-setup-rootfs index 98c63bae9..0eabea755 100755 --- a/bin/lkp-setup-rootfs +++ b/bin/lkp-setup-rootfs @@ -261,6 +261,7 @@ while true; do fi install_rpms install_deb + install_pkgs echo $LKP_DEBUG_PREFIX $LKP_SRC/bin/run-lkp $LKP_DEBUG_PREFIX $LKP_SRC/bin/run-lkp $job diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index a7776d687..ce34a6329 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -705,3 +705,10 @@ boot_init() mount_rootfs } + +install_pkgs() +{ + [ -d /opt/pkgs ] || return + echo "install $(ls /opt/pkgs/*.pkg.tar*)" + pacman -U --needed --noconfirm /opt/pkgs/*.pkg.tar* +} -- 2.23.0

1 0

[PATCH v2 lkp-tests] tests/iozone: explicit support test=all for iozone
by Lu Kaiyi 21 Nov '20

21 Nov '20

[example] iozone-bs.yaml ... suite: iozone category: benchmark file_size: 4g test: all block_size: - 64k ... [output] ... Run began: Sat Nov 21 16:48:22 2020 Record Size 64 kB File size set to 4194304 kB Command line used: iozone -r 64k -s 4g -f /fs/sdb/iozonetest Output is in kBytes/sec Time Resolution = 0.000001 seconds. Processor cache size set to 1024 kBytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. random random bkwd record stride kB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 4194304 64 13650392 11529057 7878373 7763936 7166639 10774159 7520609 16188351 7362482 9751567 9453804 7981606 7932187 iozone test complete. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- tests/iozone | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/iozone b/tests/iozone index b6a1e206..bafc0087 100755 --- a/tests/iozone +++ b/tests/iozone @@ -19,6 +19,8 @@ if [ -n "$block_size" ]; then for ele in ${array[@]} do case $ele in + "all") args+=" -a" + ;; "write") args+=" -i 0" ;; "read") args+=" -i 1" -- 2.23.0

1 0

[PATCH lkp-tests] tests/iozone: explicit support test=all for iozone
by Lu Kaiyi 21 Nov '20

21 Nov '20

[example] iozone-bs.yaml ... suite: iozone category: benchmark file_size: 4g test: all block_size: - 64k ... [output] ... Run began: Sat Nov 21 16:10:59 2020 Record Size 64 kB File size set to 4194304 kB Command line used: iozone -r 64k -s 4g -f /fs/sda/iozonetest Output is in kBytes/sec Time Resolution = 0.000001 seconds. Processor cache size set to 1024 kBytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. random random bkwd record stride kB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 4194304 64 13566379 14323483 8305849 8185793 7770078 13206535 4562592 15656107 4448821 4868781 4908219 4736366 4720413 iozone test complete. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- tests/iozone | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/iozone b/tests/iozone index b6a1e206..48688ebb 100755 --- a/tests/iozone +++ b/tests/iozone @@ -19,6 +19,8 @@ if [ -n "$block_size" ]; then for ele in ${array[@]} do case $ele in + "all") args+= " -a" + ;; "write") args+=" -i 0" ;; "read") args+=" -i 1" -- 2.23.0

1 0

[PATCH v2 compass-ci 3/3] container/git-mirror: fix that can't get group id of committer in z9
by Li Yuanchao 21 Nov '20

21 Nov '20

[why] In z9, 'committer' is only a group, not a user, so command 'id -g committer' can't get id of committer group Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/git-mirror/start | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/git-mirror/start b/container/git-mirror/start index e1d7d8f..f5321a1 100755 --- a/container/git-mirror/start +++ b/container/git-mirror/start @@ -10,7 +10,7 @@ DOCKER_REPO = '/c/upstream-repos' docker_rm "git-mirror" lkp = %x(id -u lkp) -committer = %x(id -g committer) +committer = %x(cat /etc/group | grep '^committer' | awk -F ':' '{print $3}') cmd = %W[ docker run -- 2.23.0

3 4