- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci 3/3] doc/job/: add initrd_deps.md
by Li Ping 23 Nov '20

23 Nov '20

add a document for introducing initrd_deps field. Signed-off-by: Li Ping <15396232681(a)163.com> --- doc/job/initrd_deps.md | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 doc/job/initrd_deps.md diff --git a/doc/job/initrd_deps.md b/doc/job/initrd_deps.md new file mode 100644 index 0000000..0c3d79c --- /dev/null +++ b/doc/job/initrd_deps.md @@ -0,0 +1,5 @@ +# initrd_deps + +Meaning: +- Most testcases depend on many other package to run. If user has already packed all the other package for running testcase. Then initrd_deps field would give the location of the package. +- The initrd_deps field do not need to be specified by user in the $testcase yaml. It is filled in by the scheduler. User can get the package through the wget command to check the rpm in compressed package. -- 2.23.0

1 0

[PATCH compass-ci 2/3] doc/job/: add initrd_pkg.md
by Li Ping 23 Nov '20

23 Nov '20

add a document for introducing the initrd_pkg field. Signed-off-by: Li Ping <15396232681(a)163.com> --- doc/job/initrd_pkg.md | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 doc/job/initrd_pkg.md diff --git a/doc/job/initrd_pkg.md b/doc/job/initrd_pkg.md new file mode 100644 index 0000000..2ef3d03 --- /dev/null +++ b/doc/job/initrd_pkg.md @@ -0,0 +1,5 @@ +# initrd_pkg + +Meaning: +- Most testcases rely on many other files such as configuration files to run. If user has already packed all the files for running testcase. Then initrd_pkg field would give the location of the packed compression file. +- The initrd_pkg field do not need to be specified by user in the $testcase yaml. It is filled in by the scheduler. User can get the package through the wget command to check the files in compressed package. -- 2.23.0

1 0

[PATCH lkp-tests 2/2] bin/my-submit: this script is used together with the my-qemu
by Xiao Shenwei 23 Nov '20

23 Nov '20

ignore the queue field, it will add default value which is "$tbox_group~$USER" usage: my-submit testbox=vm-2p8g jobs/borrow-1h.yaml Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- sbin/my-submit | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 sbin/my-submit diff --git a/sbin/my-submit b/sbin/my-submit new file mode 100755 index 00000000..3a7c741e --- /dev/null +++ b/sbin/my-submit @@ -0,0 +1,5 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +submit --my-queue $@ -- 2.23.0

1 1

[PATCH lkp-tests 1/2] sbin/submit: add job into my queue
by Xiao Shenwei 23 Nov '20

23 Nov '20

add --my-queue parameter the job will be add into your own queue usage: submit --my-queue testbox=vm-2p8g ../jobs/borrow-1h.yaml Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- sbin/submit | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/sbin/submit b/sbin/submit index 9bba1018..6af86496 100755 --- a/sbin/submit +++ b/sbin/submit @@ -16,6 +16,7 @@ opt_output_dir = nil opt_auto_define_files = false opt_monitor = false opt_monitor_query = {} +opt_my_queue = false actions = ['output', 'stop'] options = OptionParser.new do |opts| @@ -54,6 +55,11 @@ options = OptionParser.new do |opts| ARGV.shift end end + + opts.on('--my-queue', 'add to my queue') do + opt_my_queue = true + end + end options.parse!(ARGV) @@ -93,6 +99,11 @@ def find_jobfile(jobfile) exit 1 end +def add_my_queue(job) + queue = "#{job['tbox_group']}~#{ENV['USER']}" + job['queue'] = queue +end + ARGV.each do |jobfile| jobfile = find_jobfile(jobfile) jobs = Job2sh.new @@ -134,6 +145,7 @@ ARGV.each do |jobfile| job_hash_list << job_hash job_hash = job_hash.merge(sh_hash) + add_my_queue(job_hash) if opt_my_queue # save job to yaml if opt_output_dir prefix = File.join(opt_output_dir, File.basename(jobfile, '.yaml')) -- 2.23.0

1 0

[PATCH compass-ci 2/2] lib/job.cr: check the account
by Wu Zhende 23 Nov '20

23 Nov '20

When a job submit, check it's account info. Must key:my_email, my_name, my_uuid Check whether the information is the same as that stored in the ES database Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 4bef329..05c5462 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -9,6 +9,7 @@ require "scheduler/constants.cr" require "scheduler/jobfile_operate.cr" require "scheduler/kernel_params.cr" require "scheduler/pp_params.cr" +require "../scheduler/elasticsearch_client" struct JSON::Any def []=(key : String, value : String) @@ -58,8 +59,10 @@ class Job return if @hash["id"] == "#{id}" end + @es = Elasticsearch::Client.new @hash["id"] = JSON::Any.new("#{id}") check_required_keys() + check_account_info() set_defaults() end @@ -271,6 +274,9 @@ class Job id suite testbox + my_email + my_uuid + my_name ] private def check_required_keys @@ -281,6 +287,17 @@ class Job end end + private def check_account_info + error_msg = "Failed to verify the account. Please check your configuration" + account_info = @es.get_account(self["my_email"]) + + raise error_msg if account_info["found"]? == false + raise error_msg unless self["my_name"] == account_info["my_name"].to_s + raise error_msg unless self["my_uuid"] == account_info["my_uuid"] + + @hash.delete("my_uuid") + end + private def initialized? initialized_keys = [] of String @@ -302,6 +319,8 @@ class Job "SCHED_HOST", "SCHED_PORT"] + initialized_keys -= ["my_uuid"] + initialized_keys.each do |key| return false unless @hash.has_key?(key) end @@ -441,7 +460,7 @@ class Job else program_version = "latest" end - + deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}/#{program}.cgz" pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}/#{program_version}.cgz" -- 2.23.0

2 1

[PATCH v3 compass-ci 1/3] git_mirror: fix that README.md in upstream-repos cause a load error
by Li Yuanchao 23 Nov '20

23 Nov '20

[why] As README.md or other docs in upstream-repos, and they are not yaml file. But git-mirror load all files in upstream-repos as yaml file, so there comes out a load error. Now files whose dirname is 'xxx/upstream-repos' will not be loaded. [errmsg] ...in `load_repo_file': undefined method `[]=' for x:Integer (NoMethodError) Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 9648063..72ca328 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -135,6 +135,8 @@ class MirrorMain traverse_repodir("#{repodir}/#{entry}") end else + return if File.dirname(repodir) == REPO_DIR + project = File.dirname(repodir).delete_prefix("#{REPO_DIR}/") fork_name = File.basename(repodir) load_repo_file(repodir, project, fork_name) -- 2.23.0

2 1

[PATCH v3 compass-ci 3/3] container/git-mirror: fix that can't get group id of committer in z9
by Li Yuanchao 23 Nov '20

23 Nov '20

In z9, 'committer' is only a group, not a user, so command 'id -g committer' can't get id of committer group Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/git-mirror/start | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/git-mirror/start b/container/git-mirror/start index e1d7d8f..a618ec3 100755 --- a/container/git-mirror/start +++ b/container/git-mirror/start @@ -10,13 +10,13 @@ DOCKER_REPO = '/c/upstream-repos' docker_rm "git-mirror" lkp = %x(id -u lkp) -committer = %x(id -g committer) +group_id = %x(cat /etc/group | grep '^committer' | awk -F ':' '{print $3}') cmd = %W[ docker run --restart=always --name git-mirror - -u #{lkp.to_i}:#{committer.to_i} + -u #{lkp.to_i}:#{group_id.to_i} -d -e CCI_SRC=#{DOCKER_CCI} -e REPO_SRC=#{DOCKER_REPO} -- 2.23.0

2 1

[PATCH lkp-tests 2/2] sbin/my-submit: submit a task to personal queue
by Xiao Shenwei 23 Nov '20

23 Nov '20

usage: my-submit testbox=vm-2p8g jobs/borrow-1h.yaml use this script to submit job, your job will be add to persoanl queue which is $tbox_group~$USER Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- sbin/my-submit | 3 +++ 1 file changed, 3 insertions(+) create mode 100755 sbin/my-submit diff --git a/sbin/my-submit b/sbin/my-submit new file mode 100755 index 00000000..57489948 --- /dev/null +++ b/sbin/my-submit @@ -0,0 +1,3 @@ +#!/bin/bash + +submit --my-queue $@ -- 2.23.0

2 2

[PATCH v3 compass-ci 2/3] git_mirror: fix that too few repos in upstream-repos cause memory leak
by Li Yuanchao 21 Nov '20

21 Nov '20

When there are only a few repo files in upstream-repos, that means the same repo will be called very frequently. As the main loop is too fast, it seems the new flow would be called before the last flow of the same repo end completely. That cause a problem. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 72ca328..5b6b889 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -203,6 +203,7 @@ class MirrorMain push_git_queue handle_feedback Signal.trap(:SIGCHLD, 'SIG_IGN') + sleep(0.1) end end end -- 2.23.0

1 0

[PATCH compass-ci 1/2] scheduler/elasticsearch_client.cr: add function get_account
by Wu Zhende 21 Nov '20

21 Nov '20

Add a new function "get_account" to get account info by "my_email". Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/elasticsearch_client.cr | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/src/scheduler/elasticsearch_client.cr b/src/scheduler/elasticsearch_client.cr index ec6de0d..6ea8e2d 100644 --- a/src/scheduler/elasticsearch_client.cr +++ b/src/scheduler/elasticsearch_client.cr @@ -69,6 +69,16 @@ class Elasticsearch::Client return job end + def get_account(my_email : String) + if @client.exists({:index => "accounts", :type => "_doc", :id => my_email}) + response = @client.get_source({:index => "accounts", :type => "_doc", :id => my_email}) + else + response = {"_id" => my_email, "found" => false} + end + + return response + end + private def create(job_content : JSON::Any, job_id : String) return @client.create( { -- 2.23.0

1 0