- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] sched: add the secrets field processing logic
by Cao Xueliang 23 Feb '21

23 Feb '21

You can add the secrets field which you don't want to make public but the SUT need it in job.yaml. secrets: my_token:xxx The scheduler will save the secrets value to the redis id2secrets table, then delete the field from job.yaml. When consume the job, the scheduler will read the value from redis, then create a secrets.yaml in job.cgz. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- sbin/create-job-cpio.sh | 1 + src/scheduler/create_job_cpio.cr | 13 +++++++++++++ src/scheduler/submit_job.cr | 7 +++++++ 3 files changed, 21 insertions(+) diff --git a/sbin/create-job-cpio.sh b/sbin/create-job-cpio.sh index e18bc0d..afb316c 100755 --- a/sbin/create-job-cpio.sh +++ b/sbin/create-job-cpio.sh @@ -9,6 +9,7 @@ cd "$1" || exit install -m775 -D -t lkp/scheduled job.sh install -m664 -D -t lkp/scheduled job.yaml +[ -f secrets.yaml ] && install -m664 -D -t lkp/scheduled secrets.yaml find lkp | cpio --quiet -o -H newc | gzip > job.cgz diff --git a/src/scheduler/create_job_cpio.cr b/src/scheduler/create_job_cpio.cr index 51d79f7..5634f67 100644 --- a/src/scheduler/create_job_cpio.cr +++ b/src/scheduler/create_job_cpio.cr @@ -74,8 +74,21 @@ class Sched script_lines = JSON.parse(script_lines) end + def create_secrets_yaml(job_id, base_dir) + secrets = @redis.hash_get("id2secrets", job_id) + return nil unless secrets + + secrets_yaml = base_dir + "/#{job_id}/secrets.yaml" + prepare_dir(secrets_yaml) + + File.open(secrets_yaml, "w") do |file| + YAML.dump(JSON.parse(secrets), file) + end + end + def create_job_cpio(job_content : JSON::Any, base_dir : String) job_content = job_content.as_h + create_secrets_yaml(job_content["id"], base_dir) # put job2sh in an array if job_content.has_key?("job2sh") diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index 1837cb1..fe9efa1 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -144,8 +144,15 @@ class Sched JSON.parse(response[1].to_json)["id"].to_s if response[0] == 200 end + def save_secrets(job, job_id) + return nil unless job["secrets"]? + + @redis.hash_set("id2secrets", job_id, job["secrets"]?.to_json) + job.delete("secrets") + end # add job content to es and return a response def add_job(job, job_id) + save_secrets(job, job_id) job.update_id(job_id) @es.set_job_content(job) end -- 2.23.0

1 0

[PATCH lab-z9] add muti-qemu hosts and alter the multi-qemu.yaml
by Bai Jing 23 Feb '21

23 Feb '21

Signed-off-by: Bai Jing <799286817(a)qq.com> --- allot/idle/taishan200-2280-2s48p-256g--a63 | 1 + allot/idle/taishan200-2280-2s48p-256g--a64 | 1 + allot/idle/taishan200-2280-2s48p-256g--a65 | 1 + allot/idle/taishan200-2280-2s48p-256g--a66 | 1 + allot/idle/taishan200-2280-2s48p-256g--a67 | 1 + allot/idle/taishan200-2280-2s48p-256g--a68 | 1 + allot/idle/taishan200-2280-2s48p-256g--a69 | 1 + allot/idle/taishan200-2280-2s48p-256g--a70 | 1 + .../multi-qemu.yaml | 15 ++++++++++++--- 9 files changed, 20 insertions(+), 3 deletions(-) create mode 120000 allot/idle/taishan200-2280-2s48p-256g--a63 create mode 120000 allot/idle/taishan200-2280-2s48p-256g--a64 create mode 120000 allot/idle/taishan200-2280-2s48p-256g--a65 create mode 120000 allot/idle/taishan200-2280-2s48p-256g--a66 create mode 120000 allot/idle/taishan200-2280-2s48p-256g--a67 create mode 120000 allot/idle/taishan200-2280-2s48p-256g--a68 create mode 120000 allot/idle/taishan200-2280-2s48p-256g--a69 create mode 120000 allot/idle/taishan200-2280-2s48p-256g--a70 diff --git a/allot/idle/taishan200-2280-2s48p-256g--a63 b/allot/idle/taishan200-2280-2s48p-256g--a63 new file mode 120000 index 0000000..3c3d0c3 --- /dev/null +++ b/allot/idle/taishan200-2280-2s48p-256g--a63 @@ -0,0 +1 @@ +taishan200-2280-2s48p-256g--multi-qemu \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a64 b/allot/idle/taishan200-2280-2s48p-256g--a64 new file mode 120000 index 0000000..3c3d0c3 --- /dev/null +++ b/allot/idle/taishan200-2280-2s48p-256g--a64 @@ -0,0 +1 @@ +taishan200-2280-2s48p-256g--multi-qemu \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a65 b/allot/idle/taishan200-2280-2s48p-256g--a65 new file mode 120000 index 0000000..3c3d0c3 --- /dev/null +++ b/allot/idle/taishan200-2280-2s48p-256g--a65 @@ -0,0 +1 @@ +taishan200-2280-2s48p-256g--multi-qemu \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a66 b/allot/idle/taishan200-2280-2s48p-256g--a66 new file mode 120000 index 0000000..3c3d0c3 --- /dev/null +++ b/allot/idle/taishan200-2280-2s48p-256g--a66 @@ -0,0 +1 @@ +taishan200-2280-2s48p-256g--multi-qemu \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a67 b/allot/idle/taishan200-2280-2s48p-256g--a67 new file mode 120000 index 0000000..3c3d0c3 --- /dev/null +++ b/allot/idle/taishan200-2280-2s48p-256g--a67 @@ -0,0 +1 @@ +taishan200-2280-2s48p-256g--multi-qemu \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a68 b/allot/idle/taishan200-2280-2s48p-256g--a68 new file mode 120000 index 0000000..3c3d0c3 --- /dev/null +++ b/allot/idle/taishan200-2280-2s48p-256g--a68 @@ -0,0 +1 @@ +taishan200-2280-2s48p-256g--multi-qemu \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a69 b/allot/idle/taishan200-2280-2s48p-256g--a69 new file mode 120000 index 0000000..3c3d0c3 --- /dev/null +++ b/allot/idle/taishan200-2280-2s48p-256g--a69 @@ -0,0 +1 @@ +taishan200-2280-2s48p-256g--multi-qemu \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a70 b/allot/idle/taishan200-2280-2s48p-256g--a70 new file mode 120000 index 0000000..3c3d0c3 --- /dev/null +++ b/allot/idle/taishan200-2280-2s48p-256g--a70 @@ -0,0 +1 @@ +taishan200-2280-2s48p-256g--multi-qemu \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--multi-qemu/multi-qemu.yaml b/allot/idle/taishan200-2280-2s48p-256g--multi-qemu/multi-qemu.yaml index ed55c08..87f684e 100644 --- a/allot/idle/taishan200-2280-2s48p-256g--multi-qemu/multi-qemu.yaml +++ b/allot/idle/taishan200-2280-2s48p-256g--multi-qemu/multi-qemu.yaml @@ -12,14 +12,23 @@ fs: simplify-ci: multi-qemu-0: - nr_vm: 20 - tbox_group: vm-2p8g + nr_vm: 2 + tbox_group: vm-1p1g multi-qemu-1: nr_vm: 2 tbox_group: vm-2p4g multi-qemu-2: + nr_vm: 12 + tbox_group: vm-2p8g +multi-qemu-3: + nr_vm: 6 + tbox_group: vm-2p16g +multi-qemu-4: nr_vm: 2 - tbox_group: vm-1p1g + tbox_group: vm-2p32g +multi-qemu-5: + nr_vm: 2 + tbox_group: vm-4p64g sleep: 100d os: openeuler -- 2.23.0

1 0

[PATCH lkp-tests] setup/simplify-ci: git failed to exit directly
by Cui Yili 23 Feb '21

23 Feb '21

Signed-off-by: Cui Yili <2268260388(a)qq.com> --- setup/simplify-ci | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/setup/simplify-ci b/setup/simplify-ci index c8f8a5a53..5aede9cdb 100755 --- a/setup/simplify-ci +++ b/setup/simplify-ci @@ -14,7 +14,7 @@ git_ci() EOF fi - git clone https://gitee.com/wu_fengguang/compass-ci.git /c/compass-ci + git clone https://gitee.com/wu_fengguang/compass-ci.git /c/compass-ci || return 1 } dev_env() @@ -68,7 +68,7 @@ run_service() main() { - git_ci + git_ci || return 1 install_env run_service } -- 2.23.0

1 0

[PATCH compass-ci] sbin/auto_submit: Adapting to new config style of DEFAULTS files
by Li Yuanchao 23 Feb '21

23 Feb '21

As the config style of DEFAULTS files in upstream-repos changes, the auto_submit need change to adapt to it. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- sbin/auto_submit | 53 ++++++++++++++++-------------------------------- 1 file changed, 18 insertions(+), 35 deletions(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index b408158..857af29 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -57,28 +57,13 @@ class AutoSubmit system(real_argvs.join(' ')) end - def get_branch_commands(submit_commands) - submit_commands.each do |element| - return element['branch_commands'] if element['branch_commands'] - end - return nil - end + def submit_specific_branch(submit_argv, newrefs_info, branches) + branches.each do |branch_name| + branch = "refs/heads/#{branch_name}" + next unless newrefs_info['new_refs']['heads'][branch] - def submit_specific_branch(submit_argv, newrefs_info, branch_commands) - return newrefs_info['new_refs']['heads'] if branch_commands.nil? - - non_specific_commits = {} - newrefs_info['new_refs']['heads'].each do |branch, commit_id| - branch_name = branch.delete_prefix('refs/heads/') - if branch_commands[branch_name].nil? - non_specific_commits[branch] = commit_id - next - end - argvs = Array.new(submit_argv) - argvs.push(branch_commands[branch_name]) - submit_one_job(argvs, newrefs_info['git_repo'], branch_name, commit_id) + submit_one_job(submit_argv, newrefs_info['git_repo'], branch_name, newrefs_info['new_refs']['heads'][branch]) end - return non_specific_commits end def submit_non_specific(non_specific_commits, git_repo, submit_argv) @@ -90,13 +75,13 @@ class AutoSubmit # Add support of different branch different job # newrefs_info['submit'] is like: - # ["submit command 1", - # "submit command 2", + # [ + # { "command" => "submit command 1", + # "branches" => [ "master", "next", ... ] + # }, # ..., - # { "branch_commands" => { "branch_1" => "submit command 1", - # ..., - # "branch_n" => "submit command n" - # } + # { "command" => "submit command n", + # "branches" => [ "branch_name_a", "branch_name_b", ... ] # } # ] def submit_job(newrefs_info) @@ -105,16 +90,14 @@ class AutoSubmit return unless newrefs_info['submit'] - branch_commands = get_branch_commands(newrefs_info['submit']) - non_specific_commits = submit_specific_branch(submit_argv, newrefs_info, branch_commands) - return if non_specific_commits.empty? - - newrefs_info['submit'].each do |argv_config| - next if argv_config.is_a? Hash - + newrefs_info['submit'].each do |element| argvs = Array.new(submit_argv) - argvs.push(argv_config) - submit_non_specific(non_specific_commits, newrefs_info['git_repo'], argvs) + argvs.push(element['command']) + if element['branches'] + submit_specific_branch(argvs, newrefs_info, element['branches']) + else + submit_non_specific(newrefs_info['new_refs']['heads'], newrefs_info['git_repo'], argvs) + end end end -- 2.23.0

1 0

[PATCH lkp-tests] tests/rpmbuild-pkg: redirect the sync_dest directory
by Li Ping 23 Feb '21

23 Feb '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- tests/rpmbuild-pkg | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index 9f455f0d0..38de16e66 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -6,6 +6,7 @@ [ -n "$rpm_repo" ] || die "rpm_repo is empty" package_name=${rpm_repo##*/} +[[ "$os_mount" = "cifs" ]] && os_mount="nfs" sync_dest="/initrd/rpmbuild-pkg/${os_mount}/${os}/${os_arch}/${os_version}/${package_name}" init_workspace() -- 2.23.0

1 0

[PATCH v2 lkp-tests] jobs/: add os-arch-version-mount.yaml for multi-dimension test
by Li Ping 23 Feb '21

23 Feb '21

1 1

[PATCH compass-ci] container/scheduler/start: fix read config
by Wei Jihui 23 Feb '21

23 Feb '21

scheduler service need read config from /etc/compass-ci/[service|scheduler], not only /etc/compass-ci/default. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- container/scheduler/start | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/container/scheduler/start b/container/scheduler/start index c32a7a8..4aaa71d 100755 --- a/container/scheduler/start +++ b/container/scheduler/start @@ -46,7 +46,6 @@ DEFAULT_CCI = "#{CCI_REPOS}/compass-ci" LAB = defaults['lab'] || 'z9' SCHED_PORT = defaults['SCHED_PORT'] SCHED_HOST = defaults['SCHED_HOST'] -DEFAULT_CONFIG_DIR = '/etc/compass-ci/defaults' docker_rm "s001-alpine-#{SCHED_PORT}" @@ -68,9 +67,9 @@ cmd = %W[ -e CCI_SRC=#{DEFAULT_CCI} -v #{ENV['LKP_SRC']}:#{DEFAULT_LKP} -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} - -v #{DEFAULT_CONFIG_DIR}:#{DEFAULT_CONFIG_DIR}:ro -v #{CCI_REPOS}/lab-#{LAB}:#{CCI_REPOS}/lab-#{LAB} -v /srv/cci/scheduler/alpine:/srv/cci/scheduler + -v /etc/compass-ci:/etc/compass-ci:ro -v /etc/localtime:/etc/localtime:ro -v /srv/result:/srv/result -v /srv/initrd:/srv/initrd:ro -- 2.23.0

1 0

[PATCH compass-ci] container/assign-account: optimize account login permission
by Luan Shengde 23 Feb '21

23 Feb '21

[why] the mail robot sets no keyword 'enable_login' when assigning account. the applied account via sending 'apply account' email was set no permission to login. before: set enable login: has key: enable_login key value: true set disable login has key: enable_login key value: false or has no key: enable_login now: set enable login: has key: enable_login key value: true set disable login: has key: enable_login key value: false keep the default: has no key: enable_login Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/get_account_info.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index 7e0a5a7..228d468 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -136,7 +136,7 @@ class AccountStorage def permit_login_config(login_name) if @data.key?('enable_login') && @data['enable_login'] %x(usermod -s /usr/bin/zsh #{login_name}) - else + elsif @data.key?('enable_login') && ! @data['enable_login'] %x(usermod -s /sbin/nologin #{login_name}) end end -- 2.23.0

1 0

[PATCH lab-z9] idle/*/multi-qemu.yaml: raid the added disk
by Cui Yili 23 Feb '21

23 Feb '21

Signed-off-by: Cui Yili <2268260388(a)qq.com> --- .../multi-qemu.yaml | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/allot/idle/taishan200-2280-2s48p-256g--multi-qemu/multi-qemu.yaml b/allot/idle/taishan200-2280-2s48p-256g--multi-qemu/multi-qemu.yaml index 8182a97..ed55c08 100644 --- a/allot/idle/taishan200-2280-2s48p-256g--multi-qemu/multi-qemu.yaml +++ b/allot/idle/taishan200-2280-2s48p-256g--multi-qemu/multi-qemu.yaml @@ -1,8 +1,14 @@ suite: multi-qemu category: functional -disk: 1HDD -fs: ext4 +nr_hdd: {{nr_hdd_partitions}} +disk: + +md: + - RAID0 + +fs: + - ext4 simplify-ci: multi-qemu-0: -- 2.23.0

2 1

[PATCH compass-ci] sched: delete job.cgz from result root dir
by Cao Xueliang 22 Feb '21

22 Feb '21

We has the job.yaml and job.sh in result root dir, no need to save the job.cgz. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/scheduler/create_job_cpio.cr | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/src/scheduler/create_job_cpio.cr b/src/scheduler/create_job_cpio.cr index 1101a0f..51d79f7 100644 --- a/src/scheduler/create_job_cpio.cr +++ b/src/scheduler/create_job_cpio.cr @@ -134,8 +134,7 @@ class Sched # the job.yaml is not final version files = ["#{src_dir}/job.sh", - "#{src_dir}/job.yaml", - "#{src_dir}/job.cgz"] + "#{src_dir}/job.yaml"] FileUtils.cp(files, dst_dir) end end -- 2.23.0

1 0

[PATCH lkp-tests 2/2] spec/submit/link_jobs: replace latency_stats to perf-sched
by Wei Jihui 22 Feb '21

22 Feb '21

commit 4b1eaa97a0d7dd2455929c0734ed88631b1aa263: include/category/benchmark: replace latency_stats to perf-ddsched Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- ...4-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/spec/submit/link_jobs1/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml b/spec/submit/link_jobs1/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml index 5457a131..0c3eb69d 100644 --- a/spec/submit/link_jobs1/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml +++ b/spec/submit/link_jobs1/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml @@ -51,7 +51,7 @@ meminfo: slabinfo: interrupts: lock_stat: -latency_stats: +perf-sched: softirqs: bdi_dev_mapping: diskstats: @@ -103,7 +103,7 @@ monitors: slabinfo: interrupts: lock_stat: - latency_stats: + perf-sched: softirqs: bdi_dev_mapping: diskstats: -- 2.23.0

1 0

[PATCH lkp-tests 1/2] spec/submit: update submit spec result
by Wei Jihui 22 Feb '21

22 Feb '21

all vm testboxs add hdd_partitions Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- spec/submit/job_on_fail/job-3000.yaml | 1 + ...1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml | 1 + spec/submit/link_jobs2/job-1000-2-100000ss-add.yaml | 1 + spec/submit/link_jobs2/job-10000-2-100000ss-add.yaml | 1 + spec/submit/link_jobs2/job-20-2-6000ss-cache.yaml | 1 + spec/submit/matrix/job-1-300-whetstone-double.yaml | 1 + spec/submit/matrix/job-3-300-shell1.yaml | 1 + spec/submit/merge_yaml/job-defaults.yaml | 1 + spec/submit/separate_yaml/job-1000-2-100000ss-add.yaml | 1 + spec/submit/separate_yaml/job-10000-2-100000ss-add.yaml | 1 + spec/submit/separate_yaml/job-20-2-6000ss-cache.yaml | 1 + 11 files changed, 11 insertions(+) diff --git a/spec/submit/job_on_fail/job-3000.yaml b/spec/submit/job_on_fail/job-3000.yaml index 11d89b2b..5d766ba0 100644 --- a/spec/submit/job_on_fail/job-3000.yaml +++ b/spec/submit/job_on_fail/job-3000.yaml @@ -11,6 +11,7 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 8G +hdd_partitions: "/dev/vda" arch: aarch64 cpufreq_governor: need_kconfig: diff --git a/spec/submit/link_jobs1/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml b/spec/submit/link_jobs1/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml index 466a2054..5457a131 100644 --- a/spec/submit/link_jobs1/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml +++ b/spec/submit/link_jobs1/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml @@ -34,6 +34,7 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 8G +hdd_partitions: "/dev/vda" arch: aarch64 kmsg: boot-time: diff --git a/spec/submit/link_jobs2/job-1000-2-100000ss-add.yaml b/spec/submit/link_jobs2/job-1000-2-100000ss-add.yaml index 9a613c00..d20eed7e 100644 --- a/spec/submit/link_jobs2/job-1000-2-100000ss-add.yaml +++ b/spec/submit/link_jobs2/job-1000-2-100000ss-add.yaml @@ -15,6 +15,7 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 8G +hdd_partitions: "/dev/vda" arch: aarch64 cpufreq_governor: need_kconfig: diff --git a/spec/submit/link_jobs2/job-10000-2-100000ss-add.yaml b/spec/submit/link_jobs2/job-10000-2-100000ss-add.yaml index 8bf52d0f..8df242aa 100644 --- a/spec/submit/link_jobs2/job-10000-2-100000ss-add.yaml +++ b/spec/submit/link_jobs2/job-10000-2-100000ss-add.yaml @@ -15,6 +15,7 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 8G +hdd_partitions: "/dev/vda" arch: aarch64 cpufreq_governor: need_kconfig: diff --git a/spec/submit/link_jobs2/job-20-2-6000ss-cache.yaml b/spec/submit/link_jobs2/job-20-2-6000ss-cache.yaml index 8cb78287..cf3a162b 100644 --- a/spec/submit/link_jobs2/job-20-2-6000ss-cache.yaml +++ b/spec/submit/link_jobs2/job-20-2-6000ss-cache.yaml @@ -15,6 +15,7 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 8G +hdd_partitions: "/dev/vda" arch: aarch64 cpufreq_governor: need_kconfig: diff --git a/spec/submit/matrix/job-1-300-whetstone-double.yaml b/spec/submit/matrix/job-1-300-whetstone-double.yaml index 78f79fa7..4ba40ba5 100644 --- a/spec/submit/matrix/job-1-300-whetstone-double.yaml +++ b/spec/submit/matrix/job-1-300-whetstone-double.yaml @@ -12,6 +12,7 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 8G +hdd_partitions: "/dev/vda" arch: aarch64 cpufreq_governor: need_kconfig: diff --git a/spec/submit/matrix/job-3-300-shell1.yaml b/spec/submit/matrix/job-3-300-shell1.yaml index afe5a953..294cbdd3 100644 --- a/spec/submit/matrix/job-3-300-shell1.yaml +++ b/spec/submit/matrix/job-3-300-shell1.yaml @@ -12,6 +12,7 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 8G +hdd_partitions: "/dev/vda" arch: aarch64 cpufreq_governor: need_kconfig: diff --git a/spec/submit/merge_yaml/job-defaults.yaml b/spec/submit/merge_yaml/job-defaults.yaml index f72fd6ac..ae2f3c39 100644 --- a/spec/submit/merge_yaml/job-defaults.yaml +++ b/spec/submit/merge_yaml/job-defaults.yaml @@ -9,6 +9,7 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 8G +hdd_partitions: "/dev/vda" arch: aarch64 cpufreq_governor: need_kconfig: diff --git a/spec/submit/separate_yaml/job-1000-2-100000ss-add.yaml b/spec/submit/separate_yaml/job-1000-2-100000ss-add.yaml index a2355b50..b7b0ef0e 100644 --- a/spec/submit/separate_yaml/job-1000-2-100000ss-add.yaml +++ b/spec/submit/separate_yaml/job-1000-2-100000ss-add.yaml @@ -13,6 +13,7 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 8G +hdd_partitions: "/dev/vda" arch: aarch64 cpufreq_governor: need_kconfig: diff --git a/spec/submit/separate_yaml/job-10000-2-100000ss-add.yaml b/spec/submit/separate_yaml/job-10000-2-100000ss-add.yaml index 8a6317cb..144fba5a 100644 --- a/spec/submit/separate_yaml/job-10000-2-100000ss-add.yaml +++ b/spec/submit/separate_yaml/job-10000-2-100000ss-add.yaml @@ -13,6 +13,7 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 8G +hdd_partitions: "/dev/vda" arch: aarch64 cpufreq_governor: need_kconfig: diff --git a/spec/submit/separate_yaml/job-20-2-6000ss-cache.yaml b/spec/submit/separate_yaml/job-20-2-6000ss-cache.yaml index 15701ad0..74c4e380 100644 --- a/spec/submit/separate_yaml/job-20-2-6000ss-cache.yaml +++ b/spec/submit/separate_yaml/job-20-2-6000ss-cache.yaml @@ -13,6 +13,7 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 8G +hdd_partitions: "/dev/vda" arch: aarch64 cpufreq_governor: need_kconfig: -- 2.23.0

1 0

[PATCH compass-ci] lib/git_mirror.rb: only store necessary info to es
by Li Yuanchao 22 Feb '21

22 Feb '21

As the config become more, the old way will store more and more info to es, and they are not all necessary, and their changes sometimes will cause conflict with old data in es. So only store necessary info now. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 346352b..76cdd1b 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -288,8 +288,8 @@ class MirrorMain end def es_repo_update(git_repo) - repo_info = @git_info[git_repo].merge(@fork_stat[git_repo]) - repo_info.delete(:cur_refs) if repo_info.key?(:cur_refs) + repo_info = { 'git_repo' => git_repo, 'url' => @git_info[git_repo]['url'] } + repo_info = repo_info.merge(@fork_stat[git_repo]) body = { "doc": repo_info, "doc_as_upsert": true -- 2.23.0

1 0

[PATCH v3 compass-ci 4/4] container/mail-robot: update mail model
by Luan Shengde 22 Feb '21

22 Feb '21

[why] key work is renamed: 'my oss commit' => 'my_oss_commit' Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/assign-account-fail-eamil.rb | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/container/mail-robot/lib/assign-account-fail-eamil.rb b/container/mail-robot/lib/assign-account-fail-eamil.rb index 3597159..3169235 100755 --- a/container/mail-robot/lib/assign-account-fail-eamil.rb +++ b/container/mail-robot/lib/assign-account-fail-eamil.rb @@ -11,17 +11,17 @@ def email_err_message(message) You should add a commit url for the 'apply account' email. For example: - my oss commit: https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… + my_oss_commit: https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… EMAIL_MESSAGE when 'URL_PREFIX_ERR' err_message = <<~EMAIL_MESSAGE Please add a correct prefix for the commit url. - my oss commit + my_oss_commit For example: - my oss commit: https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… + my_oss_commit: https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… EMAIL_MESSAGE when 'NOT_REGISTERED' err_message = <<~EMAIL_MESSAGE -- 2.23.0

1 0

[PATCH v3 compass-ci 3/4] container/mail-robot: add login message
by Luan Shengde 22 Feb '21

22 Feb '21

add login command for account VM server. case forwarded user info has key account_vm with value: yes/true, the email will send the login command. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../mail-robot/lib/assign-account-email.rb | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/container/mail-robot/lib/assign-account-email.rb b/container/mail-robot/lib/assign-account-email.rb index b71b3fe..2171de7 100755 --- a/container/mail-robot/lib/assign-account-email.rb +++ b/container/mail-robot/lib/assign-account-email.rb @@ -3,13 +3,26 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true -def build_apply_account_email(my_info) +def build_apply_account_email(my_info, account_info, assign_account_vm) email_greeting = if my_info['bisect'] 'We have automatically created the following information for you.' else 'Thank you for joining us.' end + login_msg = <<~LOGIN_MSG + + You can use the following command to login the jumper server: + + Login command: + ssh -p #{account_info['jumper_port']} #{account_info['my_login_name']}@#{account_info['jumper_host']} + + Account password: + #{account_info['my_password']} + LOGIN_MSG + + login_account_vm = assign_account_vm ? login_msg : '' + email_msg = <<~EMAIL_MESSAGE To: #{my_info['my_email']} Subject: [compass-ci] Account Ready @@ -17,8 +30,8 @@ def build_apply_account_email(my_info) Dear #{my_info['my_name']}, #{email_greeting} - - You can use the following info to submit jobs: + #{login_account_vm} + You need to do the following configurations before submitting a job: notice: (1-2) are ONE-TIME setup -- 2.23.0

1 0

[PATCH v3 compass-ci 2/4] container/mail-robot: add parsing forwarded email
by Luan Shengde 22 Feb '21

22 Feb '21

for forwarded 'apply account' email, my_email/my_name is store in a fixed format in the mail content. extract a list of users info from the mail content update regular expression: 'my oss commit' is renamed to 'my_oss_commit'. both support 'my oss commit' and 'my_oss_commit' Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../lib/parse-apply-account-email.rb | 21 +++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-) diff --git a/container/mail-robot/lib/parse-apply-account-email.rb b/container/mail-robot/lib/parse-apply-account-email.rb index 92db1da..272f1a1 100755 --- a/container/mail-robot/lib/parse-apply-account-email.rb +++ b/container/mail-robot/lib/parse-apply-account-email.rb @@ -57,18 +57,27 @@ class ParseApplyAccountEmail end def extract_users - users = extract_mail_content_body.split(/\r|\n/) + users_info = [] + + users = extract_mail_content_body.split(/---+/) users.delete('') - users + users.each do |user| + user_info = YAML.safe_load(user) + + users_info << user_info if user_info.include?('my_email') + end + + return users_info end def extract_commit_url mail_content_line = extract_mail_content_body.gsub(/\n/, '') - # the commit url should be headed with a prefix: my oss commit + # the commit url should be headed with a prefix: my_oss_commit # the commit url should be in a standart format, example: - # my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… - unless mail_content_line.match?(%r{my oss commit:\s*https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}}) - raise 'URL_PREFIX_ERR' unless mail_content_line.match?(%r{my oss commit:\s*https?://}) + # my_oss_commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… + # the prefix is renamed to 'my_oss_commit', but 'my oss commit' is still supportted. + unless mail_content_line.match?(%r{my[ _]oss[ _]commit:\s*https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}}) + raise 'URL_PREFIX_ERR' unless mail_content_line.match?(%r{my[ _]oss[ _]commit:\s*https?://}) raise 'NO_COMMIT_URL' unless mail_content_line.match?(%r{https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}}) end -- 2.23.0

1 0

[PATCH v3 compass-ci 1/4] container/mail-robot: add handle forwarded my_name/my_email(s)
by Luan Shengde 22 Feb '21

22 Feb '21

[why] the forwarded email from account issuer may will contain my_name/my_email(s). it allows contain multi my_name/my_email(s), just loop them. the forwarded email content may like: --- my_name: name_1 my_email: email_1 account_vm: true/false/yes/no --- my_name: name_2 my_email: email_2 --- available values for key_word: account_vm - yes/no - true/false only if the section for a user has a key_word account_vm, and the key_word has a value: yes/true, the replied email from mail robot will contains the login command for account vm server. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/apply-account.rb | 66 ++++++++++++++++++++--- 1 file changed, 58 insertions(+), 8 deletions(-) diff --git a/container/mail-robot/lib/apply-account.rb b/container/mail-robot/lib/apply-account.rb index e8288d7..f3f61c3 100755 --- a/container/mail-robot/lib/apply-account.rb +++ b/container/mail-robot/lib/apply-account.rb @@ -55,6 +55,9 @@ class ApplyAccount @es_host = @send_mail_host @es_port = ES_PORT + # email info file for account issuers. + @account_issuer = YAML.load_file '/etc/compass-ci/account-issuer.yaml' + @my_info = {} end @@ -63,11 +66,36 @@ class ApplyAccount # in order to successfully send email for failed parsing # firstly get my_email before execute parse_mail_content is needed @my_info['my_email'] = @mail_content.from[0] - parse_mail_content - apply_my_account - store_account_info - send_mail('') + # for the forwarded email, it contains my_email/my_name .etc. + # all emails from account issuer will be treated as forwarded emails. + # the mail_content may like: + # --- + # my_name: name_1 + # my_name: email_1 + # account_vm: true/false/yes/no + # --- + # my_name: name_2 + # my_email: email_2 + # --- + # the forwarded email allowed to contain multi my_email/my_name(s) + # in this case, we will loop them + if @account_issuer.include? @my_info['my_email'] + users_info = forward_users + users_info.each do |user_info| + # for forwarded email for multi users, avoid rezidual information from the last, + # need to clear the old data for my_info. + @my_info.clear + assign_account_vm = user_info.delete('account_vm') + @my_info.update user_info + + applying_account(assign_account_vm) + sleep 5 + end + else + parse_mail_content + applying_account(false) + end rescue StandardError => e puts e.message puts e.backtrace @@ -75,6 +103,24 @@ class ApplyAccount send_mail(e.message) end + def forward_users + forward_email_content = ParseApplyAccountEmail.new(@mail_content) + users_info = forward_email_content.extract_users + + users_info.clone.each_index do |i| + users_info[i]['my_ssh_pubkey'] = [] + users_info[i]['lab'] = ENV['lab'] + end + + users_info + end + + def applying_account(assign_account_vm) + account_info = apply_my_account + store_account_info + send_mail('', account_info, assign_account_vm) + end + def parse_mail_content parse_apply_account_email = ParseApplyAccountEmail.new(@mail_content) @@ -93,7 +139,9 @@ class ApplyAccount apply_info['my_token'] = my_account_es['my_uuid'] if my_account_es['my_token'].nil? apply_info.update my_account_es apply_info.update @my_info - apply_info['my_ssh_pubkey'] = (apply_info['my_ssh_pubkey'] + my_ssh_pubkey_new).uniq + if my_ssh_pubkey_new + apply_info['my_ssh_pubkey'] = (apply_info['my_ssh_pubkey'] + my_ssh_pubkey_new).uniq + end @my_info.update apply_info apply_info['is_update_account'] = true apply_info @@ -102,14 +150,15 @@ class ApplyAccount def apply_my_account my_account_es = read_my_account_es apply_info = {} + if my_account_es build_apply_info(apply_info, my_account_es) else my_token = %x(uuidgen).chomp @my_info['my_token'] = my_token apply_info.update @my_info + apply_info['enable_login'] = true end - apply_info['lab'] = ENV['lab'] apply_new_account(apply_info, my_account_es) end @@ -118,6 +167,7 @@ class ApplyAccount acct_info = apply_account.apply_jumper_account @my_info['my_login_name'] = acct_info['my_login_name'] unless my_account_es + acct_info end def store_account_info @@ -125,9 +175,9 @@ class ApplyAccount es.put_source_by_id(@my_info['my_email'], @my_info) end - def send_mail(error_message) + def send_mail(error_message, account_info, assign_account_vm) email_message = if error_message.empty? - build_apply_account_email(@my_info) + build_apply_account_email(@my_info, account_info, assign_account_vm) else build_apply_account_fail_email(@my_info, error_message) end -- 2.23.0

1 0

[PATCH lkp-tests] jobs/: add os-arch-version-mount.yaml for multi-dimension test
by Li Ping 22 Feb '21

22 Feb '21

1 0

[PATCH compass-ci] sbin/compare: update help message
by Lu Weitao 22 Feb '21

22 Feb '21

[Why] the option -d/--dimensions no specific instructions [How] Usage: ... options: -d, --dimension dimensions this option activate group compare mode, require filter and dimenssion filter like: os=openeuler,centos suite=iperf dimensions like: "tbox_group os_arch os ..." Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/compare | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/sbin/compare b/sbin/compare index e8d1ac4..2ee3268 100755 --- a/sbin/compare +++ b/sbin/compare @@ -11,8 +11,8 @@ # compare "id=6000,6001" "id=7000,7001" # compare "commit=a12d232e" "commit=b3bacc31" # compare "os=debian" "os=centos" -c "suite=iperf" -# compare "os=centos" -d "os_version os_arch" -# compare "os=centos" -d "os_version os_arch" --theme="classic" +# compare os=centos -d "os_version os_arch" +# compare os=centos suite=iperf -d "os_version os_arch" --theme="classic" require 'optparse' require_relative '../lib/compare.rb' @@ -37,7 +37,8 @@ opt_parser = OptionParser.new do |opts| common_conditions = c end - opts.on('-d', '--dimension dimensions', 'dimensions to group compare: "tbox_group os_arch os ..."') do |d| + opts.on('-d', '--dimension dimensions', 'this option activate group compare mode, require filter and dimenssion', + 'filter like: os=openeuler,centos suite=iperf', 'dimensions like: "tbox_group os_arch os ..."') do |d| dimensions = d is_group = true end -- 2.23.0

1 0

[PATCH compass-ci] container/fluentd-base/Dockerfile: fix failed to build container
by Liu Yinsi 22 Feb '21

22 Feb '21

[error] root@taishan200-2280-2s64p-256g--a7 /c/compass-ci/container/fluentd-base# ./build ... ERROR: Error installing async: ERROR: Failed to build gem native extension. current directory: /usr/lib/ruby/gems/2.6.0/gems/nio4r-2.5.5/ext/nio4r /usr/bin/ruby -I /usr/lib/ruby/2.6.0 -r ./siteconf20210222-482-mws95t.rb extconf.rb checking for unistd.h... yes checking for linux/aio_abi.h... no checking for linux/io_uring.h... no checking for sys/select.h... yes checking for port_event_t in poll.h... no checking for sys/epoll.h... yes checking for sys/event.h... no checking for port_event_t in port.h... no checking for sys/resource.h... yes creating Makefile current directory: /usr/lib/ruby/gems/2.6.0/gems/nio4r-2.5.5/ext/nio4r make "DESTDIR=" clean current directory: /usr/lib/ruby/gems/2.6.0/gems/nio4r-2.5.5/ext/nio4r make "DESTDIR=" compiling bytebuffer.c compiling monitor.c compiling nio4r_ext.c In file included from nio4r_ext.c:6: ../libev/ev.c:479:11: fatal error: linux/version.h: No such file or directory 479 | # include <linux/version.h> | ^~~~~~~~~~~~~~~~~ compilation terminated. make: *** [Makefile:245: nio4r_ext.o] Error 1 make failed, exit code 2 Gem files will remain installed in /usr/lib/ruby/gems/2.6.0/gems/nio4r-2.5.5 for inspection. Results logged to /usr/lib/ruby/gems/2.6.0/extensions/aarch64-linux/2.6.0/nio4r-2.5.5/gem_make.out The command '/bin/sh -c umask 002 && gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && gem install fluentd && gem install fluent-plugin-rabbitmq && gem install fluent-plugin-elasticsearch && gem install fluent-plugin-tail-ex && gem install fluent-plugin-tail-multiline && gem install json && gem install async && gem install webrick && gem install io-console && gem install etc' returned a non-zero code: 1 [how] 'gem install async' in Dockerfile cause error, apk add linux-headers can fix it. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/fluentd-base/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/fluentd-base/Dockerfile b/container/fluentd-base/Dockerfile index 08bcd9c..2d181f5 100644 --- a/container/fluentd-base/Dockerfile +++ b/container/fluentd-base/Dockerfile @@ -10,7 +10,7 @@ RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweiclou RUN apk update RUN apk add --no-cache 'ruby-dev' \ - 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'cpio' 'bash' + 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'cpio' 'bash' 'linux-headers' RUN umask 002 && \ gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && \ -- 2.23.0

1 0

[PATCH v2 compass-ci] doc: optimize os_mount.md
by Yu Chuan 22 Feb '21

22 Feb '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- doc/job/os_mount.md | 207 +++++++++++++++++++++++++++++--------------- 1 file changed, 137 insertions(+), 70 deletions(-) diff --git a/doc/job/os_mount.md b/doc/job/os_mount.md index 226251a8bace..ebe8566b9443 100644 --- a/doc/job/os_mount.md +++ b/doc/job/os_mount.md @@ -1,9 +1,9 @@ # Summary ========= -`os_mount` defines the mount method of testbox's root partition. +`os_mount` defines the rootfs type. -It has the following optional values: +The below types are supported: - nfs - cifs - initramfs @@ -11,84 +11,151 @@ It has the following optional values: - local Usage example: - - submit iperf.yaml testbox=vm-2p8g os_mount=nfs - - submit iperf.yaml testbox=vm-2p8g os_mount=cifs - - submit iperf.yaml testbox=vm-2p8g os_mount=initramfs - - submit iperf.yaml testbox=dc-8g os_mount=container - - submit iperf.yaml testbox=vm-2p8g os_mount=local - -# Optional Values -================= - -## Work flow when os_mount=local - -1. user submit job with os_mount: local - - optional {kernel_append_root}: - - save_root_partition: give an iconic string, then the root partition data used by this job will be retained. - and you can use the root partition data of this job in the futher job by {use_root_partition}. - - use_root_partition : give an iconic string, then the root partition data generated by the - pervious job which has the {save_root_partition} field will be used. - - root_partition_size: specify the size of root partition, *default is 10G*. + ``` + submit -m -c borrow-1h.yaml testbox=vm-2p8g os_mount=nfs + submit -m -c borrow-1h.yaml testbox=vm-2p8g os_mount=cifs + submit -m -c borrow-1h.yaml testbox=vm-2p16g os_mount=initramfs + submit -m -c borrow-1h.yaml testbox=dc-8g os_mount=container + submit -m -c borrow-1h.yaml testbox=taishan200-2280-2s48p-256g os_mount=local + ``` - - tips: - - value of {save_root_partition} and {use_root_partition} need to be iconic enough, uuid is recommended. - - value of {save_root_partition} and {use_root_partition} will be the suffix of logical volume. +# os_mount=local +================ + +## description + +If you specifies os_mount=local, the rootfs type of your job will be a logical volume in the testbox. + +> logical volume format: `/dev/mapper/os-${os}_${os_arch}_${os_version}` + +> optional vars in ${kernel_append_root}: + - ${save_root_partition}, ${use_root_partition} + - purpose: + - when os_mount=local, rootfs is stored in the logical volume on the disk, + but it will be wiped by default in the preprocessing action performed by each job. + just is initrd stage. + - so we need a solution to solve scenarios that rootfs data needs to be persisted. + - how to implement: + - design a unique string for your job, it will be used as a unique symbol for your job. + the rootfs data of your job will be persistently saved to the corresponding testbox. + - so, when you need to use the rootfs data of this job in subsequent jobs, you only need + to use this unique string to submit subsequent-job.yaml, and then the subsequent-job.yaml + will be executed based on the prsistent rootfs data. + - how to use: + - in 20210218, you submit a job-20210218.yaml, and you want to persist the rootfs data of + this job-20210218 so that it can be used by the subsequent jobs. So you need add the + follow field in your job-20210218.yaml. + ``` + kernel_append_root: save_root_partition=zhangsan_local_for_iperf_20210218 + ``` + - in 20210219, you submit a job-20210219.yaml, and you want to use the rootfs data of + job-20210218.yaml, So you need add the follow field in your job-20210219.yaml + ``` + kernel_append_root: use_root_partition=zhangsan_local_for_iperf_20210218 + ``` + - description: + - ${save_root_partition} and ${use_root_partition} must be the same, and obviously, they need be unique. + - you must 'save_root_partition' firstly, then you can 'use_root_partition'. otherwise, your job will fail. + +## work flow + +1. user submit job + job fields: + ``` + os: openeuler + os_arch: aarch64 + os_version: 20.03 + os_mount: local + kernel_append_root: use_root_partition=zhangsan_local_for_iperf_20210218 save_root_partition=zhangsan_local_for_iperf_20210219 + ``` 2. scheduler return the ipxe_str to testbox ``` dhcp - initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/{timestamp}/initrd.lkp - initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/{timestamp}/boot/vmlinuz - imgargs vmlinux root=/dev/mapper/os-openeuler_aarch64_20.03 rootfs_src={nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/{timestamp} initrd=initrd.lkp {kernel_append_root} + initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/${timestamp}/initrd.lkp + kernel http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/${timestamp}/boot/vmlinuz root=/dev/mapper/os-openeuler_aarch64_20.03 rootfs_src=${nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/${timestamp} initrd=initrd.lkp ${kernel_append_root} boot ``` 3. dracut step of boot ``` - if have {use_root_partition}; then - base_lv_name=openeuler_aarch64_20.03_{value_of_use_root_partition} - lvdisplay /dev/mappaer/os-{base_lv_name} > /dev/null || exit 1 - - if have {save_root_partition}; then - boot_lv_name=openeuler_aarch64_20.03_{value_of_save_root_partition} - - if {save_root_partition} == {use_root_partition}; then - boot from /dev/mappaer/os-{boot_lv_name} - fi - - lvremove -f /dev/mappaer/os-{boot_lv_name} - lvcreate /dev/mappaer/os-{boot_lv_name} from /dev/mappaer/os-{base_lv_name} || exit 1 - boot from /dev/mappaer/os-{boot_lv_name} - else - boot_lv_name=openeuler_aarch64_20.03_{value_of_use_root_partition} - boot from /dev/mappaer/os-{boot_lv_name} - fi - - else: - base_lv_name=openeuler_aarch64_20.03_{timestamp} - lvdisplay /dev/mappaer/os-{base_lv_name} > /dev/null || { - # create logical volume - lvcreate -L {root_partition_size} -n {base_lv_name} os || exit 1 - - # rsync nfsroot to lvm:/dev/mapper/os-{base_lv_name} - mount -t nfs {nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/{timestamp} /mnt - mkdir /mnt1 && mount /dev/mapper/os-{base_lv_name} /mnt1 - cp -a /mnt/. /mnt1/ - umount /mnt /mnt1 - - # change premission of lvm:/dev/mapper/os-{base_lv_name} to readonly - lvchange -p r /dev/mapper/os-{base_lv_name} - } - - boot_lv_name=openeuler_aarch64_20.03 - if have {save_root_partition}; then - boot_lv_name=openeuler_aarch64_20.03_{value_of_save_root_partition} - fi - - lvremove -f /dev/mappaer/os-{boot_lv_name} - lvcreate /dev/mappaer/os-{boot_lv_name} from /dev/mappaer/os-{base_lv_name} || exit 1 - boot from /dev/mappaer/os-{boot_lv_name} - fi + rsync_src_lv() + { + lvdisplay ${SRC_LV} > /dev/null && return + + # create logical volume + lvcreate --size 10G --name $(basename ${SRC_LV}) os || exit + + # rsync nfsroot to ${SRC_LV} + mount -t nfs ${nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/${timestamp} /mnt + mkdir /mnt1 && mount ${SRC_LV} /mnt1 + cp -a /mnt/. /mnt1/ + umount /mnt /mnt1 + + # change permission of ${SRC_LV} to readonly + lvchange --permission r ${SRC_LV} + } + + check_src_lv() + { + lvdisplay ${SRC_LV} > /dev/null || exit + } + + snapshot_boot_lv() + { + [ "$SRC_LV" == "$BOOT_LV" ] && return + + lvremove --force ${BOOT_LV} + lvcreate --size 10G --name $(basename ${BOOT_LV}) --snapshot ${SRC_LV} || exit + } + + no_save_no_use() + { + export SRC_LV="/dev/mapper/os-openeuler_aarch64_20.03_${timestamp}" + export BOOT_LV="/dev/mapper/os-openeuler_aarch64_20.03" + + rsync_src_lv ${SRC_LV} + } + + only_save() + { + export SRC_LV="/dev/mapper/os-openeuler_aarch64_20.03_${timestamp}" + export BOOT_LV="/dev/mapper/os-openeuler_aarch64_20.03_${save_root_partition}" + + rsync_src_lv ${SRC_LV} + } + + only_use() + { + export SRC_LV="/dev/mapper/os-openeuler_aarch64_20.03_${use_root_partition}" + export BOOT_LV="/dev/mapper/os-openeuler_aarch64_20.03" + + check_src_lv ${SRC_LV} + } + + save_and_use() + { + export SRC_LV="/dev/mapper/os-openeuler_aarch64_20.03_${use_root_partition}" + export BOOT_LV="/dev/mapper/os-openeuler_aarch64_20.03_${save_root_partition}" + + check_src_lv ${SRC_LV} + } + + main() + { + [ -z ${use_root_partition} ] && [ -z ${save_root_partition} ] && no_save_no_use + + [ -z ${use_root_partition} ] && [ -n ${save_root_partition} ] && only_save + + [ -n ${use_root_partition} ] && [ -z ${save_root_partition} ] && only_use + + [ -n ${use_root_partition} ] && [ -n ${save_root_partition} ] && save_and_use + + snapshot_boot_lv + boot_from_lv + } + + main() ``` 4. execute the job -- 2.23.0

2 4

[PATCH compass-ci] doc: update document
by Luan Shengde 22 Feb '21

22 Feb '21

rename key word: my oss commit => my_oss_commit Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/mail-robot.md | 4 ++-- doc/manual/apply-account.md | 6 +++--- doc/manual/apply-account.zh.md | 6 +++--- 3 files changed, 8 insertions(+), 8 deletions(-) diff --git a/doc/mail-robot.md b/doc/mail-robot.md index 6158b65..4d38e78 100644 --- a/doc/mail-robot.md +++ b/doc/mail-robot.md @@ -17,9 +17,9 @@ send mail to: compass-ci-robot(a)qq.com subject: apply account mail content - my oss commit: commit_url + my_oss_commit: commit_url example: - my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… + my_oss_commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… attahcment ssh pub_key diff --git a/doc/manual/apply-account.md b/doc/manual/apply-account.md index 974fe63..542251d 100644 --- a/doc/manual/apply-account.md +++ b/doc/manual/apply-account.md @@ -4,7 +4,7 @@ You'll need an account to submit jobs. The account is mainly for tracking your r We have limited pool of testboxes, so there have to be some usage control. We offer free accounts to OSS (Open Source Software) contributors where the OSS project is in our test coverage. -You can prove yourself as OSS contributor by providing "my oss commit" in the below email. +You can prove yourself as OSS contributor by providing "my_oss_commit" in the below email. We offer free accounts to our collaborators too. If you are one of our collaborators, you should know the collaboration channel to apply account. @@ -19,7 +19,7 @@ Email template: To: compass-ci-robot(a)qq.com Subject: apply account - my oss commit: {{ https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… }} + my_oss_commit: {{ https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… }} {{ attach your ssh pubkey file(s), e.g. ~/.ssh/id_rsa.pub }} @@ -29,7 +29,7 @@ Email template: - the email name should be in English or Chinese Pinyin, as: "David Rientjes" in the template. -- my oss commit: a git commit URL that has your name and email. +- my_oss_commit: a git commit URL that has your name and email. The git repo should be in our test coverage and registered in [upstream-repos](https://gitee.com/wu_fengguang/upstream-repos). If your contributed OSS project is not in upstream-repos, you may refer to diff --git a/doc/manual/apply-account.zh.md b/doc/manual/apply-account.zh.md index 5e6605d..849364f 100644 --- a/doc/manual/apply-account.zh.md +++ b/doc/manual/apply-account.zh.md @@ -4,7 +4,7 @@ 因为目前我们的测试机资源池有限，因此需要做一些使用限制。我们为平台测试所覆盖的开源软件（OSS）的贡献者提供了免费的账号。 -你可以通过在下面邮件中提供‘my oss commit’来验明正身您是一个开源软件（OSS）的贡献者。 +你可以通过在下面邮件中提供‘my_oss_commit’来验明正身您是一个开源软件（OSS）的贡献者。我们同时也为我们的合作者提供了免费的账号。这种情况需要提供用的邮件，名字，ssh公钥，以及申请测试机的目的。 @@ -18,7 +18,7 @@ 收件人: compass-ci-robot(a)qq.com 邮件标题: apply account - my oss commit: {{ https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… }} + my_oss_commit: {{ https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… }} {{ 将您的ssh公钥添加为邮件附件，例如 ~/.ssh/id_rsa.pub }} @@ -26,7 +26,7 @@ - 使用您的信息替换{{ }}中的内容 - 邮件名应该使用英文或中文拼音，例如模板中的“David Rientjes” -- my oss commit: 使用您的邮件和名字提交的代码的地址。 +- my_oss_commit: 使用您的邮件和名字提交的代码的地址。您的仓库需要在平台测试覆盖的范围内，并且注册到： -- 2.23.0

1 0

[PATCH lkp-tests] distro/depends/sysbench-mysql
by Wei Jihui 22 Feb '21

22 Feb '21

1. when pack pkg/sysbench-mysql, it need mysql so add libmariadb-dev(mariadb-devel) at sysbench-mysql-dev 2. when exec test/sysbench-mysql, it will install mysql at client, no need install as depends, so delete libmariadb3 at sysbench-mysql Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- distro/depends/sysbench-mysql | 1 - distro/depends/sysbench-mysql-dev | 3 ++- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/distro/depends/sysbench-mysql b/distro/depends/sysbench-mysql index b0d33ffe..a248fd07 100644 --- a/distro/depends/sysbench-mysql +++ b/distro/depends/sysbench-mysql @@ -1,4 +1,3 @@ libnuma1 xfsprogs ethtool -libmariadb3 diff --git a/distro/depends/sysbench-mysql-dev b/distro/depends/sysbench-mysql-dev index 65633b33..992db26d 100644 --- a/distro/depends/sysbench-mysql-dev +++ b/distro/depends/sysbench-mysql-dev @@ -1,2 +1,3 @@ +libtool pkg-config -mysql-server +libmariadb-dev -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: fix fs parameter parse
by Wei Jihui 22 Feb '21

22 Feb '21

[input] fs: ext4 [error] this fs will parse error by lkp-tests/params/fs fs: fs: &1 fs: ext4 [why] job.rb will parse fs twice by lkp-test/params/fs: /(.*)/{ fs: $1} this will create two fs. [how] only parse all parameters once, that is only exec expand_params once. not at the first time is for job should split array params at the first time. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/job.rb b/lib/job.rb index eacb7ec9..7fc7cb5b 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -447,7 +447,7 @@ class Job job = deepcopy(@job) job['___'] = nil @jobx = job - if @job.include?(:expand_params) + if @job.include?(:expand_params) && !first_time expand_params @job = deepcopy(@jobx) @job.delete('___') -- 2.23.0

2 1

[PATCH v2 compass-ci 1/4] container/mail-robot: add handle forwarded my_name/my_email(s)
by Luan Shengde 22 Feb '21

22 Feb '21

[why] for forwarded email from account issuer, the email content will contain my_name/my_email(s). a forwarded email may contain multi my_name/my_email(s), just loop them. the forwarded email content may like: --- my_name: name_1 my_email: email_1 account_vm: true/false/yes/no --- my_name: name_2 my_email: email_2 --- the replied 'apply account' email will return login command for account VM server if account_vm has a value: yes/true add store lab to ES Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/apply-account.rb | 65 ++++++++++++++++++++--- 1 file changed, 57 insertions(+), 8 deletions(-) diff --git a/container/mail-robot/lib/apply-account.rb b/container/mail-robot/lib/apply-account.rb index e8288d7..818f900 100755 --- a/container/mail-robot/lib/apply-account.rb +++ b/container/mail-robot/lib/apply-account.rb @@ -55,6 +55,9 @@ class ApplyAccount @es_host = @send_mail_host @es_port = ES_PORT + # email info file for account issuers. + @account_issuer = YAML.load_file '/etc/compass-ci/account-issuer.yaml' + @my_info = {} end @@ -63,11 +66,35 @@ class ApplyAccount # in order to successfully send email for failed parsing # firstly get my_email before execute parse_mail_content is needed @my_info['my_email'] = @mail_content.from[0] - parse_mail_content - apply_my_account - store_account_info - send_mail('') + # for the forwarded email, it contains my_email/my_name .etc. + # all emails from account issuer will be treated as forwarded emails. + # the mail_content may like: + # --- + # my_name: name_1 + # my_name: email_1 + # account_vm: true/false/yes/no + # --- + # my_name: name_2 + # my_email: email_2 + # --- + # the forwarded email allowed to contain multi my_email/my_name(s) + # in this case, we will loop them + if @account_issuer.include? @my_info['my_email'] + users_info = forward_users + users_info.each do |user_info| + # for forwarded email for multi user, avoid rezidual information from the last. + @my_info.clear + assign_account_vm = user_info.delete('account_vm') + @my_info.update user_info + + applying_account(assign_account_vm) + sleep 5 + end + else + parse_mail_content + applying_account(false) + end rescue StandardError => e puts e.message puts e.backtrace @@ -75,6 +102,24 @@ class ApplyAccount send_mail(e.message) end + def forward_users + forward_email_content = ParseApplyAccountEmail.new(@mail_content) + users_info = forward_email_content.extract_users + + users_info.clone.each_index do |i| + users_info[i]['my_ssh_pubkey'] = [] + users_info[i]['lab'] = ENV['lab'] + end + + users_info + end + + def applying_account(assign_account_vm) + account_info = apply_my_account + store_account_info + send_mail('', account_info, assign_account_vm) + end + def parse_mail_content parse_apply_account_email = ParseApplyAccountEmail.new(@mail_content) @@ -93,7 +138,9 @@ class ApplyAccount apply_info['my_token'] = my_account_es['my_uuid'] if my_account_es['my_token'].nil? apply_info.update my_account_es apply_info.update @my_info - apply_info['my_ssh_pubkey'] = (apply_info['my_ssh_pubkey'] + my_ssh_pubkey_new).uniq + if my_ssh_pubkey_new + apply_info['my_ssh_pubkey'] = (apply_info['my_ssh_pubkey'] + my_ssh_pubkey_new).uniq + end @my_info.update apply_info apply_info['is_update_account'] = true apply_info @@ -102,6 +149,8 @@ class ApplyAccount def apply_my_account my_account_es = read_my_account_es apply_info = {} + apply_info['enable_login'] = true + if my_account_es build_apply_info(apply_info, my_account_es) else @@ -109,7 +158,6 @@ class ApplyAccount @my_info['my_token'] = my_token apply_info.update @my_info end - apply_info['lab'] = ENV['lab'] apply_new_account(apply_info, my_account_es) end @@ -118,6 +166,7 @@ class ApplyAccount acct_info = apply_account.apply_jumper_account @my_info['my_login_name'] = acct_info['my_login_name'] unless my_account_es + acct_info end def store_account_info @@ -125,9 +174,9 @@ class ApplyAccount es.put_source_by_id(@my_info['my_email'], @my_info) end - def send_mail(error_message) + def send_mail(error_message, account_info, assign_account_vm) email_message = if error_message.empty? - build_apply_account_email(@my_info) + build_apply_account_email(@my_info, account_info, assign_account_vm) else build_apply_account_fail_email(@my_info, error_message) end -- 2.23.0

2 4

[PATCH compass-ci] doc: optimize os_mount.md
by Yu Chuan 21 Feb '21

21 Feb '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- doc/job/os_mount.md | 181 +++++++++++++++++++++++++++++--------------- 1 file changed, 120 insertions(+), 61 deletions(-) diff --git a/doc/job/os_mount.md b/doc/job/os_mount.md index 226251a8bace..b06b27237c75 100644 --- a/doc/job/os_mount.md +++ b/doc/job/os_mount.md @@ -1,9 +1,9 @@ # Summary ========= -`os_mount` defines the mount method of testbox's root partition. +`os_mount` defines the rootfs type. -It has the following optional values: +The below types are supported: - nfs - cifs - initramfs @@ -11,84 +11,143 @@ It has the following optional values: - local Usage example: - - submit iperf.yaml testbox=vm-2p8g os_mount=nfs - - submit iperf.yaml testbox=vm-2p8g os_mount=cifs - - submit iperf.yaml testbox=vm-2p8g os_mount=initramfs - - submit iperf.yaml testbox=dc-8g os_mount=container - - submit iperf.yaml testbox=vm-2p8g os_mount=local + - submit -m -c borrow-1h.yaml testbox=vm-2p8g os_mount=nfs + - submit -m -c borrow-1h.yaml testbox=vm-2p8g os_mount=cifs + - submit -m -c borrow-1h.yaml testbox=vm-2p16g os_mount=initramfs + - submit -m -c borrow-1h.yaml testbox=dc-8g os_mount=container + - submit -m -c borrow-1h.yaml testbox=taishan200-2280-2s48p-256g os_mount=local -# Optional Values -================= +# os_mount=local +================ -## Work flow when os_mount=local +## description -1. user submit job with os_mount: local - - optional {kernel_append_root}: - - save_root_partition: give an iconic string, then the root partition data used by this job will be retained. - and you can use the root partition data of this job in the futher job by {use_root_partition}. - - use_root_partition : give an iconic string, then the root partition data generated by the - pervious job which has the {save_root_partition} field will be used. - - root_partition_size: specify the size of root partition, *default is 10G*. +If you specifies os_mount=local, the rootfs type of your job will be a logical volume in the testbox. - - tips: - - value of {save_root_partition} and {use_root_partition} need to be iconic enough, uuid is recommended. - - value of {save_root_partition} and {use_root_partition} will be the suffix of logical volume. +> logical volume format: `/dev/mapper/os-${os}_${os_arch}_${os_version}` + +> optional vars in ${kernel_append_root}: + - ${save_root_partition}, ${use_root_partition} + - you can save the rootfs data of any job, and use it in subsequent job. + - ${save_root_partition} and ${use_root_partition} must be the same, and obviously, they need be unique. + - you must 'save_root_partition' firstly, then you can 'use_root_partition'. otherwise, your job will fail. + - demo kernel_append_root field of job.yaml: + ``` + # demo 1. save the rootfs data of this job + kernel_append_root: save_root_partition=zhangsan_local_for_iperf_20210219 + + # demo 2. use the rootfs data of previous job + kernel_append_root: use_root_partition=zhangsan_local_for_iperf_20210218 + + # demo 3. use the rootfs data of previous job, and save the rootfs data of this job + kernel_append_root: use_root_partition=zhangsan_local_for_iperf_20210218 save_root_partition=zhangsan_local_for_iperf_20210219 + ``` + +## work flow + +1. user submit job + - job fields: + - os: openeuler + - os_arch: aarch64 + - os_version: 20.03 + - os_mount: local + - kernel_append_root: use_root_partition=zhangsan_local_for_iperf_20210218 save_root_partition=zhangsan_local_for_iperf_20210219 2. scheduler return the ipxe_str to testbox ``` dhcp - initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/{timestamp}/initrd.lkp - initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/{timestamp}/boot/vmlinuz - imgargs vmlinux root=/dev/mapper/os-openeuler_aarch64_20.03 rootfs_src={nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/{timestamp} initrd=initrd.lkp {kernel_append_root} + initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/${timestamp}/initrd.lkp + kernel http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/${timestamp}/boot/vmlinuz root=/dev/mapper/os-openeuler_aarch64_20.03 rootfs_src=${nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/${timestamp} initrd=initrd.lkp ${kernel_append_root} boot ``` 3. dracut step of boot ``` - if have {use_root_partition}; then - base_lv_name=openeuler_aarch64_20.03_{value_of_use_root_partition} - lvdisplay /dev/mappaer/os-{base_lv_name} > /dev/null || exit 1 - - if have {save_root_partition}; then - boot_lv_name=openeuler_aarch64_20.03_{value_of_save_root_partition} - - if {save_root_partition} == {use_root_partition}; then - boot from /dev/mappaer/os-{boot_lv_name} - fi - - lvremove -f /dev/mappaer/os-{boot_lv_name} - lvcreate /dev/mappaer/os-{boot_lv_name} from /dev/mappaer/os-{base_lv_name} || exit 1 - boot from /dev/mappaer/os-{boot_lv_name} - else - boot_lv_name=openeuler_aarch64_20.03_{value_of_use_root_partition} - boot from /dev/mappaer/os-{boot_lv_name} - fi - - else: - base_lv_name=openeuler_aarch64_20.03_{timestamp} - lvdisplay /dev/mappaer/os-{base_lv_name} > /dev/null || { + copy_nfsroot_to_src_lv() + { + src_lv=$1 + + lvdisplay ${src_lv} > /dev/null || { # create logical volume - lvcreate -L {root_partition_size} -n {base_lv_name} os || exit 1 + lvcreate -L 10G -n $(basename ${src_lv}) os || exit 1 - # rsync nfsroot to lvm:/dev/mapper/os-{base_lv_name} - mount -t nfs {nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/{timestamp} /mnt - mkdir /mnt1 && mount /dev/mapper/os-{base_lv_name} /mnt1 + # rsync nfsroot to ${src_lv} + mount -t nfs ${nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/${timestamp} /mnt + mkdir /mnt1 && mount ${src_lv} /mnt1 cp -a /mnt/. /mnt1/ umount /mnt /mnt1 - # change premission of lvm:/dev/mapper/os-{base_lv_name} to readonly - lvchange -p r /dev/mapper/os-{base_lv_name} + # change premission of ${src_lv} to readonly + lvchange -p r ${src_lv} } - - boot_lv_name=openeuler_aarch64_20.03 - if have {save_root_partition}; then - boot_lv_name=openeuler_aarch64_20.03_{value_of_save_root_partition} - fi - - lvremove -f /dev/mappaer/os-{boot_lv_name} - lvcreate /dev/mappaer/os-{boot_lv_name} from /dev/mappaer/os-{base_lv_name} || exit 1 - boot from /dev/mappaer/os-{boot_lv_name} - fi + } + + check_src_lv() + { + lvdisplay $1 > /dev/null || exit 1 + } + + clone_boot_lv_from_src_lv() + { + src_lv=$1 + boot_lv=$2 + + lvremove -f ${boot_lv} + lvcreate ${boot_lv} from ${src_lv} || exit 1 + } + + no_save_no_use() + { + src_lv="/dev/mapper/os-openeuler_aarch64_20.03_${timestamp}" + boot_lv="/dev/mapper/os-openeuler_aarch64_20.03" + + copy_nfsroot_to_src_lv ${src_lv} + clone_boot_lv_from_src_lv ${src_lv} ${boot_lv} + boot_from_lv ${boot_lv} + } + + only_save() + { + src_lv="/dev/mapper/os-openeuler_aarch64_20.03_${timestamp}" + boot_lv="/dev/mapper/os-openeuler_aarch64_20.03_${save_root_partition}" + + copy_nfsroot_to_src_lv ${src_lv} + clone_boot_lv_from_src_lv ${src_lv} ${boot_lv} + boot_from_lv ${boot_lv} + } + + only_use() + { + src_lv="/dev/mapper/os-openeuler_aarch64_20.03_${use_root_partition}" + boot_lv="/dev/mapper/os-openeuler_aarch64_20.03" + + check_src_lv ${src_lv} + clone_boot_lv_from_src_lv ${src_lv} ${boot_lv} + boot_from_lv ${boot_lv} + } + + save_and_use() + { + src_lv="/dev/mapper/os-openeuler_aarch64_20.03_${use_root_partition}" + boot_lv="/dev/mapper/os-openeuler_aarch64_20.03_${save_root_partition}" + + check_src_lv ${src_lv} + clone_boot_lv_from_src_lv ${src_lv} ${boot_lv} + boot_from_lv ${boot_lv} + } + + main() + { + [ -z ${use_root_partition} ] && [ -z ${save_root_partition} ] && no_save_no_use + + [ -z ${use_root_partition} ] && [ -n ${save_root_partition} ] && only_save + + [ -n ${use_root_partition} ] && [ -z ${save_root_partition} ] && only_use + + [ -n ${use_root_partition} ] && [ -n ${save_root_partition} ] && save_and_use + } + + main() ``` 4. execute the job -- 2.23.0

2 10

[PATCH v2 compass-ci] sbin/build-my-info: fix no key: my_ssh_pubkey
by Luan Shengde 20 Feb '21

20 Feb '21

[why] it will read my_ssh_pubkey and check it when submit jobs. $ submit -m -c job.yaml -i ~/lkp-tests/jobs/ssh-on-fail.yaml submit_id=f87d2951-c226-4e51-a3e4-c71818a4c974 submit job.yaml failed, got job id=0, error: Missing hash key: "my_ssh_pubkey" [how] add initialize my_ssh_pubkey when building my_info. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- sbin/build-my-info.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/sbin/build-my-info.rb b/sbin/build-my-info.rb index c7a7798..ecfcb42 100755 --- a/sbin/build-my-info.rb +++ b/sbin/build-my-info.rb @@ -12,6 +12,7 @@ def build_my_info(option) info_es = build_my_info.search_my_info info_es.update option info_es['my_token'] = %x(uuidgen).chomp if info_es['my_token'].nil? + info_es['my_ssh_pubkey'] = [] build_my_info.config_my_info(info_es) end -- 2.23.0

1 0

[PATCH v2 compass-ci] answerback-email: add store lab to ES
by Luan Shengde 20 Feb '21

20 Feb '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index 8dceee1..f573f0f 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -40,7 +40,8 @@ my_info = { 'my_commit_url' => nil, 'my_token' => nil, 'my_login_name' => nil, - 'my_ssh_pubkey' => [] + 'my_ssh_pubkey' => [], + 'lab' => LAB } # stdin_info is used to store infos added with option: @@ -179,7 +180,6 @@ def apply_account(my_info, conf_info) apply_info = {} apply_info.update my_info apply_info.update conf_info - apply_info['lab'] = LAB assign_account_url = "#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account" account_info_str = RestClient.post assign_account_url, apply_info.to_json -- 2.23.0

1 0

[PATCH compass-ci] auto-submit: use the lib script to config the account
by Cao Xueliang 20 Feb '21

20 Feb '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- sbin/auto_submit | 21 ++------------------- 1 file changed, 2 insertions(+), 19 deletions(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index 6d29543..b408158 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -7,8 +7,7 @@ require 'bunny' require 'json' require 'yaml' require 'set' -require_relative '../lib/build_my_info_client' -require "#{ENV['CCI_SRC']}/container/defconfig" +require_relative '../lib/config_account' # receive message and auto submit job class AutoSubmit @@ -130,23 +129,7 @@ class AutoSubmit end end -def my_email - names = Set.new %w[auto-submit] - - defaults = relevant_defaults(names) - defaults['auto-submit']['my_email'] -end - -def config_yaml - build_my_info = BuildMyInfo.new(my_email) - - my_info = build_my_info.search_my_info - build_my_info.config_default_yaml(my_info) - build_my_info.config_lab_yaml(my_info) -end - -# fix verification problems when submit jobs by auto-submit service -config_yaml +config_yaml('auto-submit') auto_submitter = AutoSubmit.new auto_submitter.listen -- 2.23.0

1 0

[PATCH v2 compass-ci 4/4] ner/mail-robot: optimize start
by Luan Shengde 20 Feb '21

20 Feb '21

attach the directory instead of the single file. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/start | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/mail-robot/start b/container/mail-robot/start index 3827e2c..bd2a62d 100755 --- a/container/mail-robot/start +++ b/container/mail-robot/start @@ -31,7 +31,7 @@ cmd = %w[ ] + env + %W[ -v /c/upstream-repos:/c/upstream-repos:ro -v #{defaults['MAILDIR']}:#{defaults['MAILDIR']}:rw - -v /etc/compass-ci/conf/sec_admins.yaml:/etc/compass-ci/conf/sec_admins.yaml:ro + -v /etc/compass-ci:/etc/compass-ci:ro -v #{ENV['CCI_SRC']}:/c/compass-ci:rw -w /c/compass-ci/container/mail-robot mail-robot -- 2.23.0

1 0

[PATCH v2 compass-ci 3/4] container/mail-robot: add login command
by Luan Shengde 20 Feb '21

20 Feb '21

add login command for account VM server. case user info has key account_vm with value: yes/true, the email will send the login command. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../mail-robot/lib/assign-account-email.rb | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/container/mail-robot/lib/assign-account-email.rb b/container/mail-robot/lib/assign-account-email.rb index b71b3fe..8e3aa6e 100755 --- a/container/mail-robot/lib/assign-account-email.rb +++ b/container/mail-robot/lib/assign-account-email.rb @@ -3,13 +3,26 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true -def build_apply_account_email(my_info) +def build_apply_account_email(my_info, account_info, assign_account_vm) email_greeting = if my_info['bisect'] 'We have automatically created the following information for you.' else 'Thank you for joining us.' end + logging_msg = <<~LOGIN_MSG + + You can use the following command to login the jumper server: + + Login command: + ssh -p #{account_info['jumper_port']} #{account_info['my_login_name']}@#{account_info['jumper_host']} + + Account password: + #{account_info['my_password']} + LOGIN_MSG + + logging_account_vm = assign_account_vm ? logging_msg : '' + email_msg = <<~EMAIL_MESSAGE To: #{my_info['my_email']} Subject: [compass-ci] Account Ready @@ -17,7 +30,7 @@ def build_apply_account_email(my_info) Dear #{my_info['my_name']}, #{email_greeting} - + #{logging_account_vm} You can use the following info to submit jobs: notice: -- 2.23.0

1 0

[PATCH v2 compass-ci 2/4] container/mail-robot: add parsing forwarded email
by Luan Shengde 20 Feb '21

20 Feb '21

for forwarded 'apply account' email, my_email/my_name is store in a fixed format in the mail content. extract a list of users info from the mail content Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../mail-robot/lib/parse-apply-account-email.rb | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/container/mail-robot/lib/parse-apply-account-email.rb b/container/mail-robot/lib/parse-apply-account-email.rb index 92db1da..3de2fb5 100755 --- a/container/mail-robot/lib/parse-apply-account-email.rb +++ b/container/mail-robot/lib/parse-apply-account-email.rb @@ -57,9 +57,18 @@ class ParseApplyAccountEmail end def extract_users - users = extract_mail_content_body.split(/\r|\n/) + users_info = [] + + users = extract_mail_content_body.split(/---+/) users.delete('') - users + users.each do |user| + user_info = YAML.safe_load(user) + next unless user_info.include?('my_email') + + users_info << user_info + end + + return users_info end def extract_commit_url -- 2.23.0

1 0

[PATCH compass-ci] delimter: Use delimiter account submit the bisect job
by Cao Xueliang 20 Feb '21

20 Feb '21

Add the config_account.rb to create the account info in container, it will use the specified account when submit the job in container. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/delimiter/start | 1 + lib/config_account.rb | 22 ++++++++++++++++++++++ src/delimiter.rb | 2 ++ 3 files changed, 25 insertions(+) create mode 100644 lib/config_account.rb diff --git a/container/delimiter/start b/container/delimiter/start index 669923b..3cff23a 100755 --- a/container/delimiter/start +++ b/container/delimiter/start @@ -31,6 +31,7 @@ cmd = %w[ -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} -v #{DEFAULT_CONFIG_DIR}:#{DEFAULT_CONFIG_DIR}:ro -v /etc/localtime:/etc/localtime:ro + -v /etc/compass-ci/register:/etc/compass-ci/register:ro -w #{DEFAULT_CCI}/src delimiter ] diff --git a/lib/config_account.rb b/lib/config_account.rb new file mode 100644 index 0000000..dc0067c --- /dev/null +++ b/lib/config_account.rb @@ -0,0 +1,22 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative "./build_my_info_client" +require_relative "../container/defconfig" + +def my_email(account) + names = %W(#{account}) + + defaults = relevant_defaults(names) + defaults[account]['my_email'] +end + +def config_yaml(account) + build_my_info = BuildMyInfo.new(my_email(account)) + + my_info = build_my_info.search_my_info + build_my_info.config_default_yaml(my_info) + build_my_info.config_lab_yaml(my_info) +end diff --git a/src/delimiter.rb b/src/delimiter.rb index bd40775..1ea5de5 100644 --- a/src/delimiter.rb +++ b/src/delimiter.rb @@ -3,8 +3,10 @@ # frozen_string_literal: true require_relative './delimiter/delimiter' +require_relative '../lib/config_account' begin + config_yaml('delimiter') delimiter = Delimiter.new delimiter.start_delimit rescue StandardError => e -- 2.23.0

1 0

[PATCH compass-ci] lib/mail_bisect_result: fix undefined method `/'
by Cao Xueliang 20 Feb '21

20 Feb '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/mail_bisect_result.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index d43785b..104f098 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -47,7 +47,7 @@ class MailBisectResult Hi #{(a)git_commit.author_name}, We found some error/warning(s) and the first bad commit in the below project: - git url: #{@upstream_url/@commit_id} + git url: #{@upstream_url}/#{@commit_id} git commit: #{@commit_id} ("#{(a)git_commit.subject}") All error/warning(s) (new ones prefixed by >>): -- 2.23.0

1 0

[PATCH v4 compass-ci 2/2] sparrow: separate service config
by Cao Xueliang 20 Feb '21

20 Feb '21

Extract service config from defaults/yaml to the service/yaml. Read service/yaml in defconfig.rb and defconfig.sh. Add scheduler/yaml for scheduler set the content to the job. Adjust the order of service config. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/defconfig.rb | 1 + container/defconfig.sh | 1 + sparrow/3-code/dev-env | 80 +++++++++++++++++++++++++++++------------- 3 files changed, 57 insertions(+), 25 deletions(-) diff --git a/container/defconfig.rb b/container/defconfig.rb index 602b498..b121de8 100755 --- a/container/defconfig.rb +++ b/container/defconfig.rb @@ -8,6 +8,7 @@ require 'yaml' def cci_defaults hash = {} Dir.glob(['/etc/compass-ci/defaults/*.yaml', + '/etc/compass-ci/service/*.yaml', '/etc/compass-ci/accounts/*.yaml', '/etc/compass-ci/register/*.yaml', "#{ENV['HOME']}/.config/compass-ci/defaults/*.yaml"]).each do |file| diff --git a/container/defconfig.sh b/container/defconfig.sh index 7c575ef..c33a790 100755 --- a/container/defconfig.sh +++ b/container/defconfig.sh @@ -10,6 +10,7 @@ load_cci_defaults() yaml_file=( /etc/compass-ci/defaults/*.yaml + /etc/compass-ci/service/*.yaml /etc/compass-ci/accounts/*.yaml "$HOME"/.config/compass-ci/defaults/*.yaml ) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 8812839..cc94e01 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -11,52 +11,82 @@ server_ip=$(awk '/^server_ip:\s/ {print $2; exit}' /etc/compass-ci/setup.yaml) server_name=$(hostname | cut -f1 -d.) : ${sched_host:=$server_ip} : ${sched_port:=3000} -: ${os_http_host:=$server_ip} -: ${initrd_http_host:=$server_ip} -: ${initrd_http_port:=8800} +: ${ES_HOST:=$server_ip} +: ${ES_PORT:=9200} +: ${OS_HTTP_HOST:=$server_ip} +: ${SEND_MAIL_HOST:=$server_ip} +: ${SEND_MAIL_PORT:=49000} +: ${SRV_HTTP_HOST:=$server_ip} +: ${SRV_HTTP_PORT:=11300} +: ${LOGGING_ES_HOST:=$server_ip} +: ${LOGGING_ES_PORT:=9202} +: ${INITRD_HTTP_HOST:=$server_ip} +: ${INITRD_HTTP_PORT:=8800} +: ${ASSIST_RESULT_HOST:=$server_ip} +: ${ASSIST_RESULT_PORT:=8102} +: ${RESULT_WEBDAV_HOST:=$server_ip} +: ${RESULT_WEBDAV_PORT:=3080} : ${MASTER_FLUENTD_HOST:=$server_ip} : ${MASTER_FLUENTD_PORT:=24224} : ${DOCKER_REGISTRY_HOST:=$server_ip} : ${DOCKER_REGISTRY_PORT:=5001} -: ${LOGGING_ES_HOST:=$server_ip} -: ${LOGGING_ES_PORT:=9202} -: ${SEND_MAIL_PORT:=49000} : ${LOCAL_SEND_MAIL_PORT:=11311} : ${LOCAL_ROBOT_EMAIL_ADDRESS:=Crystal TEAM} -: ${ASSIST_RESULT_PORT:=8102} -: ${RESULT_WEBDAV_PORT:=3080} -: ${ES_HOST:=$server_ip} -: ${ES_PORT:=9200} -: ${SRV_HTTP_PORT:=11300} mkdir -p /etc/compass-ci/defaults cat > /etc/compass-ci/defaults/$server_name.yaml <<EOF +lab: nolab +EOF + +mkdir -p /etc/compass-ci/service +cat > /etc/compass-ci/service/service-env.yaml <<EOF +ES_HOST: $ES_HOST +ES_PORT: $ES_PORT SCHED_HOST: $sched_host SCHED_PORT: $sched_port LKP_SERVER: $server_ip GIT_SERVER: $GIT_SERVER -TASKQUEUE_HOST: $sched_host -OS_HTTP_HOST: $os_http_host -INITRD_HTTP_HOST: $initrd_http_host -INITRD_HTTP_PORT: $initrd_http_port +OS_HTTP_HOST: $OS_HTTP_HOST GITCACHE_HOST: $gitcache_host GITCACHE_PORT: $gitcache_port +SRV_HTTP_HOST: $SRV_HTTP_HOST +SRV_HTTP_PORT: $SRV_HTTP_PORT +TASKQUEUE_HOST: $sched_host +SEND_MAIL_HOST: $SEND_MAIL_HOST +SEND_MAIL_PORT: $SEND_MAIL_PORT +LOGGING_ES_HOST: $LOGGING_ES_HOST +LOGGING_ES_PORT: $LOGGING_ES_PORT +INITRD_HTTP_HOST: $INITRD_HTTP_HOST +INITRD_HTTP_PORT: $INITRD_HTTP_PORT +ASSIST_RESULT_HOST: $ASSIST_RESULT_HOST +ASSIST_RESULT_PORT: $ASSIST_RESULT_PORT +RESULT_WEBDAV_HOST: $RESULT_WEBDAV_HOST +RESULT_WEBDAV_PORT: $RESULT_WEBDAV_PORT MASTER_FLUENTD_HOST: $MASTER_FLUENTD_HOST MASTER_FLUENTD_PORT: $MASTER_FLUENTD_PORT DOCKER_REGISTRY_HOST: $DOCKER_REGISTRY_HOST DOCKER_REGISTRY_PORT: $DOCKER_REGISTRY_PORT -LOGGING_ES_HOST: $LOGGING_ES_HOST -LOGGING_ES_PORT: $LOGGING_ES_PORT -SEND_MAIL_PORT: $SEND_MAIL_PORT LOCAL_SEND_MAIL_PORT: $LOCAL_SEND_MAIL_PORT -LOCAL_ROBOT_EMAIL_ADDRESS: $LOCAL_ROBOT_EMAIL_ADDRESS MAILDIR: /srv/cci/Maildir/.compass-ci -ASSIST_RESULT_PORT: $ASSIST_RESULT_PORT -RESULT_WEBDAV_PORT: $RESULT_WEBDAV_PORT -lab: nolab -ES_HOST: $ES_HOST -ES_PORT: $ES_PORT -SRV_HTTP_PORT: $SRV_HTTP_PORT +LOCAL_ROBOT_EMAIL_ADDRESS: $LOCAL_ROBOT_EMAIL_ADDRESS +EOF + +mkdir -p /etc/compass-ci/scheduler +cat > /etc/compass-ci/scheduler/local-testbox-env.yaml <<-EOF +LKP_SERVER: +GIT_SERVER: +GITCACHE_HOST: +GITCACHE_PORT: +SRV_HTTP_HOST: +SRV_HTTP_PORT: +SEND_MAIL_HOST: +SEND_MAIL_PORT: +INITRD_HTTP_HOST: +INITRD_HTTP_PORT: +ASSIST_RESULT_HOST: +ASSIST_RESULT_PORT: +RESULT_WEBDAV_HOST: +RESULT_WEBDAV_PORT: EOF cat > /etc/profile.d/compass.sh <<'EOF' -- 2.23.0

1 0

[PATCH v4 compass-ci 1/2] scheduler: add testbox env in scheduler
by Cao Xueliang 20 Feb '21

20 Feb '21

We add the testbox env use the submit script in lkp-tests, but the submit don't need to do that, so, we add the testbox env in scheduler. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 2 ++ src/scheduler/testbox_env.cr | 33 +++++++++++++++++++++++++++++++++ 2 files changed, 35 insertions(+) create mode 100644 src/scheduler/testbox_env.cr diff --git a/src/lib/job.cr b/src/lib/job.cr index 1afc22f..c3eeaf4 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -9,6 +9,7 @@ require "scheduler/constants.cr" require "scheduler/jobfile_operate.cr" require "scheduler/kernel_params.cr" require "scheduler/pp_params.cr" +require "scheduler/testbox_env.cr" require "../scheduler/elasticsearch_client" require "./json_logger" @@ -52,6 +53,7 @@ class Job def initialize(job_content : JSON::Any, id) @hash = job_content.as_h + @hash.merge!(testbox_env) @es = Elasticsearch::Client.new @account_info = Hash(String, JSON::Any).new @log = JSONLogger.new diff --git a/src/scheduler/testbox_env.cr b/src/scheduler/testbox_env.cr new file mode 100644 index 0000000..bf8cd8a --- /dev/null +++ b/src/scheduler/testbox_env.cr @@ -0,0 +1,33 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +class Job + def get_service_env + hash = Hash(String, JSON::Any).new + yaml_any = File.open("/etc/compass-ci/service/service-env.yaml") do |content| + YAML.parse(content).as_h? + end + return hash unless yaml_any + + return Hash(String, JSON::Any).from_json(yaml_any.to_json) + end + + def testbox_env(flag = "local") + service_env = get_service_env + hash = Hash(String, JSON::Any).new + + yaml_any = File.open("/etc/compass-ci/scheduler/#{flag}-testbox-env.yaml") do |content| + YAML.parse(content).as_h? + end + return hash unless yaml_any + + hash.merge!(Hash(String, JSON::Any).from_json(yaml_any.to_json)) + hash.each do |key, value| + if value == nil + hash[key] = service_env[key] + end + end + + hash + end +end -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: modify the function get_hosts_file
by Li Ping 20 Feb '21

20 Feb '21

[why]: if the hosts_file 2288hv5-2s44p-38g--a6 already in lkp-tests/hosts/ submit borrow-10d.yaml testbox=2288hv5-2s44p-38g--a6 Traceback (most recent call last): 7: from /home/liping/lkp-tests/sbin/submit:136:in `<main>' 6: from /home/liping/lkp-tests/sbin/submit:136:in `each' 5: from /home/liping/lkp-tests/sbin/submit:150:in `block in <main>' 4: from /home/liping/lkp-tests/lib/job.rb:743:in `each_jobs' 3: from /home/liping/lkp-tests/lib/job.rb:447:in `load_defaults' 2: from /home/liping/lkp-tests/lib/job.rb:432:in `load_self_config' 1: from /home/liping/lkp-tests/lib/job.rb:351:in `load_hosts_config' /home/liping/lkp-tests/lib/job.rb:378:in `get_hosts_file': hosts file not exist: 2288hv5-2s44p-38g, maybe need check testbox field (ArgumentError) [reason]: get_hosts_file use tbox_group to search for hosts_file --- lib/job.rb | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index eacb7ec95..8f8c91b28 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -366,12 +366,17 @@ class Job end def get_hosts_file - testbox_file = "#{ENV['CCI_REPOS']}/lab-#{@job['lab']}/hosts/#{@job['testbox']}" - return testbox_file if File.file?(testbox_file) - + lab_host = "#{ENV['CCI_REPOS']}/lab-#{@job['lab']}/hosts" + lkp_host = "#{LKP_SRC}/hosts" tbox_group = @job['tbox_group'].split(/\.|--/)[0] - tbox_group_file = "#{LKP_SRC}/hosts/#{tbox_group}" - return tbox_group_file if File.file?(tbox_group_file) + check_files = ["#{lab_host}/#{@job['testbox']}", + "#{lkp_host}/#{@job['testbox']}", + "#{lab_host}/#{tbox_group}", + "#{lkp_host}/#{tbox_group}"] + + check_files.each do |file| + return file if File.file?(file) + end raise ArgumentError, "hosts file not exist: #{tbox_group}, maybe need check testbox field" end -- 2.23.0

2 2

[PATCH lkp-tests] tests/rpmbuild-pkg: various cleanup
by Wu Fengguang 20 Feb '21

20 Feb '21

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- tests/rpmbuild-pkg | 33 +++++++++++++++++++++------------ 1 file changed, 21 insertions(+), 12 deletions(-) diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index 781d04a0d..9f455f0d0 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -5,8 +5,8 @@ . $LKP_SRC/lib/upload.sh [ -n "$rpm_repo" ] || die "rpm_repo is empty" -benchmark=${rpm_repo##*/} -sync_dest="/initrd/rpmbuild-pkg/${os_mount}/${os}/${os_arch}/${os_version}/${benchmark}" +package_name=${rpm_repo##*/} +sync_dest="/initrd/rpmbuild-pkg/${os_mount}/${os}/${os_arch}/${os_version}/${package_name}" init_workspace() { @@ -16,15 +16,15 @@ init_workspace() get_pkgfile() { - curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ + curl -sS -H 'Content-Type: Application/json' -XPOST "$GIT_SERVER"':8100/git_command' \ -d '{"git_repo": "'${rpm_repo}'", "git_command": ["git-show", "HEAD:'$1'"]}' -o "${2}" } download_rpm_repo() { - filelist=$(curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ + local filelist=$(curl -sS -H 'Content-Type: Application/json' -XPOST "$GIT_SERVER"':8100/git_command' \ -d '{"git_repo": "'${rpm_repo}'", "git_command": ["git-ls-files", "."]}') - for pkgfile in ${filelist[*]} + for pkgfile in "${filelist[@]}" do local dir="SOURCES" echo $pkgfile | egrep "\.spec$" && dir="SPECS" @@ -34,20 +34,29 @@ download_rpm_repo() build_rpm() { - spec_dir=${HOME}/rpmbuild/SPECS/$benchmark.spec - sed -i 's/^$Source[^ ]*:[ \t]*$https/\1http/g' `grep http -rl $spec_dir` + local spec_file=${HOME}/rpmbuild/SPECS/$package_name.spec + + # HTTP is proxy cache friendly + sed -i 's/^$Source[^ ]*:[ \t]*$https/\1http/g' `grep http -rl $spec_file` + # Install build depends - yum-builddep -y $spec_dir + yum-builddep -y $spec_file || exit # Download tar.gz to default path ${HOME}/rpmbuild/SOURCE - spectool -g -R $spec_dir + spectool -g -R $spec_file || exit # Building rpm or srpm packages - rpmbuild -ba $spec_dir + rpmbuild -ba $spec_file || exit +} + +show_rpm_files() +{ + find ${HOME}/rpmbuild/RPMS -type f -name "*.rpm" + find ${HOME}/rpmbuild/SRPMS -type f -name "*.rpm" } upload_rpm_pkg() { - # rpm package will be generated in "${HOME}/rpmbuild/SRPMS" and "${HOME}/rpmbuild/RPMS" - for file in $(find ${HOME}/rpmbuild/ -type f -name "*.rpm") + local file + for file in $(show_rpm_files) do upload_one_curl ${file} ${sync_dest} done -- 2.23.0

1 0

[PATCH v2 compass-ci] rsync-server: add support upload function for ks file
by Yu Chuan 20 Feb '21

20 Feb '21

[Why] We need to achieve automatic installation through 'pxe + kickstart + dnsmasq + httpd'. So we need to rsync the ks file by rsync-server. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- container/rsync-server/rsyncd.conf | 6 ++++++ container/rsync-server/start | 1 + 2 files changed, 7 insertions(+) diff --git a/container/rsync-server/rsyncd.conf b/container/rsync-server/rsyncd.conf index cdba89cf20c8..5a9bd2ddadec 100644 --- a/container/rsync-server/rsyncd.conf +++ b/container/rsync-server/rsyncd.conf @@ -10,3 +10,9 @@ path = /srv/result/upload/ read only = no max connections = 200 refuse options = delete + +[ks] +path = /srv/os/install/ks/ +read only = no +max connections = 200 +refuse options = delete diff --git a/container/rsync-server/start b/container/rsync-server/start index 956013268f89..334f5c2c4a69 100755 --- a/container/rsync-server/start +++ b/container/rsync-server/start @@ -11,6 +11,7 @@ cmd=( --restart=always --name=rsync_server -v /srv/result/upload:/srv/result/upload + -v /srv/os/install/ks:/srv/os/install/ks -v /etc/localtime:/etc/localtime:ro -p 11387:11387 rsync_server:latest -- 2.23.0

1 0

[PATCH lkp-tests] bootstrap.sh: add a test ping condition to network_ok
by Yu Chuan 20 Feb '21

20 Feb '21

[Why] In physical machine, there will be a delay between the start of network card and the availability of the network. So we add a test ping condition to network_ok. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- lib/bootstrap.sh | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index 06aa09bf65b2..38965ff60755 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -85,14 +85,19 @@ net_devices_link() done } +test_ping() +{ + ping -c 1 -W 1 $LKP_SERVER > /dev/null || return 1 +} + network_ok() { local i for i in /sys/class/net/*/ do [ "${i#*/lo/}" != "$i" ] && continue - [ "$(cat $i/operstate)" = 'up' ] && return 0 - [ "$(cat $i/carrier 2>/dev/null)" = '1' ] && return 0 + [ "$(cat $i/operstate)" = 'up' ] && test_ping && return 0 + [ "$(cat $i/carrier 2>/dev/null)" = '1' ] && test_ping && return 0 done is_clearlinux && { -- 2.23.0

2 7

[PATCH compass-ci] lib/git_mirror.rb: change the way to load DEFAULTS file
by Li Yuanchao 19 Feb '21

19 Feb '21

Before: DEFAULTS files are individual. After: Load DEFAULTS file and merge it with the parent directory DEFAULTS files. So the outer DEFAULTS files would keep common things for the inner ones. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 39b1619..346352b 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -130,6 +130,7 @@ class MirrorMain defaults_key = repodir == REPO_DIR ? 'default' : repodir.delete_prefix("#{REPO_DIR}/") @defaults[defaults_key] = YAML.safe_load(File.open(defaults_file)) + @defaults[defaults_key] = merge_defaults(defaults_key, @defaults[defaults_key]) end def traverse_repodir(repodir) @@ -224,7 +225,7 @@ class MirrorMain git_repo = "#{project}/#{fork_name}" @git_info[git_repo] = YAML.safe_load(File.open(repodir)) @git_info[git_repo]['git_repo'] = git_repo - @git_info[git_repo] = merge_defaults(git_repo) + @git_info[git_repo] = merge_defaults(git_repo, @git_info[git_repo]) fork_stat_init(git_repo) @priority_queue.push git_repo, @priority @priority += 1 @@ -449,15 +450,17 @@ end # main thread class MirrorMain - def merge_defaults(git_repo) - defaults_key = File.dirname(git_repo) + def merge_defaults(object_key, object) + return object if object_key == 'default' + + defaults_key = File.dirname(object_key) while defaults_key != '.' - return @defaults[defaults_key].merge(@git_info[git_repo]) if @defaults[defaults_key] + return @defaults[defaults_key].merge(object) if @defaults[defaults_key] defaults_key = File.dirname(defaults_key) end - return @defaults['default'].merge(@git_info[git_repo]) if @defaults['default'] + return @defaults['default'].merge(object) if @defaults['default'] - return @git_info[git_repo] + return object end end -- 2.23.0

1 0

[PATCH v2 lkp-tests 2/2] lkp-bootstrap: delete unnecessary exit
by Xu Xijian 19 Feb '21

19 Feb '21

The exit cmd after reboot is unnecessary, delete it. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- rootfs/addon/etc/init.d/lkp-bootstrap | 1 - 1 file changed, 1 deletion(-) diff --git a/rootfs/addon/etc/init.d/lkp-bootstrap b/rootfs/addon/etc/init.d/lkp-bootstrap index f66ec4eb5..091b29890 100755 --- a/rootfs/addon/etc/init.d/lkp-bootstrap +++ b/rootfs/addon/etc/init.d/lkp-bootstrap @@ -24,7 +24,6 @@ if [ -s /lkp/run/lkp-bootstrap.pid ]; then # the previous lkp-bootstrap init.d script failed to restart the machine echo "LKP bootstrap: first run may failed, retry rebooting" reboot - exit fi read_kernel_cmdline_vars() -- 2.23.0

2 2

[PATCH v2 lkp-tests 1/2] bin/lkp-setup-rootfs: refactor the chmod_root_ssh()
by Xu Xijian 19 Feb '21

19 Feb '21

Add blank line after return cmd. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- bin/lkp-setup-rootfs | 1 + 1 file changed, 1 insertion(+) diff --git a/bin/lkp-setup-rootfs b/bin/lkp-setup-rootfs index cf48c8f6b..bc0f04b2a 100755 --- a/bin/lkp-setup-rootfs +++ b/bin/lkp-setup-rootfs @@ -245,6 +245,7 @@ chmod_root_ssh() { [ -d "/root/.ssh" ] || return [ -f "/root/.ssh/id_rsa" ] || return + chmod 700 /root/.ssh/ chmod 600 /root/.ssh/id_rsa } -- 2.23.0

2 2

[PATCH v2 compass-ci 2/2] src/scheduler/kernel_params.cr: add noserverino option to cifs mount
by Xu Xijian 19 Feb '21

19 Feb '21

[Log] [ 7.507105] CIFS VFS: Autodisabling the use of server inode numbers on \\172.168.131.113\os. This server doesn't seem to support them properly. Hardlinks will not be recognized on this mount. Consider mounting with the "noserverino" option to silence this message. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/scheduler/kernel_params.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/scheduler/kernel_params.cr b/src/scheduler/kernel_params.cr index 6fbe405..bb5983f 100644 --- a/src/scheduler/kernel_params.cr +++ b/src/scheduler/kernel_params.cr @@ -18,7 +18,7 @@ class Job fs2root = { "nfs" => "root=#{OS_HTTP_HOST}:#{os_real_path}", "cifs" => "root=cifs://#{OS_HTTP_HOST}#{os_real_path}" + - ",guest,ro,hard,vers=1.0,noacl,nouser_xattr", + ",guest,ro,hard,vers=1.0,noacl,nouser_xattr,noserverino", "initramfs" => "rdinit=/sbin/init prompt_ramdisk=0", "local" => "root=/dev/mapper/os-#{self.os}_#{self.os_arch}_#{self.os_version}", "container" => "", -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] doc/support-os-os_arch-os_version.md: add 20.03-SP1
by Xu Xijian 19 Feb '21

19 Feb '21

1. add 20.03-SP1 version to this doc, it's already supported 2. adjust some lines to be aligned Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- doc/support-os-os_arch-os_version.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/doc/support-os-os_arch-os_version.md b/doc/support-os-os_arch-os_version.md index 5d30486..f91e91b 100644 --- a/doc/support-os-os_arch-os_version.md +++ b/doc/support-os-os_arch-os_version.md @@ -1,22 +1,24 @@ ``` os os_arch os_version supported-os_mount openeuler aarch64 1.0 cifs/nfs, - 20.03 cifs/nfs, ,initramfs - 20.09 cifs/nfs, ,initramfs + 20.03 cifs/nfs, ,initramfs + 20.03-SP1 cifs/nfs, + 20.09 cifs/nfs, ,initramfs mainline-dailybuild cifs/nfs, 20.03-dailybuild cifs/nfs, 20.09-dailybuild cifs/nfs, 20.03-sp1-dailybuild cifs/nfs, x86 20.03 cifs/nfs, + 20.03-SP1 cifs/nfs, 20.09 cifs/nfs, centos aarch64 7 container, - 7.6.1810 cifs/nfs, ,initramfs - 7.8.2003 cifs/nfs, ,initramfs + 7.6.1810 cifs/nfs, ,initramfs + 7.8.2003 cifs/nfs, ,initramfs 8.1.1911 cifs/nfs, 8 container, x86 7.6.1810 cifs/nfs, 7.8.2003 cifs/nfs, -debian aarch64 sid cifs/nfs, ,initramfs +debian aarch64 sid cifs/nfs, ,initramfs x86 sid cifs/nfs, archlinux aarch64 2020-11-12 cifs/nfs, ``` -- 2.23.0

1 0

[PATCH lkp-tests] jobs/rpmbuild-pkg.yaml: change the rpm_repo field to level-1 parameter
by Li Ping 19 Feb '21

19 Feb '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- jobs/rpmbuild-pkg.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/jobs/rpmbuild-pkg.yaml b/jobs/rpmbuild-pkg.yaml index 19fd5ec28..53e145b3b 100644 --- a/jobs/rpmbuild-pkg.yaml +++ b/jobs/rpmbuild-pkg.yaml @@ -1,5 +1,5 @@ suite: rpmbuild-pkg category: functional +rpm_repo: rpmbuild-pkg: - rpm_repo: -- 2.23.0

1 0

[PATCH lkp-tests] tests/rpmbuild-pkg: fix no such file error
by Li Ping 19 Feb '21

19 Feb '21

[why]: grep: 2ping.spec: No such file or directory sed: no input files --- tests/rpmbuild-pkg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index 156ca0f63..781d04a0d 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -35,7 +35,7 @@ download_rpm_repo() build_rpm() { spec_dir=${HOME}/rpmbuild/SPECS/$benchmark.spec - sed -i 's/^$Source[^ ]*:[ \t]*$https/\1http/g' `grep http -rl $benchmark.spec` + sed -i 's/^$Source[^ ]*:[ \t]*$https/\1http/g' `grep http -rl $spec_dir` # Install build depends yum-builddep -y $spec_dir # Download tar.gz to default path ${HOME}/rpmbuild/SOURCE -- 2.23.0

1 0

[PATCH v4 compass-ci 1/2] doc: add job/os_mount.md
by Yu Chuan 19 Feb '21

19 Feb '21

Add description of os_mount field in job.yaml. This patch is the description when os_mount=local, and other values of os_mount will be added in the patch later. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- doc/job/os_mount.md | 94 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 94 insertions(+) create mode 100644 doc/job/os_mount.md diff --git a/doc/job/os_mount.md b/doc/job/os_mount.md new file mode 100644 index 000000000000..226251a8bace --- /dev/null +++ b/doc/job/os_mount.md @@ -0,0 +1,94 @@ +# Summary +========= + +`os_mount` defines the mount method of testbox's root partition. + +It has the following optional values: + - nfs + - cifs + - initramfs + - container + - local + +Usage example: + - submit iperf.yaml testbox=vm-2p8g os_mount=nfs + - submit iperf.yaml testbox=vm-2p8g os_mount=cifs + - submit iperf.yaml testbox=vm-2p8g os_mount=initramfs + - submit iperf.yaml testbox=dc-8g os_mount=container + - submit iperf.yaml testbox=vm-2p8g os_mount=local + +# Optional Values +================= + +## Work flow when os_mount=local + +1. user submit job with os_mount: local + - optional {kernel_append_root}: + - save_root_partition: give an iconic string, then the root partition data used by this job will be retained. + and you can use the root partition data of this job in the futher job by {use_root_partition}. + - use_root_partition : give an iconic string, then the root partition data generated by the + pervious job which has the {save_root_partition} field will be used. + - root_partition_size: specify the size of root partition, *default is 10G*. + + - tips: + - value of {save_root_partition} and {use_root_partition} need to be iconic enough, uuid is recommended. + - value of {save_root_partition} and {use_root_partition} will be the suffix of logical volume. + +2. scheduler return the ipxe_str to testbox + ``` + dhcp + initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/{timestamp}/initrd.lkp + initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/{timestamp}/boot/vmlinuz + imgargs vmlinux root=/dev/mapper/os-openeuler_aarch64_20.03 rootfs_src={nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/{timestamp} initrd=initrd.lkp {kernel_append_root} + boot + ``` + +3. dracut step of boot + ``` + if have {use_root_partition}; then + base_lv_name=openeuler_aarch64_20.03_{value_of_use_root_partition} + lvdisplay /dev/mappaer/os-{base_lv_name} > /dev/null || exit 1 + + if have {save_root_partition}; then + boot_lv_name=openeuler_aarch64_20.03_{value_of_save_root_partition} + + if {save_root_partition} == {use_root_partition}; then + boot from /dev/mappaer/os-{boot_lv_name} + fi + + lvremove -f /dev/mappaer/os-{boot_lv_name} + lvcreate /dev/mappaer/os-{boot_lv_name} from /dev/mappaer/os-{base_lv_name} || exit 1 + boot from /dev/mappaer/os-{boot_lv_name} + else + boot_lv_name=openeuler_aarch64_20.03_{value_of_use_root_partition} + boot from /dev/mappaer/os-{boot_lv_name} + fi + + else: + base_lv_name=openeuler_aarch64_20.03_{timestamp} + lvdisplay /dev/mappaer/os-{base_lv_name} > /dev/null || { + # create logical volume + lvcreate -L {root_partition_size} -n {base_lv_name} os || exit 1 + + # rsync nfsroot to lvm:/dev/mapper/os-{base_lv_name} + mount -t nfs {nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/{timestamp} /mnt + mkdir /mnt1 && mount /dev/mapper/os-{base_lv_name} /mnt1 + cp -a /mnt/. /mnt1/ + umount /mnt /mnt1 + + # change premission of lvm:/dev/mapper/os-{base_lv_name} to readonly + lvchange -p r /dev/mapper/os-{base_lv_name} + } + + boot_lv_name=openeuler_aarch64_20.03 + if have {save_root_partition}; then + boot_lv_name=openeuler_aarch64_20.03_{value_of_save_root_partition} + fi + + lvremove -f /dev/mappaer/os-{boot_lv_name} + lvcreate /dev/mappaer/os-{boot_lv_name} from /dev/mappaer/os-{base_lv_name} || exit 1 + boot from /dev/mappaer/os-{boot_lv_name} + fi + ``` + +4. execute the job -- 2.23.0

2 2

[PATCH compass-ci] sched: add rpmbuild-pkg upload dir
by Cao Xueliang 19 Feb '21

19 Feb '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index b724b07..1afc22f 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -154,9 +154,9 @@ class Job set_submit_date() set_pp_params() set_rootfs() + set_os_mount() set_result_root() set_result_service() - set_os_mount() set_depends_initrd() set_kernel() set_initrds_uri() @@ -247,7 +247,7 @@ class Job def get_pkg_common_dir pkg_style = nil - ["cci-makepkg", "cci-depends", "build-pkg"].each do |item| + ["cci-makepkg", "cci-depends", "build-pkg", "rpmbuild-pkg"].each do |item| pkg_style = @hash[item]? break if pkg_style end @@ -276,6 +276,8 @@ class Job package_dir = ",/initrd/pkg/#{common_dir}/#{@hash["cci-makepkg"]["benchmark"]}" elsif @hash["cci-depends"]? package_dir = ",/initrd/deps/#{common_dir}/#{@hash["cci-depends"]["benchmark"]}" + elsif @hash["rpmbuild-pkg"]? + package_dir = ",/initrd/rpmbuild-pkg/#{common_dir}/#{@hash["rpm_repo"].to_s.split("/")[-1]}" elsif @hash["build-pkg"]? if @hash["pkgbuild_repo"].to_s =~ /(packages|community)\/\// package_name = @hash["pkgbuild_repo"].to_s.split("/")[-2] -- 2.23.0

1 0

[PATCH lkp-tests] bootstrap.sh: optimize the test_ping logic
by Yu Chuan 19 Feb '21

19 Feb '21

[Why] 1. '-W' => '-w' -w <deadline> reply wait <deadline> in seconds -W <timeout> time to wait for response deadline(-w) is more appropriate than timeout(-W). 2. delete 'return 1' If ping does not receive any reply packets at all it will exit with code 1. If a packet count and deadline are both specified, and fewer than count packets are received by the time the deadline has arrived, it will also exit with code 1. On other error it exits with code 2. Otherwise it exits with code 0. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- lib/bootstrap.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index 38965ff60755..76f07e6a7669 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -87,7 +87,7 @@ net_devices_link() test_ping() { - ping -c 1 -W 1 $LKP_SERVER > /dev/null || return 1 + ping -c 1 -w 1 $LKP_SERVER > /dev/null } network_ok() -- 2.23.0

1 0