November 2020 - Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] sparrow/3-code: umask 002 before download git trees
by Liu Yinsi 18 Nov '20

18 Nov '20

because ENV CCI_SRC LKP_SRC, REPO_SRC all under /c, some containers service start need umask 002, set it before download git trees. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/git | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sparrow/3-code/git b/sparrow/3-code/git index 5588696..494107e 100755 --- a/sparrow/3-code/git +++ b/sparrow/3-code/git @@ -2,6 +2,9 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# fix files permissions cause container service start failed +umask 002 + # download git trees cd /c || exit -- 2.23.0

2 2

[PATCH compass-ci] mail-robot: Dockerfile
by Luan Shengde 18 Nov '20

18 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/Dockerfile | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 container/mail-robot/Dockerfile diff --git a/container/mail-robot/Dockerfile b/container/mail-robot/Dockerfile new file mode 100644 index 0000000..76dc94d --- /dev/null +++ b/container/mail-robot/Dockerfile @@ -0,0 +1,14 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM debian + +MAINTAINER Luan Shengde <shdluan(a)163.com> + +ENV DEBIAN_FRONTEND noninteractive + +RUN apt-get update && \ + apt-get install -y git uuid-runtime curl ruby-listen ruby-json ruby-mail + +RUN echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem install fileutils elasticsearch activesupport -- 2.23.0

1 0

[PATCH v9 compass-ci 08/11] mail-robot: Dockerfile
by Luan Shengde 18 Nov '20

18 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/Dockerfile | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 container/mail-robot/Dockerfile diff --git a/container/mail-robot/Dockerfile b/container/mail-robot/Dockerfile new file mode 100644 index 0000000..b3bceef --- /dev/null +++ b/container/mail-robot/Dockerfile @@ -0,0 +1,16 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM debian + +MAINTAINER Luan Shengde <shdluan(a)163.com> + +ENV DEBIAN_FRONTEND noninteractive + +RUN apt-get update && \ + apt-get install -y git uuid-runtime curl ruby-listen ruby-json ruby-mail + +RUN echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem install fileutils elasticsearch activesupport + +RUN umask 0000 -- 2.23.0

2 2

[PATCH v4 lkp-tests] tests/sysbench-threads-git: update command parameters
by Hu Xuejiao 18 Nov '20

18 Nov '20

[why] The current result: value of "total_time" is the same for different threads: nr_threads: 2 "total_time" is 0.3927 nr_threads: 4 "total_time" is 0.3927 ... nr_threads: 8192 "total_time" is 0.3927 [how] After updating --threads to --num-threads: nr_threads: 2 "total_time" is 0.3927 nr_threads: 4 "total_time" is 0.5643 ... nr_threads: 8192 "total_time" is 1.678 Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- tests/sysbench-threads-git | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/sysbench-threads-git b/tests/sysbench-threads-git index df1bb5cf..be83b441 100755 --- a/tests/sysbench-threads-git +++ b/tests/sysbench-threads-git @@ -10,7 +10,7 @@ : "${thread_locks:=2}" args=( - --threads=$nr_threads + --num-threads=$nr_threads --thread-yields=$thread_yields --thread-locks=$thread_locks ) -- 2.23.0

1 0

[PATCH v9 lkp-tests] add on_fail function when submit
by Hu Xuejiao 18 Nov '20

18 Nov '20

[why] When the task is failed(state), it need in the waiting state Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/submit | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/sbin/submit b/sbin/submit index 6f603abe..4e80deef 100755 --- a/sbin/submit +++ b/sbin/submit @@ -116,12 +116,14 @@ ARGV.each do |jobfile| sh_run_job = job.sh_run_job sh_extract_stats = job.sh_extract_stats sh_define_files = job.sh_define_files + sh_on_fail = job.sh_on_state(state='on_fail') sh_hash = { 'job2sh' => { 'run_job' => sh_run_job, 'extract_stats' => sh_extract_stats, - 'define_files' => sh_define_files + 'define_files' => sh_define_files, + 'on_fail' => sh_on_fail } } -- 2.23.0

1 0

[PATCH v3 compass-ci 1/4] assign-account: Dockerfile add install ssh
by Luan Shengde 18 Nov '20

18 Nov '20

add install ssh for assign-account [why] need ssh-keygen to generate rsa public/private key for user Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/Dockerfile | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/assign-account/Dockerfile b/container/assign-account/Dockerfile index f84b5e6..12faafa 100644 --- a/container/assign-account/Dockerfile +++ b/container/assign-account/Dockerfile @@ -2,13 +2,13 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. FROM debian -MAINTAINER luanshd0525(a)163.com +MAINTAINER Luan Shengde <shdluan(a)163.com> ENV DEBIAN_FRONTEND noninteractive COPY conf/sources.list* /etc/apt/ RUN apt-get update && \ - apt-get install -y ruby-sinatra ruby-json curl && \ + apt-get install -y ssh ruby-sinatra ruby-json curl && \ ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \ echo 'Asia/Shanghai' > /etc/timezone COPY assign-account.rb /usr/local/bin/ -- 2.23.0

2 2

[PATCH v6 compass-ci 10/12] mail-robot: Dockerfile
by Luan Shengde 18 Nov '20

18 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/Dockerfile | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 container/mail-robot/Dockerfile diff --git a/container/mail-robot/Dockerfile b/container/mail-robot/Dockerfile new file mode 100644 index 0000000..76dc94d --- /dev/null +++ b/container/mail-robot/Dockerfile @@ -0,0 +1,14 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM debian + +MAINTAINER Luan Shengde <shdluan(a)163.com> + +ENV DEBIAN_FRONTEND noninteractive + +RUN apt-get update && \ + apt-get install -y git uuid-runtime curl ruby-listen ruby-json ruby-mail + +RUN echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem install fileutils elasticsearch activesupport -- 2.23.0

2 2

[PATCH lkp-tests] stats/hackbench: fix rubocop offenses
by Zhang Yu 18 Nov '20

18 Nov '20

[errors] Inspecting 1 file offenses: hackbench:26:18: C: [Corrected] Style/RedundantSort: Use min instead of sort...first. best_time = time.sort.first ^^^^^^^^^^ Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- stats/hackbench | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/stats/hackbench b/stats/hackbench index 9cd43277..89b2be35 100755 --- a/stats/hackbench +++ b/stats/hackbench @@ -23,5 +23,5 @@ while (line = gets) end kb = tasks * messages * bytes >> 10 -best_time = time.sort.first +best_time = time.min puts "throughput: #{kb / best_time}" -- 2.23.0

1 0

[PATCH compass-ci] sbin/es-jobs-mapping.sh: disable default dynamic mapping
by Zhang Yuhang 18 Nov '20

18 Nov '20

[why] By default, fields can be added dynamically to a document, or to inner objects within a document, just by indexing a document containing the new field. However, there are many fields that we do not need, and "Limit of total fields [1000] in index [jobs] has been exceeded" exception will be reported when the number of fields exceeds 1000. [how] Disable default dynamic mapping. Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- sbin/es-jobs-mapping.sh | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/sbin/es-jobs-mapping.sh b/sbin/es-jobs-mapping.sh index 9d420f7..076bdd3 100755 --- a/sbin/es-jobs-mapping.sh +++ b/sbin/es-jobs-mapping.sh @@ -38,6 +38,7 @@ else curl -H 'Content-Type: Application/json' -XPUT 'http://localhost:9200/jobs' -d '{ "mappings": { "_doc": { + "dynamic": false, "dynamic_templates": [ { "pp": { @@ -61,6 +62,10 @@ else } ], "properties": { + "pp": { + "dynamic": true, + "properties": {} + }, "suite": { "type": "keyword" }, -- 2.23.0

2 2

[PATCH v9 compass-ci 11/11] mail-robot: documentation for service
by Luan Shengde 18 Nov '20

18 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/mail-robot.md | 98 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 98 insertions(+) create mode 100644 doc/mail-robot.md diff --git a/doc/mail-robot.md b/doc/mail-robot.md new file mode 100644 index 0000000..ec1a866 --- /dev/null +++ b/doc/mail-robot.md @@ -0,0 +1,98 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ + +# email monitor robot + +## purpose + +apply account/uuid for user + uuid: used for user to submit jobs + account: used for user to check the test data + +## usage + +build email as the following section: allowed email format +send mail to: compass-ci(a)qq.com + +## allowed email format: + +subject: apply account +mail content + my oss commit: commit_url + example: + my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… +attahcment + ssh pub_key + +## steps overview + +1. mail-robot + monitor_new_email + - the monitor will listen to the mailbox for new email files + handle_new_email: + read email file content + apply_account + invoke AssignAccount for new account + +2. apply-account + init my_info + - my_email + - my_name + check_to_send_account + - parse_mail_content + invoke ParseApplyAccountEmail + parse_commit_url + extract_commit_url + check whether there is a standard commit url + base_url_in_upstream_repos + check whether the repo's url in upstream-repos + commit_url_availability + check whether the commit url valid + - gitee_commit + clone the repo and check the commit + - non_gitee_commit(url) + check the commit with curl + + parse_pub_key + check whether there has an attachment file to the email file + attachment: + first attachment + + - apply_my_account + my_uuid + generate uuid + apply_account + invoke ApplyJumperAccount to apply new account with my_info and pub_key + my_info: + - my_email + - my_name + - my_uuid + apply_jumper_account + required data: pub_key + complete my_info + my_info add: + - my_login_name + - my_commit_url + store my_info + invoke es to store the apply infos + + - rescue error + error type: + - no commit url + - commit url not in upstream-repos + - commit url not available + - no pub_key + - no more available account + build_error_email with raised error message + + - send_mail + - build success email + - build failed email + send mail + +3. continue the monitor + continue to monitor the mailbox for new email file + cycle run step 1 and 2 if matched email files + +4. service log + use the following command to get the logs + docker logs -f --tail=100 fluentd | grep mail-robot -- 2.23.0

1 0