- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] sparrow/7-systemd: fix failed to resume dnsmasq and es service
by Liu Yinsi 03 Nov '20

03 Nov '20

[why] when execute sparrow/4-docker/buildall reboot, error: ==================== es ======================= [2020-11-02T10:24:23,046][INFO ][o.e.n.Node ] [] initializing ... [2020-11-02T10:24:23,084][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main] org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/srv/es/elasticsearch]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])? at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:140) ~[elasticsearch-6.4.3.jar:6.4.3] at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) ~[elasticsearch-6.4.3.jar:6.4.3] at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.4.3.jar:6.4.3] at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.4.3.jar:6.4.3] at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.4.3.jar:6.4.3] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.4.3.jar:6.4.3] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.4.3.jar:6.4.3] =================== dnsmasq =================== dnsmasq: failed to bind DHCP server socket: Address in use because sometimes even if 'docker stop&&rm -f $container', there will still be a legacy process running, causing container to fail to start. [how] kill process before start container Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/7-systemd/cci-network | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sparrow/7-systemd/cci-network b/sparrow/7-systemd/cci-network index 6e59dbb..dfe7cc9 100755 --- a/sparrow/7-systemd/cci-network +++ b/sparrow/7-systemd/cci-network @@ -12,6 +12,9 @@ $CCI_SRC/sparrow/2-network/iptables $CCI_SRC/sparrow/2-network/nfs $CCI_SRC/sparrow/2-network/cifs +# fix failed to start container dnsmasq and es. +kill $(ps -ef| grep -E "dnsmasq|elasticsearch"| awk '{print $2}' ) 2> /dev/null + # --restart=always option is not absolutely reliable # container need to start its dependences firstly. $CCI_SRC/sparrow/4-docker/buildall reboot -- 2.23.0

2 1

[PATCH v2 compass-ci 1/3] container/assign-account: assign-account.rb
by Luan Shengde 03 Nov '20

03 Nov '20

disable assigning account for user if there is no my_email, my_name, my_uuid [why]: my_email, my_name, my_uuid is required when initialize the default config file [how]: check if the data has keys: my_email, my_name, my_uuid when get apply account request. the apply account will return with 'no my info' if missing any of my_email, my_name or my_uuid. Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/assign-account/assign-account.rb | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/container/assign-account/assign-account.rb b/container/assign-account/assign-account.rb index e356c18..3afcc9f 100755 --- a/container/assign-account/assign-account.rb +++ b/container/assign-account/assign-account.rb @@ -7,7 +7,7 @@ require 'sinatra' require 'open3' require 'json' require 'yaml' -require_relative 'get_account_info.rb' +require_relative 'get_account_info' set :bind, '0.0.0.0' set :port, 29999 @@ -17,8 +17,12 @@ get '/assign_account' do data = YAML.safe_load request.body.read rescue StandardError => e puts e.message + puts e.backtrace end + message = 'lack of my infos(my_email, my_name, my_uuid)' + raise message unless %w[my_email my_name my_uuid].to_set.subset? data.keys.to_set + ref_account_info = AccountStorage.new(data) account_info = ref_account_info.setup_jumper_account_info -- 2.23.0

3 4

[PATCH compass-ci] sched: hand over the job from job.queue to extract queue
by Cao Xueliang 03 Nov '20

03 Nov '20

[why] The job add to job.queue, when lkp finish the job need hand over the job from job.queue to extract queue. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/sched.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 46b06da..a4b12b4 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -650,7 +650,7 @@ class Sched end response = @task_queue.hand_over_task( - "sched/#{job.tbox_group}", "extract_stats", job_id + "sched/#{job.queue}", "extract_stats", job_id ) if response[0] != 201 raise "#{response}" -- 2.23.0

1 0

[PATCH compass-ci] sparrow/3-code: add ENV TASKQUEUE_PORT
by Liu Yinsi 03 Nov '20

03 Nov '20

need env TASKQUEUE_PORT to start taskqueue service. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 1 + 1 file changed, 1 insertion(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index f36aaca..64f34f8 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -18,6 +18,7 @@ mkdir -p /etc/compass-ci/defaults cat > /etc/compass-ci/defaults/$server_name.yaml <<EOF SCHED_HOST: $sched_host SCHED_PORT: $sched_port +TASKQUEUE_HOST: $sched_host OS_HTTP_HOST: $os_http_host INITRD_HTTP_HOST: $initrd_http_host INITRD_HTTP_PORT: $initrd_http_port -- 2.23.0

1 0

[PATCH v2 compass-ci] lib/job.cr: fix pp dir layout
by Wang Yong 03 Nov '20

03 Nov '20

[Why] due to cci-makepkg and cci-depends dir layout change, pp need update together Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- src/lib/job.cr | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 955bc4b..4076745 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -434,8 +434,8 @@ class Job program = $1 end - deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}/latest.cgz" - pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}/latest.cgz" + deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}/#{program}.cgz" + pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}/#{program}.cgz" if File.exists?(deps_dest_file) initrd_deps_arr << "#{initrd_http_prefix}" + JobHelper.service_path(deps_dest_file) -- 2.23.0

2 2

[PATCH compass-ci] container: fix failed to start git-mirror
by Liu Yinsi 03 Nov '20

03 Nov '20

[why] when start git-mirror container in machine, error: /usr/lib/ruby/2.5.0/net/http.rb:939:in `rescue in block in connect': Failed to open TCP connection to 172.17.0.1:9200 (Connection refused - connect(2) for "172.17.0.1" port 9200) (Faraday::ConnectionFailed) Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/git-mirror/start-depends | 1 + 1 file changed, 1 insertion(+) diff --git a/container/git-mirror/start-depends b/container/git-mirror/start-depends index 5232abf..38fc696 100755 --- a/container/git-mirror/start-depends +++ b/container/git-mirror/start-depends @@ -1 +1,2 @@ rabbitmq +es -- 2.23.0

1 0

[PATCH compass-ci 1/8] lib/mail-robot.rb
by Luan Shengde 03 Nov '20

03 Nov '20

mail robot for applying account monitor mailbox check new added email file in new email directory handle new email file check to apply account only if email's subject match 'apply account' Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- lib/mail-robot.rb | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) create mode 100755 lib/mail-robot.rb diff --git a/lib/mail-robot.rb b/lib/mail-robot.rb new file mode 100755 index 0000000..9437078 --- /dev/null +++ b/lib/mail-robot.rb @@ -0,0 +1,43 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'yaml' +require 'listen' +require 'mail' +require 'fileutils' + +def monitor_dir(mail_inbox, mail_drafts) + listener = Listen.to(mail_inbox) do |_modified, added, _removed| + next if added.empty? + + added.each do |mail_file| + begin + handle_new_email(mail_file, mail_drafts) + rescue StandardError => e + puts e.message + puts e.backtrace + end + end + end + listener.start + sleep +end + +def handle_new_email(mail_file, mail_drafts) + mail_content = Mail.read(mail_file) + check_to_apply_account(mail_content) + + FileUtils.mv(mail_file, mail_drafts) +end + +def check_to_apply_account(mail_content) + return unless mail_content.subject == 'apply account' + + assign_uuid = AssignAccount.new(mail_content) + assign_uuid.check_to_send_account +end + +monitor_dir("#{ENV['MAILDIR']}/new", "#{ENV['MAILDIR']}/cur") -- 2.23.0

2 1

[PATCH lkp-tests] bin/lkp-setup-rootfs: change the script that called when packaging
by Li Ping 02 Nov '20

02 Nov '20

Signed-off-by: Li Ping <15396232681(a)163.com> --- bin/lkp-setup-rootfs | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/bin/lkp-setup-rootfs b/bin/lkp-setup-rootfs index 2890601e..680027e6 100755 --- a/bin/lkp-setup-rootfs +++ b/bin/lkp-setup-rootfs @@ -187,14 +187,14 @@ define_files_pkg_pack() ( local benchmark=${1#pkg/} export benchmark=${benchmark%/PKGBUILD} - $LKP_SRC/tests/makepkg + $LKP_SRC/tests/cci-makepkg ) } # There may be $program-dev or $program-dev-dev in the # "$LKP_SRC/distro/depends" directory, # when the value passed by $define_files is "distro/depends/$program-dev", -# you don't know which script in "makepkg" or "pack-deps" to execute, +# you don't know which script in "cci-makepkg" or "cci-depends" to execute, # so it needs to be handled sparately. # Save the result to # "/srv/initrd/deps/$os/$os_arch/$os_version/$deps_benchmark.cgz", @@ -208,7 +208,7 @@ define_files_deps_pack() [ $suffix == "dev" ] && [ -f "$LKP_SRC/pkg/$pkg_benchmark/PKGBUILD" ] && { ( export benchmark=$pkg_benchmark - $LKP_SRC/tests/makepkg + $LKP_SRC/tests/cci-makepkg ) return 0 } @@ -216,7 +216,7 @@ define_files_deps_pack() ( export benchmark=$deps_benchmark export pack_arch=$os_arch - $LKP_SRC/tests/pack-deps + $LKP_SRC/tests/cci-depends ) } -- 2.23.0

1 0

[PATCH compass-ci 8/8] doc/mail-robot.md
by Luan Shengde 02 Nov '20

02 Nov '20

documentation for mail-robot service Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- doc/mail-robot.md | 76 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+) create mode 100644 doc/mail-robot.md diff --git a/doc/mail-robot.md b/doc/mail-robot.md new file mode 100644 index 0000000..d8ac188 --- /dev/null +++ b/doc/mail-robot.md @@ -0,0 +1,76 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ + +# email monitor robot + +## purpose + +apply account/uuid for user + uuid: used for user to submit jobs + account: used for user to check the test data + +## allowed email format: + +subject: apply account +mail content + my oss commit: commit_url +attahcment + ssh pub_key + - id_rsa.pub + +## steps overview + +1. monitor + - the monitor will listen to the mailbox for new email files + handle new email file: + check to send account + email subject: apply account + +2. assign account + - check email + check commit url + check whether there is a standard commit url + check whether the repo's url in upstream-repos + check whether the commit url valid + if the hub name is gitee.com + # authentication is needed qwhen use curl to check the url + clone the repo and check the commit + if the hub name is not gitee.com + check the comit with curl + + check pub_key + check whether there is a ssh pub_key in the email file + + it will return failure email if any check end with failure + + - generate uuid + - apply account + build user_info + - my_email + - my_name + - my_uuid + - lab + - pub_key + execute command to apply account on jumper and return the account info + curl -XGET "${JUMPER_HOST}:${JUMPER_PORT}/assign_account" -d '${user_info}' + write my_email, my_name, my_uuid, lab to the defaule config file: + + - store apply infos + apply_infos: + - my_email + - my_name + - my_commit_url + - my_login_name + - my_uuid + invoke es to store the apply infos + + - send uuid + build send uuid email message with the apply_infos + send mail + +3. continue the monitor + continue to monitor the mailbox for new email file + cycle run step 1 and 2 if matched email files + +4. service log + use the following command to get the logs + docker logs -f --tail=100 fluentd |grep mail-robot -- 2.23.0

1 0

[PATCH compass-ci 7/8] container/mail-robot: start
by Luan Shengde 02 Nov '20

02 Nov '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/start | 41 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) create mode 100755 container/mail-robot/start diff --git a/container/mail-robot/start b/container/mail-robot/start new file mode 100755 index 0000000..671eaf2 --- /dev/null +++ b/container/mail-robot/start @@ -0,0 +1,41 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig' + +docker_rm 'mail-robot' + +names = Set.new %w[ + JUMPER_HOST + JUMPER_PORT + SEND_MAIL_PORT + MAILDIR + ROBOT_TMP_REPOS_DIR +] + +defaults = relevant_defaults(names) + +defaults['CCI_SRC'] = '/c/compass-ci' + +env = docker_env(defaults) + +cmd = %w[ + docker run + --restart=always + --name=mail-robot + -d +] + env + %W[ + -v /c/upstream-repos:/c/upstream-repos:ro + -v #{defaults['MAILDIR']}:#{defaults['MAILDIR']}:rw + -v #{ENV['CCI_SRC']}:#{defaults['CCI_SRC']}:rw + -w #{defaults['CCI_SRC']}/container/mail-robot + --log-driver json-file + mail-robot +] + +cmd += ['ruby', 'run.rb'] + +system(*cmd) -- 2.23.0

1 0