- Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci 3/3] container/assign-account: answerback-email.rb
by Luan Shengde 03 Nov '20

03 Nov '20

1. disable use golbal variable apply_info [why] Style/GlobalVars: Do not introduce global variables. [how] use local variables instead of global variables 2. enable transfer user info when execute apply account command [why] when applying account, the assign-account service will write the my info to user's default config file ~/.config/compass-ci/defaults/account.yaml my info: - my_email - my_name - my_uuid [how] transfer the user info along with the pub_key to assign-account service Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/assign-account/answerback-email.rb | 62 ++++++++++++-------- 1 file changed, 38 insertions(+), 24 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index bb8e809..c174302 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -12,6 +12,7 @@ require 'mail' require 'set' require 'optparse' require_relative '../defconfig' +require_relative '../../lib/es_client' names = Set.new %w[ JUMPER_HOST @@ -24,23 +25,25 @@ defaults = relevant_defaults(names) JUMPER_HOST = defaults['JUMPER_HOST'] || 'api.compass-ci.openeuler.org' JUMPER_PORT = defaults['JUMPER_PORT'] || 29999 -SEND_MAIL_HOST = defaults['SEND_MAIL_HOST_INTERNET'] || 'localhost' -SEND_MAIL_PORT = defaults['SEND_MAIL_PORT_INTERNET'] || 11312 +SEND_MAIL_HOST = defaults['SEND_MAIL_HOST'] || 'localhost' +SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] || 49000 -$apply_info = { +apply_info = { 'my_email' => nil, + 'my_name' => nil, + 'my_uuid' => %x(uuidgen).chomp, 'my_ssh_pubkey' => nil } -def init_info(email_file) +def init_info(email_file, apply_info) mail_content = Mail.read(email_file) + apply_info['my_email'] = mail_content.from[0] + apply_info['my_name'] = mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') + apply_info['my_ssh_pubkey'] = if mail_content.part[1].filename == 'id_rsa.pub' + mail_content.part[1].body.decoded + end - $apply_info['my_email'] = mail_content.from[0] - $apply_info['my_ssh_pubkey'] = if mail_content.part[1].filename == 'id_rsa.pub' - mail_content.part[1].body.decoded.gsub(/\r|\n/, '') - end - - $apply_info + apply_info end options = OptionParser.new do |opts| @@ -52,15 +55,17 @@ options = OptionParser.new do |opts| opts.separator 'options:' opts.on('-e|--email email_address', 'appoint email address') do |email_address| - $apply_info['my_email'] = email_address + apply_info['my_email'] = email_address + # when apply account with email address, will get no user name + apply_info['my_name'] = '' end opts.on('-s|--ssh-pubkey pub_key_file', 'ssh pub_key file, enable password-less login') do |pub_key_file| - $apply_info['my_ssh_pubkey'] = File.read(pub_key_file) + apply_info['my_ssh_pubkey'] = File.read(pub_key_file) end opts.on('-f|--raw-email email_file', 'email file') do |email_file| - init_info(email_file) + init_info(email_file, apply_info) end opts.on_tail('-h|--help', 'show this message') do @@ -94,26 +99,35 @@ def build_message(email, acct_infos) return message end -def account_info(pub_key) - account_info_str = if pub_key.nil? - %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account') - else - %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d "pub_key: #{pub_key}") - end +def account_info(user_info) + account_info_str = %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d '#{user_info.to_json}') JSON.parse account_info_str end -def send_account +def send_account(apply_info) message = "No email address specified\n" message += "use -e email_address add a email address\n" message += 'or use -f to add a email file' - raise message if $apply_info['my_email'].nil? + raise message if apply_info['my_email'].nil? - acct_info = account_info($apply_info['my_ssh_pubkey']) + acct_info = account_info(apply_info) + my_info = { + 'my_email' => apply_info['my_email'], + 'my_name' => apply_info['my_name'], + 'my_commit_url' => '', + 'my_login_name' => acct_info['account'], + 'my_uuid' => apply_info['my_uuid'] + } - message = build_message($apply_info['my_email'], acct_info) + store_account_info(my_info) + message = build_message(apply_info['my_email'], acct_info) %x(curl -XPOST '#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text' -d "#{message}") end -send_account +def store_account_info(my_info) + es = ESClient.new(index: 'accounts') + es.put_source_by_id(my_info['my_email'], my_info) +end + +send_account(apply_info) -- 2.23.0

2 2

[PATCH v2 compass-ci 2/3] container/assign-account: get_account_info
by Luan Shengde 03 Nov '20

03 Nov '20

add new function to config default yaml file [why]: easier for user to config the default yaml file [how]: when get request data, parse the data and write the user infos to default yaml file: ~/.config/compass-ci/default/account.yaml include: - my_email - my_name - my_uuid Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/assign-account/get_account_info.rb | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index 2f93d5b..ae55dcb 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -44,6 +44,8 @@ the returned data for setup_jumper_account_info like: =end +require 'fileutils' + # get jumper and account info class AccountStorage ACCOUNT_DIR = '/opt/account_data/' @@ -93,7 +95,7 @@ class AccountStorage def setup_jumper_account_info account_info = read_account_info jumper_info = read_jumper_info - pub_key = @data['pub_key'] unless @data.nil? + pub_key = @data['my_ssh_pubkey'] unless @data['my_ssh_pubkey'].nil? jumper_ip = jumper_info[0].chomp jumper_port = jumper_info[1].chomp @@ -111,9 +113,23 @@ class AccountStorage } setup_authorized_key(account, pub_key) + setup_default_yaml(account) + return jumper_account_info end + def setup_default_yaml(account) + default_yaml_dir = File.join('/home', account, '.config/compass-ci/defaults') + FileUtils.mkdir_p default_yaml_dir + + File.open("#{default_yaml_dir}/#{account}.yaml", 'a') do |file| + file.puts "my_email: #{@data['my_email']}" + file.puts "my_name: #{@data['my_name']}" + file.puts "my_uuid: #{@data['my_uuid']}" + end + %x(chown -R #{account}:#{account} "/home/#{account}/.config") + end + def setup_authorized_key(account, pub_key) ssh_dir = File.join('/home/', account, '.ssh') Dir.mkdir ssh_dir, 0o700 -- 2.23.0

2 2

[PATCH v3 compass-ci] lib/job.cr: fix pp dir layout
by Wang Yong 03 Nov '20

03 Nov '20

[Why] due to cci-makepkg and cci-depends dir layout change, pp need update together Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- src/lib/job.cr | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 0fd57e0..1e49da6 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -405,8 +405,8 @@ class Job program = $1 end - deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}.cgz" - pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" + deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}/#{program}.cgz" + pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}/latest.cgz" if File.exists?(deps_dest_file) initrd_deps_arr << "#{initrd_http_prefix}" + JobHelper.service_path(deps_dest_file) -- 2.23.0

1 0

[PATCH v3 lkp-tests 2/2] tests/*: fix cci job dir layout
by Wang Yong 03 Nov '20

03 Nov '20

[Why] for multi version support, cgz file for cci-makepkg need change dir layout, and change cci-depends dir layout together to easily update scheduler code path: /srv/initrd/pkg/${os_mount}/${os}/${os_arch}/${os_version}/ => /srv/initrd/pkg/${os_mount}/${os}/${os_arch}/${os_version}/${pkgname}/ filename: cci-makepkg ${pkgver}-${pkgrel}.cgz latest.cgz -> ${pkgver}-${pkgrel}.cgz filename: cci-depends stay original ${benchmark}.cgz ${benchmark}.cgz -> ${benchmark}_${date}.cgz E.g. output: cci-depends stay original $ tree iperf ├── iperf_20201102.cgz └── iperf.cgz -> iperf_20201102.cgz output: cci-makepkg $ tree sar ├── git-1.cgz └── latest.cgz -> git-1.cgz Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/cci-depends | 2 +- tests/cci-makepkg | 12 ++++++------ 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/tests/cci-depends b/tests/cci-depends index 36923ea4d..efa9a025b 100755 --- a/tests/cci-depends +++ b/tests/cci-depends @@ -18,7 +18,7 @@ DEPS_MNT=/initrd/deps && mkdir -p "$DEPS_MNT" DISTRO=${os} [[ "$os_mount" = "cifs" ]] && os_mount="nfs" -pack_to=${DEPS_MNT}/${os_mount}/${os}/${os_arch}/${os_version} +pack_to=${DEPS_MNT}/${os_mount}/${os}/${os_arch}/${os_version}/${benchmark} pack_arch=$os_arch [[ "$benchmark" = "${benchmark##*.}" ]] || diff --git a/tests/cci-makepkg b/tests/cci-makepkg index 9d5082629..9551ebfe2 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -26,7 +26,7 @@ mkdir -p "$PKG_MNT" export DISTRO=${os} [[ "$os_mount" = "cifs" ]] && os_mount="nfs" -pack_to=${os_mount}/${os}/${os_arch}/${os_version} +pack_to=${os_mount}/${os}/${os_arch}/${os_version}/${benchmark} . $LKP_SRC/distro/${DISTRO} . $LKP_SRC/lib/install.sh @@ -65,8 +65,8 @@ update_shared_pkg() [ "$bm_name" = "$benchmark" ] && return # benchmark is a symlink - ln -sf "$bm_link" "$sync_dest/${benchmark}.cgz" || return - echo "update shared pkg link ${benchmark}.cgz -> $bm_link" + ln -sf latest.cgz "$sync_dest/${benchmark}.cgz" || return + echo "update shared pkg link ${benchmark}.cgz -> ${bm_name}/${cgz_name}" } distro_install_depends lkp-dev @@ -86,7 +86,7 @@ date=$(date +"%Y%m%d") pkgver=$(get_pkg_info pkgver) pkgrel=$(get_pkg_info pkgrel) bm_name=$(check_shared_pkg) -cgz_name="${bm_name}-${pkgver:-0}-${pkgrel:-0}_${date}.cgz" +cgz_name="${pkgver:-0}-${pkgrel:-0}.cgz" setup_proxy @@ -98,8 +98,8 @@ update_softlink() { [ -e "$sync_dest/$cgz_name" ] || return - ln -sf "$(basename $(realpath $sync_dest/$cgz_name))" "$sync_dest/${bm_name}.cgz" || return - echo "create package: $sync_dest/${bm_name}.cgz -> $(realpath $sync_dest/$cgz_name)" + ln -sf "$(basename $(realpath $sync_dest/$cgz_name))" "$sync_dest/latest.cgz" || return + echo "create package: $sync_dest/latest.cgz -> $(realpath $sync_dest/$cgz_name)" update_shared_pkg "${bm_name}.cgz" -- 2.23.0

1 0

[PATCH v3 lkp-tests 1/2] fix(distro/depends): add depends file
by Wang Yong 03 Nov '20

03 Nov '20

[Why] there are pack-deps/makepkg depends file before and use nfs-common, now we use cci-depends/cci-makepkg file and use cifs-utils. Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/depends/cci-depends | 2 ++ distro/depends/cci-makepkg | 10 ++++++++++ 2 files changed, 12 insertions(+) create mode 100644 distro/depends/cci-depends create mode 100644 distro/depends/cci-makepkg diff --git a/distro/depends/cci-depends b/distro/depends/cci-depends new file mode 100644 index 000000000..1f7f73ac3 --- /dev/null +++ b/distro/depends/cci-depends @@ -0,0 +1,2 @@ +cifs-utils +cpio diff --git a/distro/depends/cci-makepkg b/distro/depends/cci-makepkg new file mode 100644 index 000000000..a7ee67537 --- /dev/null +++ b/distro/depends/cci-makepkg @@ -0,0 +1,10 @@ +curl +libarchive-tools +bzip2 +fakeroot +gnupg +gettext +openssl +ncurses-bin +binutils +cifs-utils -- 2.23.0

1 0

[PATCH compass-ci] sparrow/7-systemd: fix failed to resume dnsmasq and es service
by Liu Yinsi 03 Nov '20

03 Nov '20

[why] when execute sparrow/4-docker/buildall reboot, error: ==================== es ======================= [2020-11-02T10:24:23,046][INFO ][o.e.n.Node ] [] initializing ... [2020-11-02T10:24:23,084][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main] org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/srv/es/elasticsearch]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])? at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:140) ~[elasticsearch-6.4.3.jar:6.4.3] at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) ~[elasticsearch-6.4.3.jar:6.4.3] at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.4.3.jar:6.4.3] at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.4.3.jar:6.4.3] at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.4.3.jar:6.4.3] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.4.3.jar:6.4.3] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.4.3.jar:6.4.3] =================== dnsmasq =================== dnsmasq: failed to bind DHCP server socket: Address in use because sometimes even if 'docker stop&&rm -f $container', there will still be a legacy process running, causing container to fail to start. [how] kill process before start container Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/7-systemd/cci-network | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sparrow/7-systemd/cci-network b/sparrow/7-systemd/cci-network index 6e59dbb..dfe7cc9 100755 --- a/sparrow/7-systemd/cci-network +++ b/sparrow/7-systemd/cci-network @@ -12,6 +12,9 @@ $CCI_SRC/sparrow/2-network/iptables $CCI_SRC/sparrow/2-network/nfs $CCI_SRC/sparrow/2-network/cifs +# fix failed to start container dnsmasq and es. +kill $(ps -ef| grep -E "dnsmasq|elasticsearch"| awk '{print $2}' ) 2> /dev/null + # --restart=always option is not absolutely reliable # container need to start its dependences firstly. $CCI_SRC/sparrow/4-docker/buildall reboot -- 2.23.0

2 1

[PATCH v2 compass-ci 1/3] container/assign-account: assign-account.rb
by Luan Shengde 03 Nov '20

03 Nov '20

disable assigning account for user if there is no my_email, my_name, my_uuid [why]: my_email, my_name, my_uuid is required when initialize the default config file [how]: check if the data has keys: my_email, my_name, my_uuid when get apply account request. the apply account will return with 'no my info' if missing any of my_email, my_name or my_uuid. Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/assign-account/assign-account.rb | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/container/assign-account/assign-account.rb b/container/assign-account/assign-account.rb index e356c18..3afcc9f 100755 --- a/container/assign-account/assign-account.rb +++ b/container/assign-account/assign-account.rb @@ -7,7 +7,7 @@ require 'sinatra' require 'open3' require 'json' require 'yaml' -require_relative 'get_account_info.rb' +require_relative 'get_account_info' set :bind, '0.0.0.0' set :port, 29999 @@ -17,8 +17,12 @@ get '/assign_account' do data = YAML.safe_load request.body.read rescue StandardError => e puts e.message + puts e.backtrace end + message = 'lack of my infos(my_email, my_name, my_uuid)' + raise message unless %w[my_email my_name my_uuid].to_set.subset? data.keys.to_set + ref_account_info = AccountStorage.new(data) account_info = ref_account_info.setup_jumper_account_info -- 2.23.0

3 4

[PATCH compass-ci] sched: hand over the job from job.queue to extract queue
by Cao Xueliang 03 Nov '20

03 Nov '20

[why] The job add to job.queue, when lkp finish the job need hand over the job from job.queue to extract queue. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/sched.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 46b06da..a4b12b4 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -650,7 +650,7 @@ class Sched end response = @task_queue.hand_over_task( - "sched/#{job.tbox_group}", "extract_stats", job_id + "sched/#{job.queue}", "extract_stats", job_id ) if response[0] != 201 raise "#{response}" -- 2.23.0

1 0

[PATCH compass-ci] sparrow/3-code: add ENV TASKQUEUE_PORT
by Liu Yinsi 03 Nov '20

03 Nov '20

need env TASKQUEUE_PORT to start taskqueue service. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 1 + 1 file changed, 1 insertion(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index f36aaca..64f34f8 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -18,6 +18,7 @@ mkdir -p /etc/compass-ci/defaults cat > /etc/compass-ci/defaults/$server_name.yaml <<EOF SCHED_HOST: $sched_host SCHED_PORT: $sched_port +TASKQUEUE_HOST: $sched_host OS_HTTP_HOST: $os_http_host INITRD_HTTP_HOST: $initrd_http_host INITRD_HTTP_PORT: $initrd_http_port -- 2.23.0

1 0

[PATCH v2 compass-ci] lib/job.cr: fix pp dir layout
by Wang Yong 03 Nov '20

03 Nov '20

[Why] due to cci-makepkg and cci-depends dir layout change, pp need update together Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- src/lib/job.cr | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 955bc4b..4076745 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -434,8 +434,8 @@ class Job program = $1 end - deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}/latest.cgz" - pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}/latest.cgz" + deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}/#{program}.cgz" + pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}/#{program}.cgz" if File.exists?(deps_dest_file) initrd_deps_arr << "#{initrd_http_prefix}" + JobHelper.service_path(deps_dest_file) -- 2.23.0

2 2