- Kernel - mailweb.openeuler.org

openEuler 20.03 LTS SP1 SP2 兼容性 KABI 白名单初稿预审
by Xie XiuQi 08 Mar '21

08 Mar '21

各位好，根据前期沟通，OSV、驱动兼容性SIG都对内核接口兼容性提出了诉求。根据前期沟通的信息，以及收集的驱动接口使用情况，形成一份 20.03 LTS SP1 SP2 兼容性 KABI 白名单初稿，请大家评审。 KABI (Kernel Application Binary Interface) 兼容，即内核与驱动的二进制兼容。就是驱动不用重新编译，就可以在新内核上安装使用。如果驱动用到的接口都是兼容的，那么驱动就可以不用重新编译就可以在新版本安装使用。上游社区考虑到便于开发、和防止架构腐化，不考虑 KABI 的兼容性。业界的 KABI 兼容都是由 Linux 发行版来做的，而且兼容的接口越多、维护时间越长，维护成本也越高。 openEuler 通过多次与下游OSV和驱动团队交流，根据下游反馈的诉求， openEuler 20.03 LTS SP1、SP2 版本，提供一定范围的 KABI 兼容能力，目的是使常用的板卡驱动，在SP1、后续的 SP1 update 版本、SP2及SP2 update版本能兼容。 kabi 兼容列表白名单的主要输入是驱动（ko），我们根据下游的反馈，收集到部分常用驱动，作为兼容的目标，驱动列表如下。 ARM64 版本拟兼容的驱动（ko）列表 |amdgpu.ko bnx2.ko bnx2x.ko bnxt_en.ko bnxt_re.ko hclge.ko hclgevf.ko hifc.ko hinic.ko hnae3.ko hns3.ko i40e.ko ice.ko igb.ko ixgbe.ko ixgbevf.ko lpfc.ko megaraid_sas.ko mlx4_core.ko mlx4_ib.ko mlx5_core.ko mlx5_ib.ko mpt3sas.ko nouveau.ko nvme.ko qed.ko qede.ko qla2xxx.ko smartpqi.ko tg3.ko txgbe.ko | X86 版本拟兼容的驱动（ko）列表 |amdgpu.ko bnx2.ko bnx2x.ko bnxt_en.ko bnxt_re.ko hifc.ko hinic.ko i40e.ko ice.ko igb.ko ixgbe.ko ixgbevf.ko lpfc.ko megaraid_sas.ko mlx4_core.ko mlx4_ib.ko mlx5_core.ko mlx5_ib.ko mpt3sas.ko nouveau.ko nvme.ko qed.ko qede.ko qla2xxx.ko smartpqi.ko tg3.ko txgbe.ko | 我们收集了上表中驱动使用到接口，形成了一份 kabi 白名单列表初稿，请大家评审。重要的说明和提示： 1. 上述列表中多数驱动，还没有针对 openEuler SP1 的正式二进制版本，因此我们根据开源的版本或相近的版本收集了相关 kabi 列表，可能与板卡厂商的最终发布的版本存在少量差异。下游OSV或驱动团队如果发现有接口没在给出的列表中的，可以在评审过程中提出来。 2. 如果有新增 KABI 兼容性的诉求，需要给出需要兼容的 kabi 接口名称，以及使用到的驱动名称，以方便评估。 3. 评审反馈，可以在 issue 中反馈，https://gitee.com/openeuler/kernel/issues/I3ABVJ 也可以通过回复邮件反馈。 4. 收集评审反馈信息的时间为1周，截止到下周五（3月12日）下午17:00. --- openEuler kernel SIG， 2021-3-6 --- 附1：ARM64 平台 KABI 白名单列表初稿（1905个） acpi_bus_get_device acpi_check_dsm acpi_dev_found acpi_disabled acpi_dma_configure acpi_evaluate_dsm acpi_evaluate_object acpi_format_exception acpi_gbl_FADT acpi_get_devices acpi_get_handle acpi_get_name acpi_get_table acpi_gsi_to_irq acpi_handle_printk acpi_has_method acpi_lid_open acpi_os_map_memory acpi_os_unmap_memory acpi_register_gsi acpi_unregister_gsi add_timer add_wait_queue add_wait_queue_exclusive admin_timeout alloc_chrdev_region alloc_cpu_rmap __alloc_disk_node alloc_etherdev_mqs alloc_netdev_mqs alloc_pages_current __alloc_pages_nodemask __alloc_percpu __alloc_skb __alloc_workqueue_key __arch_clear_user __arch_copy_from_user __arch_copy_in_user __arch_copy_to_user arch_timer_read_counter arch_wb_cache_pmem arm64_const_caps_ready arp_tbl async_schedule _atomic_dec_and_lock atomic_notifier_call_chain atomic_notifier_chain_register atomic_notifier_chain_unregister attribute_container_find_class_device autoremove_wake_function backlight_device_register backlight_device_unregister backlight_force_update bdevname bdev_read_only bdget_disk _bin2bcd bio_add_page bio_alloc_bioset bio_clone_fast bio_endio bio_free_pages bio_init bio_put bioset_exit bioset_init __bitmap_and __bitmap_andnot __bitmap_clear __bitmap_complement __bitmap_equal bitmap_find_free_region bitmap_find_next_zero_area_off bitmap_free __bitmap_intersects __bitmap_or __bitmap_parse bitmap_parselist bitmap_print_to_pagebuf bitmap_release_region __bitmap_set __bitmap_weight __bitmap_xor bitmap_zalloc bit_wait blk_alloc_queue blk_check_plugged blk_cleanup_queue blkdev_get_by_path blkdev_issue_discard blkdev_issue_write_same blkdev_issue_zeroout blkdev_put blk_execute_rq blk_execute_rq_nowait blk_finish_plug blk_get_queue blk_get_request blk_mq_alloc_tag_set blk_mq_complete_request blk_mq_end_request blk_mq_free_request blk_mq_free_tag_set blk_mq_init_queue blk_mq_map_queues blk_mq_pci_map_queues blk_mq_quiesce_queue blk_mq_run_hw_queues blk_mq_start_request blk_mq_tagset_busy_iter blk_mq_tag_to_rq blk_mq_unique_tag blk_mq_unquiesce_queue blk_mq_update_nr_hw_queues blk_put_queue blk_put_request blk_queue_bounce_limit blk_queue_dma_alignment blk_queue_flag_clear blk_queue_flag_set blk_queue_io_min blk_queue_io_opt blk_queue_logical_block_size blk_queue_make_request blk_queue_max_discard_sectors blk_queue_max_hw_sectors blk_queue_max_segments blk_queue_max_segment_size blk_queue_max_write_same_sectors blk_queue_physical_block_size blk_queue_rq_timeout blk_queue_segment_boundary blk_queue_split blk_queue_stack_limits blk_queue_update_dma_alignment blk_queue_virt_boundary blk_queue_write_cache blk_rq_append_bio blk_rq_count_integrity_sg blk_rq_map_integrity_sg blk_rq_map_kern blk_rq_map_sg blk_rq_map_user_iov blk_rq_unmap_user blk_set_stacking_limits blk_start_plug blk_status_to_errno blk_verify_command blocking_notifier_call_chain blocking_notifier_chain_register blocking_notifier_chain_unregister bpf_prog_add bpf_prog_inc bpf_prog_put bpf_trace_run1 bpf_trace_run2 bpf_trace_run3 bpf_trace_run5 bpf_warn_invalid_xdp_action bsg_job_done btree_destroy btree_geo32 btree_geo64 btree_get_prev btree_init btree_insert btree_last btree_lookup btree_remove btree_update build_skb cache_line_size call_netdevice_notifiers call_rcu_sched call_srcu call_usermodehelper cancel_delayed_work cancel_delayed_work_sync cancel_work_sync capable cdev_add cdev_del cdev_init __chash_table_copy_in __chash_table_copy_out __check_object_size __class_create class_destroy __class_register class_unregister _cleanup_srcu_struct clk_get_rate commit_creds compat_alloc_user_space complete complete_all complete_and_exit completion_done component_add component_del _cond_resched console_lock console_unlock __const_udelay consume_skb cpu_bit_bitmap cpufreq_quick_get __cpuhp_remove_state __cpuhp_setup_state cpu_hwcap_keys cpu_hwcaps cpumask_local_spread cpumask_next cpumask_next_and cpu_number __cpu_online_mask __cpu_possible_mask __cpu_present_mask cpus_read_lock cpus_read_unlock crc32c crc32_le crc8 crc8_populate_msb crc_t10dif crypto_ahash_digest crypto_ahash_final crypto_ahash_setkey crypto_alloc_ahash crypto_destroy_tfm crypto_register_shash crypto_unregister_shash csum_ipv6_magic csum_partial csum_tcpudp_nofold _ctype dcb_getapp dcb_ieee_delapp dcb_ieee_getapp_mask dcb_ieee_setapp dcbnl_cee_notify dcb_setapp debugfs_create_atomic_t debugfs_create_dir debugfs_create_file debugfs_create_symlink debugfs_create_u16 debugfs_create_u32 debugfs_create_u64 debugfs_create_u8 debugfs_lookup debugfs_remove default_llseek default_wake_function __delay delayed_work_timer_fn del_gendisk del_timer del_timer_sync destroy_workqueue dev_add_pack dev_addr_add dev_addr_del dev_base_lock dev_close dev_driver_string _dev_err __dev_get_by_index dev_get_by_index dev_get_by_index_rcu dev_get_by_name dev_get_iflink dev_get_stats device_add_disk device_create device_create_file device_destroy device_link_add device_release_driver device_remove_file device_reprobe device_set_wakeup_capable device_set_wakeup_enable _dev_info __dev_kfree_skb_any __dev_kfree_skb_irq devlink_alloc devlink_free devlink_param_driverinit_value_get devlink_param_driverinit_value_set devlink_params_register devlink_params_unregister devlink_param_value_changed devlink_port_attrs_set devlink_port_register devlink_port_type_clear devlink_port_type_eth_set devlink_port_type_ib_set devlink_port_unregister devlink_region_create devlink_region_destroy devlink_region_shapshot_id_get devlink_region_snapshot_create devlink_register devlink_unregister dev_mc_add dev_mc_add_excl dev_mc_del devm_free_irq devm_hwmon_device_register_with_groups devm_ioremap devm_iounmap devm_kfree devm_kmalloc devm_kmemdup devm_mdiobus_alloc_size devm_request_threaded_irq _dev_notice dev_open dev_printk dev_queue_xmit dev_remove_pack dev_set_mac_address dev_set_mtu dev_set_promiscuity dev_trans_start dev_uc_add dev_uc_add_excl dev_uc_del _dev_warn disable_irq disable_irq_nosync dma_alloc_from_dev_coherent dma_common_get_sgtable dma_common_mmap dma_fence_add_callback dma_fence_array_create dma_fence_context_alloc dma_fence_free dma_fence_get_status dma_fence_init dma_fence_release dma_fence_signal dma_fence_signal_locked dma_fence_wait_any_timeout dma_fence_wait_timeout dma_get_required_mask dmam_alloc_coherent dmam_free_coherent dma_pool_alloc dma_pool_create dma_pool_destroy dma_pool_free dma_release_from_dev_coherent dmi_check_system dmi_get_system_info dmi_match __do_once_done __do_once_start do_wait_intr down downgrade_write down_interruptible down_read down_read_trylock down_timeout down_trylock down_write down_write_killable down_write_trylock dput dql_completed dql_reset drain_workqueue driver_create_file driver_for_each_device driver_remove_file drm_add_edid_modes drm_add_modes_noedid drm_atomic_add_affected_connectors drm_atomic_add_affected_planes drm_atomic_commit drm_atomic_get_connector_state drm_atomic_get_crtc_state drm_atomic_get_plane_state drm_atomic_helper_check drm_atomic_helper_check_modeset drm_atomic_helper_check_planes drm_atomic_helper_check_plane_state drm_atomic_helper_cleanup_planes drm_atomic_helper_commit drm_atomic_helper_commit_cleanup_done drm_atomic_helper_commit_hw_done __drm_atomic_helper_connector_destroy_state drm_atomic_helper_connector_destroy_state __drm_atomic_helper_connector_duplicate_state __drm_atomic_helper_connector_reset __drm_atomic_helper_crtc_destroy_state __drm_atomic_helper_crtc_duplicate_state drm_atomic_helper_disable_plane drm_atomic_helper_legacy_gamma_set drm_atomic_helper_page_flip __drm_atomic_helper_plane_destroy_state drm_atomic_helper_plane_destroy_state __drm_atomic_helper_plane_duplicate_state drm_atomic_helper_prepare_planes drm_atomic_helper_resume drm_atomic_helper_set_config drm_atomic_helper_setup_commit drm_atomic_helper_shutdown drm_atomic_helper_suspend drm_atomic_helper_swap_state drm_atomic_helper_update_legacy_modeset_state drm_atomic_helper_update_plane drm_atomic_helper_wait_for_dependencies drm_atomic_helper_wait_for_fences drm_atomic_helper_wait_for_flip_done drm_atomic_state_alloc drm_atomic_state_default_clear drm_atomic_state_default_release __drm_atomic_state_free drm_atomic_state_init drm_calc_vbltimestamp_from_scanoutpos drm_color_lut_extract drm_compat_ioctl drm_connector_attach_encoder drm_connector_cleanup drm_connector_init drm_connector_list_iter_begin drm_connector_list_iter_end drm_connector_list_iter_next drm_connector_register drm_connector_set_path_property drm_connector_unregister drm_connector_update_edid_property drm_crtc_accurate_vblank_count drm_crtc_add_crc_entry drm_crtc_arm_vblank_event drm_crtc_cleanup __drm_crtc_commit_free drm_crtc_enable_color_mgmt drm_crtc_force_disable_all drm_crtc_from_index drm_crtc_handle_vblank drm_crtc_helper_set_config drm_crtc_helper_set_mode drm_crtc_init drm_crtc_init_with_planes drm_crtc_send_vblank_event drm_crtc_vblank_count drm_crtc_vblank_get drm_crtc_vblank_off drm_crtc_vblank_on drm_crtc_vblank_put drm_cvt_mode drm_dbg drm_debugfs_create_files drm_detect_hdmi_monitor drm_detect_monitor_audio drm_dev_alloc drm_dev_put drm_dev_register drm_dev_unregister drm_dp_atomic_find_vcpi_slots drm_dp_atomic_release_vcpi_slots drm_dp_aux_register drm_dp_aux_unregister drm_dp_bw_code_to_link_rate drm_dp_calc_pbn_mode drm_dp_channel_eq_ok drm_dp_check_act_status drm_dp_clock_recovery_ok drm_dp_dpcd_read drm_dp_dpcd_read_link_status drm_dp_dpcd_write drm_dp_find_vcpi_slots drm_dp_get_adjust_request_pre_emphasis drm_dp_get_adjust_request_voltage drm_dp_link_rate_to_bw_code drm_dp_link_train_channel_eq_delay drm_dp_link_train_clock_recovery_delay drm_dp_mst_allocate_vcpi drm_dp_mst_deallocate_vcpi drm_dp_mst_detect_port drm_dp_mst_get_edid drm_dp_mst_hpd_irq drm_dp_mst_reset_vcpi_slots drm_dp_mst_topology_mgr_destroy drm_dp_mst_topology_mgr_init drm_dp_mst_topology_mgr_resume drm_dp_mst_topology_mgr_set_mst drm_dp_mst_topology_mgr_suspend drm_dp_update_payload_part1 drm_dp_update_payload_part2 drm_edid_header_is_valid drm_edid_is_valid drm_edid_to_sad drm_edid_to_speaker_allocation drm_encoder_cleanup drm_encoder_init drm_err drm_fb_helper_add_one_connector drm_fb_helper_alloc_fbi drm_fb_helper_blank drm_fb_helper_cfb_copyarea drm_fb_helper_cfb_fillrect drm_fb_helper_cfb_imageblit drm_fb_helper_check_var drm_fb_helper_debug_enter drm_fb_helper_debug_leave drm_fb_helper_fill_fix drm_fb_helper_fill_var drm_fb_helper_fini drm_fb_helper_hotplug_event drm_fb_helper_init drm_fb_helper_initial_config drm_fb_helper_ioctl drm_fb_helper_lastclose drm_fb_helper_output_poll_changed drm_fb_helper_pan_display drm_fb_helper_prepare drm_fb_helper_remove_one_connector drm_fb_helper_setcmap drm_fb_helper_set_par drm_fb_helper_set_suspend drm_fb_helper_set_suspend_unlocked drm_fb_helper_single_add_all_connectors drm_fb_helper_unregister_fbi drm_format_plane_cpp drm_framebuffer_cleanup drm_framebuffer_init drm_framebuffer_unregister_private drm_gem_dmabuf_kmap drm_gem_dmabuf_kunmap drm_gem_dmabuf_mmap drm_gem_dmabuf_release drm_gem_dmabuf_vmap drm_gem_dmabuf_vunmap drm_gem_fb_create_handle drm_gem_fb_destroy drm_gem_handle_create drm_gem_handle_delete drm_gem_map_attach drm_gem_map_detach drm_gem_map_dma_buf drm_gem_object_init drm_gem_object_lookup drm_gem_object_put_unlocked drm_gem_object_release drm_gem_prime_export drm_gem_prime_fd_to_handle drm_gem_prime_handle_to_fd drm_gem_prime_import drm_gem_private_object_init drm_gem_unmap_dma_buf drm_get_edid drm_get_edid_switcheroo drm_get_format_name drm_get_max_iomem drm_global_item_ref drm_global_item_unref drm_handle_vblank drm_hdmi_avi_infoframe_from_display_mode drm_hdmi_vendor_infoframe_from_display_mode drm_helper_connector_dpms drm_helper_disable_unused_functions drm_helper_hpd_irq_event drm_helper_mode_fill_fb_struct drm_helper_probe_single_connector_modes drm_helper_resume_force_mode drm_i2c_encoder_detect drm_i2c_encoder_init drm_i2c_encoder_mode_fixup drm_i2c_encoder_restore drm_i2c_encoder_save drm_invalid_op drm_ioctl drm_irq_install drm_irq_uninstall drm_is_current_master drm_kms_helper_hotplug_event drm_kms_helper_is_poll_worker drm_kms_helper_poll_disable drm_kms_helper_poll_enable drm_kms_helper_poll_fini drm_kms_helper_poll_init drm_match_cea_mode drm_mm_init drm_mm_insert_node_in_range drm_mm_print drm_mm_remove_node drm_mm_takedown drm_mode_config_cleanup drm_mode_config_init drm_mode_config_reset drm_mode_copy drm_mode_create_dvi_i_properties drm_mode_create_scaling_mode_property drm_mode_create_tv_properties drm_mode_crtc_set_gamma_size drm_mode_debug_printmodeline drm_mode_destroy drm_mode_duplicate drm_mode_equal drm_mode_get_hv_timing drm_mode_is_420_only drm_mode_legacy_fb_format drm_mode_object_find drm_mode_object_put drm_mode_probed_add drm_mode_set_crtcinfo drm_modeset_lock drm_modeset_lock_all drm_modeset_lock_all_ctx drm_mode_set_name drm_modeset_unlock drm_modeset_unlock_all drm_mode_vrefresh drm_object_attach_property drm_object_property_set_value drm_open drm_plane_cleanup drm_plane_create_alpha_property drm_plane_create_color_properties drm_plane_create_zpos_immutable_property drm_plane_create_zpos_property drm_plane_force_disable drm_plane_init drm_poll drm_primary_helper_destroy drm_primary_helper_funcs drm_prime_gem_destroy drm_prime_pages_to_sg drm_prime_sg_to_page_addr_arrays drm_printf __drm_printfn_seq_file drm_property_add_enum drm_property_create drm_property_create_enum drm_property_create_range __drm_puts_seq_file drm_read drm_release drm_scdc_read drm_scdc_write drm_sched_dependency_optimized drm_sched_entity_destroy drm_sched_entity_fini drm_sched_entity_flush drm_sched_entity_init drm_sched_entity_push_job drm_sched_entity_set_rq drm_sched_fini drm_sched_hw_job_reset drm_sched_init drm_sched_job_init drm_sched_job_recovery drm_send_event_locked drm_syncobj_create drm_syncobj_find drm_syncobj_find_fence drm_syncobj_free drm_syncobj_get_fd drm_syncobj_get_handle drm_syncobj_replace_fence drm_universal_plane_init drm_vblank_init drm_vma_node_allow drm_vma_node_is_allowed drm_vma_node_revoke dst_release dummy_dma_ops dump_stack __dynamic_dev_dbg __dynamic_netdev_dbg __dynamic_pr_debug elfcorehdr_addr emergency_restart enable_irq errno_to_blk_status ether_setup eth_get_headlen eth_platform_get_mac_address ethtool_convert_legacy_u32_to_link_mode ethtool_convert_link_mode_to_legacy_u32 ethtool_intersect_link_masks ethtool_op_get_link ethtool_op_get_ts_info eth_type_trans eth_validate_addr eventfd_ctx_fdget eventfd_ctx_put eventfd_signal event_triggers_call fasync_helper fc_attach_transport fc_block_scsi_eh fc_eh_timed_out fc_get_event_number fc_host_post_event fc_host_post_vendor_event fc_release_transport fc_remote_port_add fc_remote_port_delete fc_remote_port_rolechg fc_remove_host fc_vport_create fc_vport_terminate fd_install fget filemap_fault filp_close filp_open find_get_pid find_last_bit find_next_bit find_next_zero_bit find_pid_ns find_vma finish_wait firmware_request_nowarn fixed_size_llseek flow_keys_dissector flush_delayed_work flush_signals flush_work flush_workqueue force_sig fortify_panic fput free_fib_info free_irq free_irq_cpu_rmap free_netdev __free_pages free_pages free_percpu from_kgid from_kuid fs_bio_set gcd generate_random_uuid generic_end_io_acct generic_file_llseek generic_handle_irq generic_make_request generic_start_io_acct genlmsg_put genl_register_family genl_unregister_family genphy_read_status genphy_restart_aneg get_device __get_free_pages get_gendisk get_pid_task get_random_bytes __get_task_comm get_task_mm get_unused_fd_flags get_user_pages get_user_pages_remote get_zeroed_page gic_pmr_sync groups_alloc groups_free handle_simple_irq hdmi_avi_infoframe_pack hdmi_infoframe_pack hnae3_register_ae_algo hnae3_register_ae_dev hnae3_register_client hnae3_set_client_init_flag hnae3_unregister_ae_algo hnae3_unregister_ae_dev hnae3_unregister_client hrtimer_cancel hrtimer_forward hrtimer_init hrtimer_start_range_ns hrtimer_try_to_cancel __hw_addr_sync_dev __hw_addr_unsync_dev hwmon_device_register hwmon_device_register_with_groups hwmon_device_register_with_info hwmon_device_unregister i2c_add_adapter i2c_bit_add_bus i2c_bit_algo i2c_del_adapter i2c_new_device i2c_smbus_read_byte_data i2c_smbus_write_byte_data i2c_transfer i2c_unregister_device __ib_alloc_cq ib_alloc_device ib_alloc_odp_umem __ib_alloc_pd __ib_alloc_xrcd __ib_create_cq ib_create_qp ib_create_send_mad ib_dealloc_device ib_dealloc_pd ib_dereg_mr ib_destroy_cq ib_destroy_qp ib_dispatch_event ib_find_cached_pkey ib_free_cq ib_free_send_mad ib_get_cached_pkey ib_get_eth_speed ib_get_gids_from_rdma_hdr ib_get_rdma_header_version ib_modify_qp ib_modify_qp_is_ok ib_post_send_mad ib_process_cq_direct ib_query_pkey ib_query_port ib_query_qp ib_register_device ib_register_mad_agent ib_sa_cancel_query ib_sa_guid_info_rec_query ib_sa_register_client ib_sa_unregister_client ib_sg_to_pages ib_ud_header_init ib_ud_header_pack ib_ud_ip4_csum ib_umem_copy_from ib_umem_get ib_umem_odp_map_dma_pages ib_umem_odp_unmap_dma_pages ib_umem_page_count ib_umem_release ib_unregister_device ib_unregister_mad_agent ib_uverbs_get_ucontext ida_alloc_range ida_destroy ida_free idr_alloc idr_alloc_cyclic idr_alloc_u32 idr_destroy idr_find idr_for_each idr_get_next idr_get_next_ul idr_preload idr_remove idr_replace in4_pton in6_pton in_egroup_p __inet6_lookup_established __inet_lookup_established in_group_p init_net __init_rwsem init_srcu_struct init_task init_timer_key init_uts_ns init_wait_entry __init_waitqueue_head interval_tree_insert interval_tree_iter_first interval_tree_iter_next interval_tree_remove int_to_scsilun invalidate_partition iomem_resource iommu_get_domain_for_dev iommu_iova_to_phys iommu_map iommu_unmap __ioremap ioremap_cache io_schedule io_schedule_timeout __iounmap __iowrite32_copy __iowrite64_copy ip6_dst_hoplimit ip_compute_csum ipmi_create_user ipmi_destroy_user ipmi_free_recv_msg ipmi_poll_interface ipmi_request_settime ipmi_set_gets_events ipmi_set_my_address ipmi_smi_msg_received ipmi_unregister_smi ipmi_validate_addr ip_route_output_flow ip_send_check __ipv6_addr_type ipv6_ext_hdr ipv6_find_hdr ipv6_skip_exthdr ipv6_stub irq_cpu_rmap_add irq_create_mapping __irq_domain_add irq_domain_remove irq_find_mapping irq_get_irq_data irq_poll_complete irq_poll_disable irq_poll_enable irq_poll_init irq_poll_sched irq_set_affinity_hint irq_set_affinity_notifier irq_set_chip_and_handler_name irq_to_desc is_acpi_device_node jiffies jiffies_64 jiffies_to_msecs jiffies_to_timespec64 jiffies_to_usecs kallsyms_lookup_name kasprintf kernel_recvmsg kernel_sendmsg kernel_setsockopt kfree kfree_call_rcu kfree_const kfree_skb kgdb_active kgdb_breakpoint kill_fasync kimage_voffset __kmalloc kmalloc_caches __kmalloc_node kmalloc_order_trace kmem_cache_alloc kmem_cache_alloc_node kmem_cache_alloc_node_trace kmem_cache_alloc_trace kmem_cache_create kmem_cache_create_usercopy kmem_cache_destroy kmem_cache_free kmem_cache_shrink kmemdup kobject_add kobject_create_and_add kobject_del kobject_get kobject_init kobject_init_and_add kobject_put kobject_set_name kobject_uevent kobject_uevent_env krealloc kset_create_and_add kset_find_obj kset_register kset_unregister ksize kstrdup kstrdup_const kstrndup kstrtobool kstrtobool_from_user kstrtoint kstrtoint_from_user kstrtoll kstrtoll_from_user kstrtou16 kstrtouint kstrtouint_from_user kstrtoul_from_user kstrtoull kstrtoull_from_user kthread_bind kthread_create_on_node kthread_park kthread_should_stop kthread_stop kthread_unpark ktime_get ktime_get_coarse_real_ts64 ktime_get_raw ktime_get_raw_ts64 ktime_get_real_seconds ktime_get_real_ts64 ktime_get_seconds ktime_get_ts64 ktime_get_with_offset kvasprintf kvfree kvmalloc_node kzfree led_classdev_resume led_classdev_suspend led_classdev_unregister __list_add_valid __list_del_entry_valid llist_add_batch __ll_sc_atomic64_add __ll_sc_atomic64_add_return __ll_sc_atomic64_andnot __ll_sc_atomic64_fetch_add __ll_sc_atomic64_fetch_andnot __ll_sc_atomic64_fetch_andnot_release __ll_sc_atomic64_fetch_or __ll_sc_atomic64_fetch_or_acquire __ll_sc_atomic64_fetch_xor __ll_sc_atomic64_or __ll_sc_atomic64_sub __ll_sc_atomic64_sub_return __ll_sc_atomic_add __ll_sc_atomic_add_return __ll_sc_atomic_add_return_acquire __ll_sc_atomic_sub __ll_sc_atomic_sub_return __ll_sc_atomic_sub_return_release __ll_sc___cmpxchg_case_acq_4 __ll_sc___cmpxchg_case_mb_4 __ll_sc___cmpxchg_case_mb_8 __local_bh_enable_ip __lock_page lock_page_memcg lockref_get lock_sock_nested logic_inw logic_outw make_kgid make_kuid mark_page_accessed match_strdup match_string match_token _mcount mdev_dev mdev_from_dev mdev_get_drvdata mdev_parent_dev mdev_register_device mdev_register_driver mdev_set_drvdata mdev_unregister_device mdev_unregister_driver mdio45_probe mdiobus_alloc_size mdiobus_free mdiobus_get_phy mdiobus_read __mdiobus_register mdiobus_unregister mdiobus_write mdio_mii_ioctl memchr memchr_inv memcmp memcpy __memcpy_fromio __memcpy_toio memdup_user memdup_user_nul memmove memory_read_from_buffer memparse mempool_alloc mempool_alloc_slab mempool_create mempool_create_node mempool_destroy mempool_free mempool_free_slab mempool_kfree mempool_kmalloc memscan mem_section memset __memset_io memstart_addr memzero_explicit metadata_dst_alloc misc_deregister misc_register mlxfw_firmware_flash mmput __mmu_notifier_register mmu_notifier_register mmu_notifier_unregister mmu_notifier_unregister_no_release mod_delayed_work_on mod_timer mod_timer_pending __module_get module_layout module_put module_refcount __msecs_to_jiffies msleep msleep_interruptible __mutex_init mutex_lock mutex_lock_interruptible mutex_lock_killable mutex_trylock mutex_unlock __napi_alloc_skb napi_complete_done napi_consume_skb napi_disable napi_gro_flush napi_gro_receive napi_hash_del __napi_schedule __napi_schedule_irqoff napi_schedule_prep __ndelay ndo_dflt_bridge_getlink ndo_dflt_fdb_add neigh_destroy __neigh_event_send neigh_lookup netdev_alloc_frag __netdev_alloc_skb netdev_bind_sb_channel_queue netdev_crit netdev_err netdev_features_change netdev_info netdev_lower_get_next netdev_master_upper_dev_get netdev_master_upper_dev_get_rcu netdev_notice netdev_printk netdev_reset_tc netdev_rss_key_fill netdev_rx_handler_register netdev_rx_handler_unregister netdev_set_num_tc netdev_set_sb_channel netdev_set_tc_queue netdev_unbind_sb_channel netdev_update_features netdev_walk_all_upper_dev_rcu netdev_warn netif_carrier_off netif_carrier_on netif_device_attach netif_device_detach netif_get_num_default_rss_queues netif_napi_add netif_napi_del netif_receive_skb netif_rx netif_schedule_queue netif_set_real_num_rx_queues netif_set_real_num_tx_queues netif_set_xps_queue netif_tx_stop_all_queues netif_tx_wake_queue netlink_ack netlink_broadcast __netlink_kernel_create netlink_kernel_release netlink_unicast net_ratelimit nla_find nla_parse nla_put __nlmsg_put node_data __node_distance node_states node_to_cpumask_map no_llseek noop_llseek nr_cpu_ids nr_node_ids nsecs_to_jiffies ns_to_timespec ns_to_timespec64 ns_to_timeval numa_node nvme_alloc_request nvme_cancel_request nvme_change_ctrl_state nvme_cleanup_cmd nvme_complete_async_event nvme_complete_rq nvme_disable_ctrl nvme_enable_ctrl nvme_fc_register_localport nvme_fc_register_remoteport nvme_fc_set_remoteport_devloss nvme_fc_unregister_localport nvme_fc_unregister_remoteport nvme_init_ctrl nvme_init_identify nvme_io_timeout nvme_kill_queues nvme_remove_namespaces nvme_reset_ctrl nvme_reset_ctrl_sync nvme_set_queue_count nvme_setup_cmd nvme_shutdown_ctrl nvme_start_ctrl nvme_start_freeze nvme_start_queues nvme_stop_ctrl nvme_stop_queues nvme_submit_sync_cmd nvmet_fc_rcv_fcp_abort nvmet_fc_rcv_fcp_req nvmet_fc_rcv_ls_req nvmet_fc_register_targetport nvmet_fc_unregister_targetport nvme_unfreeze nvme_uninit_ctrl nvme_wait_freeze nvme_wait_freeze_timeout nvme_wq of_led_classdev_register on_each_cpu orderly_poweroff out_of_line_wait_on_bit override_creds __page_file_index __page_frag_cache_drain page_frag_free __page_mapcount page_mapped page_pool_alloc_pages page_pool_create page_pool_destroy __page_pool_put_page pagevec_lookup_range pagevec_lookup_range_tag __pagevec_release panic panic_notifier_list param_array_ops param_get_int param_ops_bool param_ops_byte param_ops_charp param_ops_int param_ops_long param_ops_short param_ops_string param_ops_uint param_ops_ullong param_ops_ulong param_ops_ushort param_set_bool param_set_int pci_alloc_irq_vectors_affinity pci_assign_unassigned_bus_resources pcibios_resource_to_bus pci_bus_read_config_dword pci_bus_resource_n pci_bus_type pci_cfg_access_lock pci_cfg_access_unlock pci_check_and_mask_intx pci_choose_state pci_cleanup_aer_uncorrect_error_status pci_clear_master pci_clear_mwi pci_d3cold_disable pci_dev_driver pci_dev_get pci_device_is_present pci_dev_present pci_dev_put pci_disable_device pci_disable_link_state pci_disable_msi pci_disable_msix pci_disable_pcie_error_reporting pci_disable_rom pci_disable_sriov pcie_bandwidth_available pcie_capability_clear_and_set_word pcie_capability_read_dword pcie_capability_read_word pcie_capability_write_word pcie_flr pcie_get_speed_cap pcie_get_width_cap pci_enable_atomic_ops_to_root pci_enable_device pci_enable_device_mem pci_enable_msi pci_enable_msix_range pci_enable_pcie_error_reporting pci_enable_rom pci_enable_sriov pci_enable_wake pcie_print_link_status pcie_relaxed_ordering_enabled pcie_set_readrq pci_find_capability pci_find_ext_capability pci_free_irq pci_free_irq_vectors pci_get_class pci_get_device pci_get_domain_bus_and_slot pci_get_slot pci_ignore_hotplug pci_intx pci_iomap pci_ioremap_bar pci_irq_get_affinity pci_irq_vector pci_map_rom pci_match_id pcim_enable_device pcim_iomap pcim_iomap_regions pcim_iomap_table pcim_iounmap pci_msi_mask_irq pci_msi_unmask_irq pci_num_vf pci_platform_rom pci_prepare_to_sleep pci_read_config_byte pci_read_config_dword pci_read_config_word pci_read_vpd __pci_register_driver pci_release_regions pci_release_resource pci_release_selected_regions pci_request_irq pci_request_regions pci_request_selected_regions pci_rescan_bus pci_resize_resource pci_restore_state pci_save_state pci_select_bars pci_set_master pci_set_mwi pci_set_power_state pci_sriov_configure_simple pci_sriov_get_totalvfs pci_sriov_set_totalvfs pci_stop_and_remove_bus_device pci_stop_and_remove_bus_device_locked pci_try_set_mwi pci_unmap_rom pci_unregister_driver pci_vfs_assigned pci_vpd_find_info_keyword pci_vpd_find_tag pci_wait_for_pending_transaction pci_wake_from_d3 pci_write_config_byte pci_write_config_dword pci_write_config_word pcix_set_mmrbc PDE_DATA __per_cpu_offset perf_trace_buf_alloc perf_trace_run_bpf_submit pfn_valid phy_attached_info phy_connect phy_connect_direct phy_disconnect phy_ethtool_ksettings_get phy_ethtool_ksettings_set phy_loopback phy_mii_ioctl phy_resume phy_start phy_start_aneg phy_stop phy_suspend pid_task pid_vnr pm_power_off pm_runtime_allow __pm_runtime_disable pm_runtime_enable pm_runtime_forbid __pm_runtime_idle __pm_runtime_resume pm_runtime_set_autosuspend_delay __pm_runtime_set_status __pm_runtime_suspend __pm_runtime_use_autosuspend pm_schedule_suspend power_supply_is_system_supplied prandom_bytes prandom_u32 prepare_creds prepare_to_wait prepare_to_wait_event prepare_to_wait_exclusive print_hex_dump printk __printk_ratelimit print_stack_trace proc_create_data proc_mkdir __pskb_copy_fclone pskb_expand_head __pskb_pull_tail ___pskb_trim ptp_clock_event ptp_clock_index ptp_clock_register ptp_clock_unregister ptp_find_pin __put_cred put_device put_disk __put_page __put_task_struct put_unused_fd qed_get_eth_ops qed_put_eth_ops queue_delayed_work_on queued_read_lock_slowpath queued_spin_lock_slowpath queued_write_lock_slowpath queue_work_on radix_tree_delete radix_tree_gang_lookup __radix_tree_insert radix_tree_lookup radix_tree_tagged raid_class_attach raid_class_release ___ratelimit raw_notifier_call_chain raw_notifier_chain_register raw_notifier_chain_unregister rb_erase __rb_erase_color rb_first rb_first_postorder __rb_insert_augmented rb_insert_color rb_next rb_next_postorder rb_replace_node rbt_ib_umem_for_each_in_range rbt_ib_umem_lookup rcu_barrier rdma_create_ah rdma_destroy_ah rdma_is_zero_gid rdma_port_get_link_layer rdma_query_ah rdma_query_gid rdma_restrack_get rdma_restrack_put rdma_roce_rescan_device read_cache_pages recalc_sigpending refcount_dec_and_mutex_lock refcount_dec_and_test_checked refcount_dec_checked refcount_inc_checked refcount_inc_not_zero_checked register_acpi_notifier register_blkdev __register_chrdev register_chrdev_region register_fib_notifier register_inet6addr_notifier register_inetaddr_notifier register_netdev register_netdevice_notifier register_netevent_notifier register_reboot_notifier release_firmware release_pages __release_region release_sock remap_pfn_range remove_conflicting_framebuffers remove_proc_entry remove_wait_queue request_firmware request_firmware_direct request_firmware_nowait __request_module __request_region request_threaded_irq reservation_object_add_excl_fence reservation_object_add_shared_fence reservation_object_get_fences_rcu reservation_object_reserve_shared reservation_object_wait_timeout_rcu reservation_ww_class reset_devices revert_creds rhashtable_destroy rhashtable_free_and_destroy rhashtable_init rhashtable_insert_slow rhashtable_walk_enter rhashtable_walk_exit rhashtable_walk_next rhashtable_walk_start_check rhashtable_walk_stop rhltable_init rht_bucket_nested rht_bucket_nested_insert round_jiffies round_jiffies_relative rps_may_expire_flow rtc_time64_to_tm rtnl_is_locked rtnl_lock rtnl_trylock rtnl_unlock sas_attach_transport sas_disable_tlr sas_enable_tlr sas_end_device_alloc sas_expander_alloc sas_is_tlr_enabled sas_phy_add sas_phy_alloc sas_phy_free sas_port_add sas_port_add_phy sas_port_alloc_num sas_port_delete sas_port_delete_phy sas_port_free sas_read_port_mode_page sas_release_transport sas_remove_host sas_rphy_add save_stack_trace save_stack_trace_tsk sbitmap_queue_clear __sbitmap_queue_get sched_clock sched_setscheduler schedule schedule_hrtimeout schedule_hrtimeout_range schedule_timeout schedule_timeout_interruptible schedule_timeout_uninterruptible scmd_printk scnprintf scsi_add_device scsi_add_host_with_dma scsi_block_requests scsi_build_sense_buffer scsi_change_queue_depth scsi_device_get scsi_device_lookup scsi_device_put scsi_device_set_state scsi_device_type scsi_dma_map scsi_dma_unmap __scsi_execute scsi_get_vpd_page scsi_host_alloc scsi_host_busy scsi_host_get scsi_host_lookup scsi_host_put scsi_internal_device_block_nowait scsi_internal_device_unblock_nowait scsi_is_fc_rport scsi_is_host_device scsi_is_sdev_device __scsi_iterate_devices scsilun_to_int scsi_normalize_sense scsi_print_command scsi_register_driver scsi_remove_device scsi_remove_host scsi_remove_target scsi_sanitize_inquiry_string scsi_scan_host scsi_unblock_requests sdev_prefix_printk secpath_dup security_d_instantiate send_sig seq_lseek seq_open seq_printf seq_putc seq_puts seq_read seq_release seq_write set_cpus_allowed_ptr set_current_groups set_device_ro set_disk_ro set_freezable set_normalized_timespec64 set_page_dirty set_page_dirty_lock set_user_nice sg_alloc_table_from_pages sg_copy_from_buffer sg_copy_to_buffer sg_free_table sg_init_table sg_miter_next sg_miter_start sg_miter_stop sg_next sigprocmask si_meminfo simple_attr_open simple_attr_read simple_attr_release simple_attr_write simple_open simple_read_from_buffer simple_strtol simple_strtoul simple_strtoull simple_write_to_buffer single_open single_release skb_add_rx_frag skb_checksum skb_checksum_help skb_clone skb_clone_tx_timestamp skb_copy skb_copy_bits skb_copy_expand skb_dequeue __skb_flow_dissect __skb_get_hash __skb_gso_segment skb_gso_validate_mac_len __skb_pad skb_pull skb_push skb_put skb_queue_purge skb_queue_tail skb_realloc_headroom skb_store_bits skb_trim skb_tstamp_tx skb_vlan_pop sk_free smp_call_function_many smp_call_function_single snprintf sock_create_kern sock_edemux sock_queue_err_skb sock_release softnet_data sort sprintf srcu_barrier __srcu_read_lock __srcu_read_unlock sscanf __stack_chk_fail __stack_chk_guard starget_for_each_device strcasecmp strcat strchr strcmp strcpy strcspn strim strlcat strlcpy strlen strncasecmp strncat strncmp strncpy strncpy_from_user strnlen strnstr strpbrk strrchr strsep strspn strstr submit_bio __sw_hweight32 __sw_hweight64 __sw_hweight8 swiotlb_nr_tbl switchdev_port_same_parent_id __symbol_put sync_file_create synchronize_irq synchronize_net synchronize_sched synchronize_srcu sysfs_add_file_to_group sysfs_create_bin_file sysfs_create_file_ns sysfs_create_group sysfs_remove_bin_file sysfs_remove_file_from_group sysfs_remove_file_ns sysfs_remove_group sysfs_streq system_state system_unbound_wq system_wq sys_tz task_active_pid_ns tasklet_init tasklet_kill __tasklet_schedule __task_pid_nr_ns tcf_block_cb_register tcf_block_cb_unregister tcp_gro_complete tcp_hashinfo tc_setup_cb_egdev_register tc_setup_cb_egdev_unregister time64_to_tm timecounter_cyc2time timecounter_init timecounter_read tls_get_record tls_validate_xmit_skb to_drm_sched_fence trace_define_field trace_event_buffer_commit trace_event_buffer_reserve trace_event_ignore_this_pid trace_event_raw_init trace_event_reg trace_handle_return __tracepoint_dma_fence_emit __tracepoint_xdp_exception trace_print_array_seq trace_print_flags_seq trace_raw_output_prep trace_seq_printf trace_seq_putc try_module_get try_wait_for_completion ttm_bo_add_to_lru ttm_bo_clean_mm ttm_bo_del_sub_from_lru ttm_bo_device_init ttm_bo_device_release ttm_bo_dma_acc_size ttm_bo_eviction_valuable ttm_bo_evict_mm ttm_bo_global_init ttm_bo_global_release ttm_bo_init ttm_bo_init_mm ttm_bo_init_reserved ttm_bo_kmap ttm_bo_kunmap ttm_bo_lock_delayed_workqueue ttm_bo_manager_func ttm_bo_mem_put ttm_bo_mem_space ttm_bo_mmap ttm_bo_move_accel_cleanup ttm_bo_move_memcpy ttm_bo_move_to_lru_tail ttm_bo_move_ttm ttm_bo_pipeline_move ttm_bo_put ttm_bo_unlock_delayed_workqueue ttm_bo_validate ttm_bo_wait ttm_dma_page_alloc_debugfs ttm_dma_populate ttm_dma_tt_fini ttm_dma_tt_init ttm_dma_unpopulate ttm_eu_backoff_reservation ttm_eu_fence_buffer_objects ttm_eu_reserve_buffers ttm_fbdev_mmap ttm_mem_global_init ttm_mem_global_release ttm_page_alloc_debugfs ttm_pool_populate ttm_pool_unpopulate ttm_populate_and_map_pages ttm_sg_tt_init ttm_tt_bind ttm_tt_set_placement_caching ttm_unmap_and_unpopulate_pages __udelay udp4_hwcsum uio_event_notify __uio_register_device uio_unregister_device unlock_page unlock_page_memcg unmap_mapping_range unregister_acpi_notifier unregister_blkdev __unregister_chrdev unregister_chrdev_region unregister_fib_notifier unregister_inet6addr_notifier unregister_inetaddr_notifier unregister_netdev unregister_netdevice_notifier unregister_netevent_notifier unregister_reboot_notifier unuse_mm up up_read up_write __usecs_to_jiffies use_mm usleep_range _uverbs_alloc uverbs_copy_to uverbs_destroy_def_handler uverbs_fd_class uverbs_get_flags32 uverbs_get_flags64 uverbs_idr_class vfree vga_client_register vlan_dev_real_dev vlan_dev_vlan_id vlan_dev_vlan_proto vmalloc __vmalloc vmalloc_node vmalloc_to_page vmap vm_insert_page vm_mmap vm_munmap vprintk vscnprintf vsnprintf vsprintf vunmap vzalloc vzalloc_node wait_for_completion wait_for_completion_interruptible wait_for_completion_interruptible_timeout wait_for_completion_io_timeout wait_for_completion_killable wait_for_completion_timeout wait_on_page_bit __wake_up wake_up_bit __wake_up_locked wake_up_process __warn_printk work_busy write_cache_pages ww_mutex_lock ww_mutex_lock_interruptible ww_mutex_unlock xdp_do_flush_map xdp_do_redirect xdp_return_frame xdp_return_frame_rx_napi xdp_rxq_info_is_reg xdp_rxq_info_reg xdp_rxq_info_reg_mem_model xdp_rxq_info_unreg xdp_rxq_info_unused xfrm_replay_seqhi xz_dec_end xz_dec_init xz_dec_run yield zap_vma_ptes zlib_inflate zlib_inflateEnd zlib_inflateInit2 zlib_inflate_workspacesize 附2：x86 平台 KABI 白名单列表初稿（2228个） acpi_bus_get_device acpi_bus_register_driver acpi_bus_unregister_driver acpi_check_dsm acpi_dev_found acpi_disabled acpi_dma_configure acpi_evaluate_dsm acpi_evaluate_integer acpi_evaluate_object acpi_format_exception acpi_gbl_FADT acpi_get_devices acpi_get_handle acpi_get_name acpi_get_table acpi_gsi_to_irq acpi_handle_printk acpi_has_method acpi_install_notify_handler acpi_lid_open acpi_os_map_memory acpi_os_unmap_memory acpi_register_gsi acpi_remove_notify_handler acpi_unregister_gsi acpi_video_get_edid acpi_walk_namespace address_space_init_once add_timer add_wait_queue add_wait_queue_exclusive admin_timeout alloc_chrdev_region alloc_cpumask_var alloc_cpu_rmap __alloc_disk_node alloc_etherdev_mqs alloc_netdev_mqs alloc_pages_current __alloc_pages_nodemask __alloc_percpu __alloc_skb __alloc_workqueue_key anon_inode_getfile apic arch_dma_alloc_attrs arch_io_free_memtype_wc arch_io_reserve_memtype_wc arch_phys_wc_add arch_phys_wc_del arch_wb_cache_pmem arp_tbl async_schedule _atomic_dec_and_lock atomic_notifier_call_chain atomic_notifier_chain_register atomic_notifier_chain_unregister attribute_container_find_class_device autoremove_wake_function backlight_device_register backlight_device_unregister backlight_force_update bdevname bdev_read_only bdget_disk _bin2bcd bio_add_page bio_alloc_bioset bio_clone_fast bio_endio bio_free_pages bio_init bio_put bioset_exit bioset_init __bitmap_and __bitmap_andnot __bitmap_clear __bitmap_complement __bitmap_equal bitmap_find_free_region bitmap_find_next_zero_area_off __bitmap_intersects __bitmap_or __bitmap_parse bitmap_parselist bitmap_print_to_pagebuf bitmap_release_region __bitmap_set __bitmap_shift_left __bitmap_shift_right __bitmap_subset __bitmap_weight __bitmap_xor bitmap_zalloc bit_wait blk_alloc_queue blk_check_plugged blk_cleanup_queue blkdev_get_by_path blkdev_issue_discard blkdev_issue_write_same blkdev_issue_zeroout blkdev_put blk_execute_rq blk_execute_rq_nowait blk_finish_plug blk_get_queue blk_get_request blk_init_tags blk_mq_alloc_tag_set blk_mq_complete_request blk_mq_end_request blk_mq_free_request blk_mq_free_tag_set blk_mq_init_queue blk_mq_map_queues blk_mq_pci_map_queues blk_mq_quiesce_queue blk_mq_run_hw_queues blk_mq_start_request blk_mq_tagset_busy_iter blk_mq_tag_to_rq blk_mq_unique_tag blk_mq_unquiesce_queue blk_mq_update_nr_hw_queues blk_put_queue blk_put_request blk_queue_bounce_limit blk_queue_dma_alignment blk_queue_flag_clear blk_queue_flag_set blk_queue_free_tags blk_queue_init_tags blk_queue_io_min blk_queue_io_opt blk_queue_logical_block_size blk_queue_make_request blk_queue_max_discard_sectors blk_queue_max_hw_sectors blk_queue_max_segments blk_queue_max_segment_size blk_queue_max_write_same_sectors blk_queue_physical_block_size blk_queue_rq_timeout blk_queue_segment_boundary blk_queue_split blk_queue_stack_limits blk_queue_update_dma_alignment blk_queue_virt_boundary blk_queue_write_cache blk_rq_append_bio blk_rq_count_integrity_sg blk_rq_map_integrity_sg blk_rq_map_kern blk_rq_map_sg blk_rq_map_user_iov blk_rq_unmap_user blk_set_stacking_limits blk_start_plug blk_status_to_errno blk_verify_command blocking_notifier_call_chain blocking_notifier_chain_register blocking_notifier_chain_unregister boot_cpu_data bpf_prog_add bpf_prog_inc bpf_prog_put bpf_trace_run1 bpf_trace_run2 bpf_trace_run3 bpf_trace_run5 bpf_warn_invalid_xdp_action bsg_job_done btree_destroy btree_geo32 btree_geo64 btree_get_prev btree_init btree_insert btree_last btree_lookup btree_remove btree_update build_skb bus_find_device_by_name __cachemode2pte_tbl call_netdevice_notifiers call_rcu_sched call_usermodehelper cancel_delayed_work cancel_delayed_work_sync cancel_work_sync capable cdev_add cdev_del cdev_device_add cdev_device_del cdev_init __chash_table_copy_in __chash_table_copy_out __check_object_size __class_create class_create_file_ns class_destroy __class_register class_remove_file_ns class_unregister _cleanup_srcu_struct clear_user clk_get_rate cm_class cnic_register_driver cnic_unregister_driver commit_creds compat_alloc_user_space complete complete_all complete_and_exit completion_done component_add component_del _cond_resched configfs_register_subsystem configfs_remove_default_groups configfs_unregister_subsystem config_group_init config_group_init_type_name config_item_put console_lock console_unlock __const_udelay consume_skb _copy_from_user _copy_to_user copy_user_enhanced_fast_string copy_user_generic_string copy_user_generic_unrolled cpu_bit_bitmap cpu_core_map cpufreq_get cpufreq_quick_get __cpuhp_remove_state __cpuhp_setup_state cpu_info cpu_khz cpumask_local_spread cpumask_next cpumask_next_and cpu_number __cpu_online_mask __cpu_possible_mask __cpu_present_mask cpu_sibling_map cpus_read_lock cpus_read_unlock crc32c crc32_le crc8 crc8_populate_msb crc_t10dif crypto_ahash_digest crypto_ahash_final crypto_ahash_setkey crypto_alloc_ahash crypto_destroy_tfm crypto_register_shash crypto_unregister_shash csum_ipv6_magic csum_partial _ctype current_task dca3_get_tag dca_add_requester dca_register_notify dca_remove_requester dca_unregister_notify dcb_getapp dcb_ieee_delapp dcb_ieee_getapp_mask dcb_ieee_setapp dcbnl_cee_notify dcbnl_ieee_notify dcb_setapp debugfs_create_atomic_t debugfs_create_dir debugfs_create_file debugfs_create_u32 debugfs_create_u64 debugfs_create_u8 debugfs_lookup debugfs_remove __default_kernel_pte_mask default_llseek default_wake_function __delay delayed_work_timer_fn del_gendisk del_timer del_timer_sync destroy_workqueue dev_add_pack dev_addr_add dev_addr_del dev_base_lock dev_close dev_driver_string _dev_err __dev_get_by_index dev_get_by_index dev_get_by_index_rcu dev_get_by_name dev_get_iflink dev_get_stats device_add_disk device_create device_create_file device_destroy device_initialize device_link_add device_release_driver device_remove_file device_reprobe device_set_wakeup_capable device_set_wakeup_enable _dev_info __dev_kfree_skb_any __dev_kfree_skb_irq devlink_alloc devlink_free devlink_param_driverinit_value_get devlink_param_driverinit_value_set devlink_params_register devlink_params_unregister devlink_param_value_changed devlink_port_attrs_set devlink_port_register devlink_port_type_clear devlink_port_type_ib_set devlink_port_unregister devlink_region_create devlink_region_destroy devlink_region_shapshot_id_get devlink_region_snapshot_create devlink_register devlink_unregister devmap_managed_key dev_mc_add dev_mc_add_excl dev_mc_del devm_free_irq devm_hwmon_device_register_with_groups devm_ioremap devm_iounmap devm_kfree devm_kmalloc devm_kmemdup devm_request_threaded_irq _dev_notice dev_printk dev_queue_xmit __dev_remove_pack dev_remove_pack dev_set_mac_address dev_set_mtu dev_set_name dev_set_promiscuity dev_trans_start dev_uc_add dev_uc_add_excl dev_uc_del _dev_warn disable_irq disable_irq_nosync dma_fence_add_callback dma_fence_array_create dma_fence_context_alloc dma_fence_free dma_fence_get_status dma_fence_init dma_fence_release dma_fence_signal dma_fence_signal_locked dma_fence_wait_any_timeout dma_fence_wait_timeout dma_get_required_mask dmam_alloc_coherent dmam_free_coherent dma_ops dma_pool_alloc dma_pool_create dma_pool_destroy dma_pool_free dmi_check_system dmi_get_system_info dmi_match __do_once_done __do_once_start do_wait_intr down downgrade_write down_interruptible down_read down_read_trylock down_timeout down_trylock down_write down_write_killable down_write_trylock dput dql_completed dql_reset drain_workqueue driver_create_file driver_for_each_device driver_remove_file drm_add_edid_modes drm_add_modes_noedid drm_atomic_add_affected_connectors drm_atomic_add_affected_planes drm_atomic_commit drm_atomic_get_connector_state drm_atomic_get_crtc_state drm_atomic_get_plane_state drm_atomic_helper_check drm_atomic_helper_check_modeset drm_atomic_helper_check_planes drm_atomic_helper_check_plane_state drm_atomic_helper_cleanup_planes drm_atomic_helper_commit drm_atomic_helper_commit_cleanup_done drm_atomic_helper_commit_hw_done __drm_atomic_helper_connector_destroy_state drm_atomic_helper_connector_destroy_state __drm_atomic_helper_connector_duplicate_state __drm_atomic_helper_connector_reset __drm_atomic_helper_crtc_destroy_state __drm_atomic_helper_crtc_duplicate_state drm_atomic_helper_disable_plane drm_atomic_helper_legacy_gamma_set drm_atomic_helper_page_flip __drm_atomic_helper_plane_destroy_state drm_atomic_helper_plane_destroy_state __drm_atomic_helper_plane_duplicate_state drm_atomic_helper_prepare_planes drm_atomic_helper_resume drm_atomic_helper_set_config drm_atomic_helper_setup_commit drm_atomic_helper_shutdown drm_atomic_helper_suspend drm_atomic_helper_swap_state drm_atomic_helper_update_legacy_modeset_state drm_atomic_helper_update_plane drm_atomic_helper_wait_for_dependencies drm_atomic_helper_wait_for_fences drm_atomic_helper_wait_for_flip_done drm_atomic_state_alloc drm_atomic_state_default_clear drm_atomic_state_default_release __drm_atomic_state_free drm_atomic_state_init drm_calc_vbltimestamp_from_scanoutpos drm_color_lut_extract drm_compat_ioctl drm_connector_attach_encoder drm_connector_cleanup drm_connector_init drm_connector_list_iter_begin drm_connector_list_iter_end drm_connector_list_iter_next drm_connector_register drm_connector_set_path_property drm_connector_unregister drm_connector_update_edid_property drm_crtc_accurate_vblank_count drm_crtc_add_crc_entry drm_crtc_arm_vblank_event drm_crtc_cleanup __drm_crtc_commit_free drm_crtc_enable_color_mgmt drm_crtc_force_disable_all drm_crtc_from_index drm_crtc_handle_vblank drm_crtc_helper_set_config drm_crtc_helper_set_mode drm_crtc_init drm_crtc_init_with_planes drm_crtc_send_vblank_event drm_crtc_vblank_count drm_crtc_vblank_get drm_crtc_vblank_off drm_crtc_vblank_on drm_crtc_vblank_put drm_cvt_mode drm_dbg drm_debug drm_debugfs_create_files drm_detect_hdmi_monitor drm_detect_monitor_audio drm_dev_alloc drm_dev_put drm_dev_register drm_dev_unref drm_dev_unregister drm_dp_atomic_find_vcpi_slots drm_dp_atomic_release_vcpi_slots drm_dp_aux_register drm_dp_aux_unregister drm_dp_bw_code_to_link_rate drm_dp_calc_pbn_mode drm_dp_channel_eq_ok drm_dp_check_act_status drm_dp_clock_recovery_ok drm_dp_dpcd_read drm_dp_dpcd_read_link_status drm_dp_dpcd_write drm_dp_find_vcpi_slots drm_dp_get_adjust_request_pre_emphasis drm_dp_get_adjust_request_voltage drm_dp_link_rate_to_bw_code drm_dp_link_train_channel_eq_delay drm_dp_link_train_clock_recovery_delay drm_dp_mst_allocate_vcpi drm_dp_mst_deallocate_vcpi drm_dp_mst_detect_port drm_dp_mst_get_edid drm_dp_mst_hpd_irq drm_dp_mst_reset_vcpi_slots drm_dp_mst_topology_mgr_destroy drm_dp_mst_topology_mgr_init drm_dp_mst_topology_mgr_resume drm_dp_mst_topology_mgr_set_mst drm_dp_mst_topology_mgr_suspend drm_dp_update_payload_part1 drm_dp_update_payload_part2 drm_edid_header_is_valid drm_edid_is_valid drm_edid_to_sad drm_edid_to_speaker_allocation drm_encoder_cleanup drm_encoder_init drm_err drm_fb_helper_add_one_connector drm_fb_helper_alloc_fbi drm_fb_helper_blank drm_fb_helper_cfb_copyarea drm_fb_helper_cfb_fillrect drm_fb_helper_cfb_imageblit drm_fb_helper_check_var drm_fb_helper_debug_enter drm_fb_helper_debug_leave drm_fb_helper_fill_fix drm_fb_helper_fill_var drm_fb_helper_fini drm_fb_helper_hotplug_event drm_fb_helper_init drm_fb_helper_initial_config drm_fb_helper_ioctl drm_fb_helper_lastclose drm_fb_helper_output_poll_changed drm_fb_helper_pan_display drm_fb_helper_prepare drm_fb_helper_remove_one_connector drm_fb_helper_setcmap drm_fb_helper_set_par drm_fb_helper_set_suspend drm_fb_helper_set_suspend_unlocked drm_fb_helper_single_add_all_connectors drm_fb_helper_unregister_fbi drm_format_plane_cpp drm_framebuffer_cleanup drm_framebuffer_init drm_framebuffer_unregister_private drm_gem_dmabuf_kmap drm_gem_dmabuf_kunmap drm_gem_dmabuf_mmap drm_gem_dmabuf_release drm_gem_dmabuf_vmap drm_gem_dmabuf_vunmap drm_gem_fb_create_handle drm_gem_fb_destroy drm_gem_handle_create drm_gem_handle_delete drm_gem_map_attach drm_gem_map_detach drm_gem_map_dma_buf drm_gem_object_free drm_gem_object_init drm_gem_object_lookup drm_gem_object_put_unlocked drm_gem_object_release drm_gem_prime_export drm_gem_prime_fd_to_handle drm_gem_prime_handle_to_fd drm_gem_prime_import drm_gem_private_object_init drm_gem_unmap_dma_buf drm_get_edid drm_get_edid_switcheroo drm_get_format_name drm_get_max_iomem drm_global_item_ref drm_global_item_unref drm_handle_vblank drm_hdmi_avi_infoframe_from_display_mode drm_hdmi_vendor_infoframe_from_display_mode drm_helper_connector_dpms drm_helper_disable_unused_functions drm_helper_hpd_irq_event drm_helper_mode_fill_fb_struct drm_helper_probe_single_connector_modes drm_helper_resume_force_mode drm_i2c_encoder_detect drm_i2c_encoder_init drm_i2c_encoder_mode_fixup drm_i2c_encoder_restore drm_i2c_encoder_save drm_invalid_op drm_ioctl drm_irq_install drm_irq_uninstall drm_is_current_master drm_kms_helper_hotplug_event drm_kms_helper_is_poll_worker drm_kms_helper_poll_disable drm_kms_helper_poll_enable drm_kms_helper_poll_fini drm_kms_helper_poll_init drm_match_cea_mode drm_mm_init drm_mm_insert_node_in_range drm_mm_print drm_mm_remove_node drm_mm_takedown drm_mode_config_cleanup drm_mode_config_init drm_mode_config_reset drm_mode_copy drm_mode_create_dvi_i_properties drm_mode_create_scaling_mode_property drm_mode_create_tv_properties drm_mode_crtc_set_gamma_size drm_mode_debug_printmodeline drm_mode_destroy drm_mode_duplicate drm_mode_equal drm_mode_get_hv_timing drm_mode_is_420_only drm_mode_legacy_fb_format drm_mode_object_find drm_mode_object_put drm_mode_probed_add drm_mode_set_crtcinfo drm_modeset_lock drm_modeset_lock_all drm_modeset_lock_all_ctx drm_mode_set_name drm_modeset_unlock drm_modeset_unlock_all drm_mode_vrefresh drm_object_attach_property drm_object_property_set_value drm_open drm_plane_cleanup drm_plane_create_alpha_property drm_plane_create_color_properties drm_plane_create_zpos_immutable_property drm_plane_create_zpos_property drm_plane_force_disable drm_plane_init drm_poll drm_primary_helper_destroy drm_primary_helper_funcs drm_prime_gem_destroy drm_prime_pages_to_sg drm_prime_sg_to_page_addr_arrays drm_printf __drm_printfn_seq_file drm_property_add_enum drm_property_create drm_property_create_enum drm_property_create_range __drm_puts_seq_file drm_read drm_release drm_scdc_read drm_scdc_write drm_sched_dependency_optimized drm_sched_entity_destroy drm_sched_entity_fini drm_sched_entity_flush drm_sched_entity_init drm_sched_entity_push_job drm_sched_entity_set_rq drm_sched_fini drm_sched_hw_job_reset drm_sched_init drm_sched_job_init drm_sched_job_recovery drm_send_event_locked drm_syncobj_create drm_syncobj_find drm_syncobj_find_fence drm_syncobj_free drm_syncobj_get_fd drm_syncobj_get_handle drm_syncobj_replace_fence drm_universal_plane_init drm_vblank_init drm_vma_node_allow drm_vma_node_is_allowed drm_vma_node_revoke dst_release dump_stack __dynamic_dev_dbg __dynamic_netdev_dbg __dynamic_pr_debug efi elfcorehdr_addr emergency_restart empty_zero_page enable_irq errno_to_blk_status ether_setup eth_get_headlen eth_platform_get_mac_address ethtool_convert_legacy_u32_to_link_mode ethtool_convert_link_mode_to_legacy_u32 __ethtool_get_link_ksettings ethtool_intersect_link_masks ethtool_op_get_link ethtool_op_get_ts_info eth_type_trans eth_validate_addr event_triggers_call ex_handler_default ex_handler_refcount fasync_helper fc_attach_transport fc_block_scsi_eh fc_disc_config fc_disc_init fc_eh_host_reset fc_eh_timed_out fc_elsct_init fc_elsct_send fc_exch_init fc_exch_mgr_alloc fc_exch_mgr_free fc_exch_mgr_list_clone fc_exch_recv fc_fabric_login fc_fabric_logoff _fc_frame_alloc fc_frame_alloc_fill fc_get_event_number fc_get_host_port_state fc_get_host_speed fc_get_host_stats fc_host_post_event fc_host_post_vendor_event fc_lport_bsg_request fc_lport_config fc_lport_destroy fc_lport_flogi_resp fc_lport_init fc_lport_logo_resp fc_lport_reset fcoe_check_wait_queue fcoe_clean_pending_queue fcoe_ctlr_destroy fcoe_ctlr_device_add fcoe_ctlr_device_delete fcoe_ctlr_els_send fcoe_ctlr_get_lesb fcoe_ctlr_init fcoe_ctlr_link_down fcoe_ctlr_link_up fcoe_ctlr_recv fcoe_ctlr_recv_flogi fcoe_fc_crc fcoe_fcf_get_selected fcoe_get_lesb fcoe_get_paged_crc_eof fcoe_get_wwn fcoe_link_speed_update fcoe_queue_timer fcoe_start_io fcoe_transport_attach fcoe_transport_detach fcoe_validate_vport_create fcoe_wwn_from_mac fcoe_wwn_to_str fc_release_transport fc_remote_port_add fc_remote_port_delete fc_remote_port_rolechg fc_remove_host fc_rport_terminate_io fc_set_mfs fc_set_rport_loss_tmo fc_slave_alloc fc_vport_create fc_vport_id_lookup fc_vport_setlink fc_vport_terminate __fdget fd_install __fentry__ fget __fib_lookup fib_table_lookup filemap_fault filp_close filp_open find_first_bit find_first_zero_bit find_get_pid find_last_bit find_next_bit find_next_zero_bit find_pid_ns find_vma finish_wait firmware_request_nowarn fixed_size_llseek flow_keys_dissector flush_delayed_work flush_signals flush_work flush_workqueue follow_pfn force_sig fortify_panic fput free_cpumask_var free_fib_info free_irq free_irq_cpu_rmap free_netdev __free_pages free_pages free_percpu from_kgid from_kuid fs_bio_set gcd generate_random_uuid generic_end_io_acct generic_handle_irq generic_make_request generic_start_io_acct genlmsg_put genl_register_family genl_unregister_family get_device __get_free_pages get_gendisk get_pid_task get_random_bytes __get_task_comm get_task_mm get_task_pid get_unused_fd_flags __get_user_2 __get_user_4 __get_user_8 get_user_pages get_user_pages_remote get_zeroed_page groups_alloc groups_free handle_simple_irq hdmi_avi_infoframe_pack hdmi_infoframe_pack hrtimer_cancel hrtimer_forward hrtimer_init hrtimer_start_range_ns hrtimer_try_to_cancel __hw_addr_sync_dev __hw_addr_unsync_dev hwmon_device_register hwmon_device_register_with_groups hwmon_device_register_with_info hwmon_device_unregister i2c_add_adapter i2c_bit_add_bus i2c_bit_algo i2c_del_adapter i2c_new_device i2c_smbus_read_byte_data i2c_smbus_write_byte_data i2c_transfer i2c_unregister_device __ib_alloc_cq ib_alloc_device ib_alloc_odp_umem __ib_alloc_pd ib_attach_mcast ib_cache_gid_parse_type_str ib_cache_gid_type_str ib_cancel_mad ib_cm_init_qp_attr ib_cm_insert_listen ib_cm_listen ib_cm_notify ibcm_reject_msg ib_copy_path_rec_from_user ib_copy_path_rec_to_user ib_copy_qp_attr_to_user ib_create_ah_from_wc ib_create_cm_id __ib_create_cq ib_create_qp ib_create_qp_security ib_create_send_mad ib_dealloc_device ib_dealloc_pd ib_dealloc_xrcd ib_destroy_cm_id ib_destroy_cq ib_destroy_qp ib_destroy_rwq_ind_table ib_destroy_wq ib_detach_mcast ib_dispatch_event ib_find_cached_pkey ib_free_cq ib_free_recv_mad ib_free_send_mad ib_get_cached_pkey ib_get_cached_port_state ib_get_eth_speed ib_get_gids_from_rdma_hdr ib_get_mad_data_offset ib_get_net_dev_by_params ib_get_rdma_header_version ib_get_rmpp_segment ib_init_ah_attr_from_path ib_init_ah_attr_from_wc ib_init_ah_from_mcmember ib_is_mad_class_rmpp ib_mad_kernel_rmpp_agent ib_modify_mad ib_modify_port ib_modify_qp ib_modify_qp_is_ok ib_modify_qp_with_udata ibnl_put_attr ibnl_put_msg ib_open_qp ib_post_send_mad ib_process_cq_direct ib_query_pkey ib_query_port ib_query_qp ib_query_srq ib_rdmacg_try_charge ib_rdmacg_uncharge ib_register_client ib_register_device ib_register_event_handler ib_register_mad_agent ib_response_mad ib_sa_cancel_query ib_sa_free_multicast ib_sa_get_mcmember_rec ib_sa_guid_info_rec_query ib_sa_join_multicast ib_sa_path_rec_get ib_sa_register_client ib_sa_sendonly_fullmem_support ib_sa_unregister_client ib_send_cm_apr ib_send_cm_drep ib_send_cm_dreq ib_send_cm_lap ib_send_cm_mra ib_send_cm_rej ib_send_cm_rep ib_send_cm_req ib_send_cm_rtu ib_send_cm_sidr_rep ib_send_cm_sidr_req ib_set_client_data ib_sg_to_pages ib_ud_header_init ib_ud_header_pack ib_ud_ip4_csum ib_umem_copy_from ib_umem_get ib_umem_odp_map_dma_pages ib_umem_odp_unmap_dma_pages ib_umem_page_count ib_umem_release ib_unregister_client ib_unregister_device ib_unregister_event_handler ib_unregister_mad_agent ib_uverbs_get_ucontext ib_wc_status_msg ida_alloc_range ida_destroy ida_free idr_alloc idr_alloc_cyclic idr_alloc_u32 idr_destroy idr_find idr_for_each idr_get_next idr_get_next_ul idr_preload idr_remove idr_replace igrab in6_dev_finish_destroy in_dev_finish_destroy in_egroup_p __inet6_lookup_established inet_get_local_port_range __inet_lookup_established in_group_p init_net __init_rwsem init_srcu_struct init_task init_timer_key init_uts_ns init_wait_entry __init_waitqueue_head interval_tree_insert interval_tree_iter_first interval_tree_iter_next interval_tree_remove int_to_scsilun invalidate_partition iomem_resource iommu_get_domain_for_dev iommu_iova_to_phys iommu_map iommu_unmap ioread16 ioread16be ioread32 ioread32be ioread8 ioremap_cache ioremap_nocache ioremap_wc io_schedule io_schedule_timeout iounmap iowrite16 iowrite32 iowrite32be __iowrite32_copy __iowrite64_copy iowrite8 ip6_dst_hoplimit ip_compute_csum ip_mc_dec_group ip_mc_inc_group ipmi_create_user ipmi_destroy_user ipmi_free_recv_msg ipmi_poll_interface ipmi_request_settime ipmi_set_gets_events ipmi_set_my_address ipmi_smi_msg_received ipmi_unregister_smi ipmi_validate_addr ip_route_output_flow ip_send_check ip_tos2prio iput __ipv6_addr_type ipv6_ext_hdr ipv6_find_hdr ipv6_skip_exthdr ipv6_stub irq_cpu_rmap_add irq_create_mapping __irq_domain_add irq_domain_remove irq_find_mapping irq_modify_status irq_poll_complete irq_poll_disable irq_poll_enable irq_poll_init irq_poll_sched irq_set_affinity_hint irq_set_affinity_notifier irq_set_chip_and_handler_name irq_to_desc is_acpi_device_node iscsi_boot_create_ethernet iscsi_boot_create_host_kset iscsi_boot_create_initiator iscsi_boot_create_target iscsi_boot_destroy_kset __iscsi_complete_pdu iscsi_complete_scsi_task iscsi_conn_bind iscsi_conn_failure iscsi_conn_get_param iscsi_conn_send_pdu iscsi_conn_setup iscsi_conn_start iscsi_conn_stop iscsi_conn_teardown iscsi_create_endpoint iscsi_create_iface iscsi_destroy_endpoint iscsi_destroy_iface iscsi_eh_abort iscsi_eh_device_reset iscsi_eh_recover_target iscsi_eh_session_reset iscsi_get_port_speed_name iscsi_get_port_state_name __iscsi_get_task iscsi_host_add iscsi_host_alloc iscsi_host_for_each_session iscsi_host_free iscsi_host_get_param iscsi_host_remove iscsi_itt_to_task iscsi_lookup_endpoint iscsi_offload_mesg __iscsi_put_task iscsi_put_task iscsi_queuecommand iscsi_register_transport iscsi_session_failure iscsi_session_get_param iscsi_session_recovery_timedout iscsi_session_setup iscsi_session_teardown iscsi_set_param iscsi_suspend_queue iscsi_target_alloc iscsi_unregister_transport is_uv_system iw_cm_accept iw_cm_connect iw_cm_disconnect iw_cm_init_qp_attr iw_cm_listen iw_cm_reject iwcm_reject_msg iw_create_cm_id iw_destroy_cm_id jiffies jiffies_64 jiffies_to_msecs jiffies_to_timespec64 jiffies_to_usecs kallsyms_lookup_name kasprintf kernel_fpu_begin kernel_fpu_end kernel_recvmsg kernel_sendmsg kernel_setsockopt kfree kfree_call_rcu kfree_const kfree_skb kgdb_active kgdb_breakpoint kill_fasync __kmalloc kmalloc_caches __kmalloc_node kmalloc_order_trace kmem_cache_alloc kmem_cache_alloc_node kmem_cache_alloc_node_trace kmem_cache_alloc_trace kmem_cache_create kmem_cache_create_usercopy kmem_cache_destroy kmem_cache_free kmem_cache_shrink kmemdup kobject_add kobject_create_and_add kobject_del kobject_get kobject_init kobject_init_and_add kobject_put kobject_set_name kobject_uevent kobject_uevent_env krealloc kset_create_and_add kset_find_obj kset_register kset_unregister ksize kstrdup kstrdup_const kstrndup kstrtobool kstrtobool_from_user kstrtoint kstrtoint_from_user kstrtoll kstrtoll_from_user kstrtou16 kstrtou8 kstrtouint kstrtouint_from_user kstrtoul_from_user kstrtoull kstrtoull_from_user kthread_bind kthread_create_on_node kthread_park kthread_should_stop kthread_stop kthread_unpark ktime_get ktime_get_coarse_real_ts64 ktime_get_raw ktime_get_raw_ts64 ktime_get_real_seconds ktime_get_real_ts64 ktime_get_seconds ktime_get_ts64 ktime_get_with_offset kvasprintf kvfree kvmalloc_node kzfree led_classdev_resume led_classdev_suspend led_classdev_unregister libfc_vport_create __list_add_valid __list_del_entry_valid llist_add_batch __local_bh_enable_ip __lock_page lock_page_memcg lockref_get lock_sock_nested make_kgid make_kuid mark_page_accessed match_strdup match_string match_token mdio45_probe mdiobus_alloc_size mdiobus_free mdiobus_get_phy __mdiobus_register mdiobus_unregister mdio_mii_ioctl memchr memchr_inv memcmp memcpy memdup_user memdup_user_nul memmove memory_read_from_buffer memparse mempool_alloc mempool_alloc_slab mempool_create mempool_create_node mempool_destroy mempool_free mempool_free_slab mempool_kfree mempool_kmalloc memscan mem_section memset memzero_explicit metadata_dst_alloc mfd_add_devices mfd_remove_devices misc_deregister misc_register mlxfw_firmware_flash __mmdrop mmput __mmu_notifier_register mmu_notifier_register mmu_notifier_unregister mmu_notifier_unregister_no_release mod_delayed_work_on mod_timer mod_timer_pending __module_get module_layout module_put module_refcount __msecs_to_jiffies msleep msleep_interruptible __mutex_init mutex_lock mutex_lock_interruptible mutex_lock_killable mutex_trylock mutex_unlock mxm_wmi_call_mxds mxm_wmi_call_mxmx mxm_wmi_supported __napi_alloc_skb napi_complete_done napi_consume_skb napi_disable napi_get_frags napi_gro_flush napi_gro_frags napi_gro_receive napi_hash_del __napi_schedule __napi_schedule_irqoff napi_schedule_prep __ndelay ndo_dflt_bridge_getlink ndo_dflt_fdb_add nd_tbl neigh_destroy __neigh_event_send neigh_lookup netdev_alloc_frag __netdev_alloc_skb netdev_bind_sb_channel_queue netdev_crit netdev_err netdev_features_change netdev_info netdev_lower_get_next netdev_master_upper_dev_get netdev_master_upper_dev_get_rcu netdev_notice netdev_printk netdev_reset_tc netdev_rss_key_fill netdev_rx_handler_register netdev_rx_handler_unregister netdev_set_num_tc netdev_set_sb_channel netdev_set_tc_queue netdev_unbind_sb_channel netdev_update_features netdev_walk_all_upper_dev_rcu netdev_warn netif_carrier_off netif_carrier_on netif_device_attach netif_device_detach netif_get_num_default_rss_queues netif_napi_add netif_napi_del netif_receive_skb netif_rx netif_schedule_queue netif_set_real_num_rx_queues netif_set_real_num_tx_queues netif_set_xps_queue netif_tx_stop_all_queues netif_tx_wake_queue netlink_broadcast netlink_unicast net_ratelimit nla_find nla_parse nla_put nla_validate node_data __node_distance node_states node_to_cpumask_map no_llseek nonseekable_open noop_llseek nr_cpu_ids nr_node_ids nsecs_to_jiffies ns_to_timespec ns_to_timespec64 ns_to_timeval numa_node nvme_alloc_request nvme_cancel_request nvme_change_ctrl_state nvme_cleanup_cmd nvme_complete_async_event nvme_complete_rq nvme_disable_ctrl nvme_enable_ctrl nvme_fc_register_localport nvme_fc_register_remoteport nvme_fc_set_remoteport_devloss nvme_fc_unregister_localport nvme_fc_unregister_remoteport nvme_init_ctrl nvme_init_identify nvme_io_timeout nvme_kill_queues nvme_remove_namespaces nvme_reset_ctrl nvme_reset_ctrl_sync nvme_set_queue_count nvme_setup_cmd nvme_shutdown_ctrl nvme_start_ctrl nvme_start_freeze nvme_start_queues nvme_stop_ctrl nvme_stop_queues nvme_submit_sync_cmd nvmet_fc_rcv_fcp_abort nvmet_fc_rcv_fcp_req nvmet_fc_rcv_ls_req nvmet_fc_register_targetport nvmet_fc_unregister_targetport nvme_unfreeze nvme_uninit_ctrl nvme_wait_freeze nvme_wait_freeze_timeout nvme_wq of_led_classdev_register on_each_cpu orderly_poweroff out_of_line_wait_on_bit out_of_line_wait_on_bit_lock override_creds __page_file_index __page_frag_cache_drain page_frag_free __page_mapcount page_mapped page_offset_base page_pool_alloc_pages page_pool_create page_pool_destroy __page_pool_put_page pagevec_lookup_range pagevec_lookup_range_tag __pagevec_release panic panic_notifier_list param_array_ops param_get_int param_ops_bool param_ops_byte param_ops_charp param_ops_int param_ops_long param_ops_short param_ops_string param_ops_uint param_ops_ullong param_ops_ulong param_ops_ushort param_set_bool param_set_int pat_enabled pci_alloc_irq_vectors_affinity pci_assign_unassigned_bus_resources pcibios_resource_to_bus pci_bus_resource_n pci_bus_type pci_cfg_access_lock pci_cfg_access_unlock pci_choose_state pci_cleanup_aer_uncorrect_error_status pci_clear_master pci_clear_mwi pci_d3cold_disable pci_dev_driver pci_dev_get pci_device_is_present pci_dev_present pci_dev_put pci_disable_device pci_disable_link_state pci_disable_msi pci_disable_msix pci_disable_pcie_error_reporting pci_disable_rom pci_disable_sriov pcie_bandwidth_available pcie_capability_clear_and_set_word pcie_capability_read_dword pcie_capability_read_word pcie_capability_write_word pcie_flr pcie_get_speed_cap pcie_get_width_cap pci_enable_atomic_ops_to_root pci_enable_device pci_enable_device_mem pci_enable_msi pci_enable_msix_range pci_enable_pcie_error_reporting pci_enable_rom pci_enable_sriov pci_enable_wake pcie_print_link_status pcie_relaxed_ordering_enabled pcie_set_readrq pci_find_capability pci_find_ext_capability pci_free_irq pci_free_irq_vectors pci_get_class pci_get_device pci_get_domain_bus_and_slot pci_get_slot pci_ignore_hotplug pci_intx pci_iomap pci_ioremap_bar pci_iounmap pci_irq_get_affinity pci_irq_vector pci_map_rom pcim_enable_device pcim_iomap_regions pcim_iomap_table pci_num_vf pci_platform_rom pci_prepare_to_sleep pci_read_config_byte pci_read_config_dword pci_read_config_word pci_read_vpd __pci_register_driver pci_release_regions pci_release_resource pci_release_selected_regions pci_request_irq pci_request_regions pci_request_selected_regions pci_rescan_bus pci_resize_resource pci_restore_state pci_save_state pci_select_bars pci_set_master pci_set_mwi pci_set_power_state pci_sriov_configure_simple pci_sriov_get_totalvfs pci_sriov_set_totalvfs pci_stop_and_remove_bus_device pci_stop_and_remove_bus_device_locked pci_try_set_mwi pci_unmap_rom pci_unregister_driver pci_vfs_assigned pci_vpd_find_info_keyword pci_vpd_find_tag pci_wait_for_pending_transaction pci_wake_from_d3 pci_walk_bus pci_write_config_byte pci_write_config_dword pci_write_config_word pcix_set_mmrbc PDE_DATA __per_cpu_offset perf_tp_event perf_trace_buf_alloc perf_trace_run_bpf_submit pgprot_writecombine phy_attached_info phy_connect phy_disconnect phy_ethtool_ksettings_get phy_ethtool_ksettings_set phy_ethtool_sset phy_mii_ioctl phys_base physical_mask phy_start phy_start_aneg phy_stop pid_task pid_vnr platform_bus_type pm_genpd_add_device pm_genpd_init pm_genpd_remove_device pm_power_off pm_runtime_allow __pm_runtime_disable pm_runtime_enable pm_runtime_forbid __pm_runtime_idle __pm_runtime_resume pm_runtime_set_autosuspend_delay __pm_runtime_set_status __pm_runtime_suspend __pm_runtime_use_autosuspend pm_schedule_suspend pm_vt_switch_required pm_vt_switch_unregister power_supply_is_system_supplied prandom_bytes prandom_seed prandom_u32 __preempt_count prepare_creds prepare_to_wait prepare_to_wait_event prepare_to_wait_exclusive print_hex_dump printk __printk_ratelimit print_stack_trace proc_create_data proc_dointvec proc_mkdir proc_mkdir_mode proc_remove proc_symlink __pskb_copy_fclone pskb_expand_head __pskb_pull_tail ___pskb_trim ptp_clock_event ptp_clock_index ptp_clock_register ptp_clock_unregister ptp_find_pin __put_cred put_device __put_devmap_managed_page put_disk __put_net __put_page put_pid __put_task_struct put_unused_fd __put_user_1 __put_user_2 __put_user_4 __put_user_8 pv_cpu_ops pv_irq_ops pv_lock_ops pv_mmu_ops qed_get_eth_ops qed_put_eth_ops queue_delayed_work_on queued_read_lock_slowpath queued_write_lock_slowpath queue_work_on radix_tree_delete radix_tree_gang_lookup __radix_tree_insert radix_tree_iter_delete radix_tree_lookup radix_tree_lookup_slot radix_tree_next_chunk __radix_tree_next_slot radix_tree_preload radix_tree_tagged raid_class_attach raid_class_release ___ratelimit raw_notifier_call_chain raw_notifier_chain_register raw_notifier_chain_unregister _raw_read_lock _raw_read_lock_bh _raw_read_lock_irq _raw_read_lock_irqsave _raw_read_unlock_bh _raw_read_unlock_irqrestore _raw_spin_lock _raw_spin_lock_bh _raw_spin_lock_irq _raw_spin_lock_irqsave _raw_spin_trylock _raw_spin_unlock_bh _raw_spin_unlock_irqrestore _raw_write_lock _raw_write_lock_bh _raw_write_lock_irq _raw_write_lock_irqsave _raw_write_unlock_bh _raw_write_unlock_irqrestore rb_erase __rb_erase_color rb_first rb_first_postorder __rb_insert_augmented rb_insert_color rb_next rb_next_postorder rb_replace_node rbt_ib_umem_for_each_in_range rbt_ib_umem_lookup rcu_barrier rdma_addr_cancel rdma_addr_size rdma_create_ah rdma_create_user_ah rdma_destroy_ah rdma_destroy_ah_attr rdma_find_gid rdma_find_gid_by_port rdma_get_gid_attr rdma_is_zero_gid rdma_move_ah_attr rdma_nl_multicast rdma_nl_register rdma_nl_unicast rdma_nl_unicast_wait rdma_nl_unregister rdma_node_get_transport rdma_port_get_link_layer rdma_put_gid_attr rdma_query_ah rdma_query_gid rdma_resolve_ip rdma_restrack_del rdma_roce_rescan_device rdma_set_cq_moderation rdma_translate_ip read_cache_pages recalc_sigpending refcount_dec_and_mutex_lock refcount_dec_and_test_checked refcount_inc_checked refcount_inc_not_zero_checked register_acpi_notifier register_blkdev __register_chrdev register_chrdev_region register_fib_notifier register_inet6addr_notifier register_inetaddr_notifier register_netdev register_netdevice_notifier register_netevent_notifier register_net_sysctl __register_nmi_handler register_pernet_subsys register_reboot_notifier release_firmware release_pages __release_region release_sock remap_pfn_range remove_conflicting_framebuffers remove_proc_entry remove_wait_queue request_firmware request_firmware_direct request_firmware_nowait __request_module __request_region request_threaded_irq reservation_object_add_excl_fence reservation_object_add_shared_fence reservation_object_get_fences_rcu reservation_object_reserve_shared reservation_object_wait_timeout_rcu reservation_ww_class reset_devices revert_creds rhashtable_destroy rhashtable_free_and_destroy rhashtable_init rhashtable_insert_slow rhashtable_walk_enter rhashtable_walk_exit rhashtable_walk_next rhashtable_walk_start_check rhashtable_walk_stop rhltable_init rht_bucket_nested rht_bucket_nested_insert ring_buffer_event_data roce_gid_type_mask_support round_jiffies round_jiffies_relative rps_may_expire_flow rt6_lookup rtc_time64_to_tm rtnl_is_locked rtnl_lock rtnl_trylock rtnl_unlock sas_attach_transport sas_disable_tlr sas_enable_tlr sas_end_device_alloc sas_expander_alloc sas_is_tlr_enabled sas_phy_add sas_phy_alloc sas_phy_free sas_port_add sas_port_add_phy sas_port_alloc_num sas_port_delete sas_port_delete_phy sas_port_free sas_read_port_mode_page sas_release_transport sas_remove_host sas_rphy_add save_stack_trace save_stack_trace_tsk sbitmap_queue_clear __sbitmap_queue_get sched_setscheduler schedule schedule_hrtimeout schedule_hrtimeout_range schedule_timeout schedule_timeout_interruptible schedule_timeout_uninterruptible scmd_printk scnprintf screen_info scsi_add_device scsi_add_host_with_dma scsi_block_requests scsi_build_sense_buffer scsi_change_queue_depth scsi_device_get scsi_device_lookup scsi_device_put scsi_device_set_state scsi_device_type scsi_dma_map scsi_dma_unmap __scsi_execute scsi_get_vpd_page scsi_host_alloc scsi_host_busy scsi_host_get scsi_host_lookup scsi_host_put scsi_internal_device_block_nowait scsi_internal_device_unblock_nowait scsi_is_fc_rport scsi_is_host_device scsi_is_sdev_device __scsi_iterate_devices scsilun_to_int scsi_normalize_sense scsi_print_command scsi_register_driver scsi_remove_device scsi_remove_host scsi_remove_target scsi_sanitize_inquiry_string scsi_scan_host scsi_track_queue_full scsi_unblock_requests sdev_prefix_printk secpath_dup security_d_instantiate send_sig seq_lseek seq_open seq_printf seq_putc seq_puts seq_read seq_release seq_write set_cpus_allowed_ptr set_current_groups set_device_ro set_disk_ro set_freezable set_memory_array_uc set_memory_array_wb set_memory_uc set_memory_wb set_memory_wc set_normalized_timespec set_normalized_timespec64 set_page_dirty set_page_dirty_lock set_user_nice sg_alloc_table_from_pages sg_copy_from_buffer sg_copy_to_buffer sg_free_table sg_init_table sg_miter_next sg_miter_start sg_miter_stop sg_next show_class_attr_string sigprocmask si_meminfo simple_open simple_read_from_buffer simple_strtol simple_strtoul simple_strtoull simple_write_to_buffer single_open single_release skb_add_rx_frag skb_checksum skb_checksum_help skb_clone skb_clone_tx_timestamp skb_copy skb_copy_bits skb_copy_expand skb_dequeue __skb_flow_dissect __skb_get_hash __skb_gso_segment skb_gso_validate_mac_len __skb_pad skb_pull skb_push skb_put skb_queue_purge skb_queue_tail skb_realloc_headroom skb_store_bits skb_trim skb_tstamp_tx skb_vlan_pop sme_active sme_me_mask smp_call_function_many smp_call_function_single snprintf sn_rtc_cycles_per_second sock_create_kern sock_edemux sock_release softnet_data sort sprintf __srcu_read_lock __srcu_read_unlock sscanf __stack_chk_fail starget_for_each_device strcasecmp strcat strchr strcmp strcpy strcspn strim strlcat strlcpy strlen strncasecmp strncat strncmp strncpy strncpy_from_user strnlen strnstr strpbrk strrchr strscpy strsep strspn strstr submit_bio __sw_hweight32 __sw_hweight64 swiotlb_nr_tbl switchdev_port_same_parent_id __symbol_get __symbol_put sync_file_create synchronize_irq synchronize_net synchronize_sched synchronize_srcu sysfs_add_file_to_group sysfs_create_bin_file sysfs_create_file_ns sysfs_create_group sysfs_format_mac sysfs_remove_bin_file sysfs_remove_file_from_group sysfs_remove_file_ns sysfs_remove_group sysfs_streq system_state system_unbound_wq system_wq sys_tz task_active_pid_ns tasklet_init tasklet_kill __tasklet_schedule __task_pid_nr_ns tcf_block_cb_register tcf_block_cb_unregister tcp_gro_complete tcp_hashinfo tc_setup_cb_egdev_register tc_setup_cb_egdev_unregister this_cpu_off time64_to_tm timecounter_cyc2time timecounter_init timecounter_read to_drm_sched_fence trace_define_field trace_event_buffer_commit trace_event_buffer_lock_reserve trace_event_buffer_reserve trace_event_ignore_this_pid trace_event_raw_init trace_event_reg trace_handle_return __tracepoint_dma_fence_emit __tracepoint_xdp_exception trace_print_flags_seq trace_raw_output_prep trace_seq_printf trace_seq_putc try_module_get try_wait_for_completion tsc_khz ttm_bo_add_to_lru ttm_bo_clean_mm ttm_bo_del_sub_from_lru ttm_bo_device_init ttm_bo_device_release ttm_bo_dma_acc_size ttm_bo_eviction_valuable ttm_bo_evict_mm ttm_bo_global_init ttm_bo_global_release ttm_bo_init ttm_bo_init_mm ttm_bo_init_reserved ttm_bo_kmap ttm_bo_kunmap ttm_bo_lock_delayed_workqueue ttm_bo_manager_func ttm_bo_mem_put ttm_bo_mem_space ttm_bo_mmap ttm_bo_move_accel_cleanup ttm_bo_move_memcpy ttm_bo_move_to_lru_tail ttm_bo_move_ttm ttm_bo_pipeline_move ttm_bo_put ttm_bo_unlock_delayed_workqueue ttm_bo_validate ttm_bo_wait ttm_dma_page_alloc_debugfs ttm_dma_populate ttm_dma_tt_fini ttm_dma_tt_init ttm_dma_unpopulate ttm_eu_backoff_reservation ttm_eu_fence_buffer_objects ttm_eu_reserve_buffers ttm_fbdev_mmap ttm_mem_global_init ttm_mem_global_release ttm_page_alloc_debugfs ttm_pool_populate ttm_pool_unpopulate ttm_populate_and_map_pages ttm_sg_tt_init ttm_tt_bind ttm_tt_set_placement_caching ttm_unmap_and_unpopulate_pages __udelay udp4_hwcsum uio_event_notify __uio_register_device uio_unregister_device unlock_page unlock_page_memcg unmap_mapping_range unregister_acpi_notifier unregister_blkdev __unregister_chrdev unregister_chrdev_region unregister_fib_notifier unregister_inet6addr_notifier unregister_inetaddr_notifier unregister_netdev unregister_netdevice_notifier unregister_netevent_notifier unregister_net_sysctl_table unregister_nmi_handler unregister_pernet_subsys unregister_reboot_notifier unuse_mm up up_read up_write __usecs_to_jiffies use_mm usleep_range __uv_cpu_info _uverbs_alloc uverbs_copy_to uverbs_destroy_def_handler uverbs_get_flags32 uverbs_get_flags64 uverbs_idr_class __uv_hub_info_list uv_possible_blades uv_setup_irq uv_teardown_irq vfree vga_client_register vgacon_text_force vga_set_legacy_decoding vga_switcheroo_client_fb_set vga_switcheroo_client_probe_defer vga_switcheroo_fini_domain_pm_ops vga_switcheroo_handler_flags vga_switcheroo_init_domain_pm_ops vga_switcheroo_lock_ddc vga_switcheroo_process_delayed_switch vga_switcheroo_register_client vga_switcheroo_register_handler vga_switcheroo_unlock_ddc vga_switcheroo_unregister_client vga_switcheroo_unregister_handler vga_tryget __virt_addr_valid vlan_dev_real_dev vlan_dev_vlan_id vlan_dev_vlan_proto vmalloc __vmalloc vmalloc_base vmalloc_node vmalloc_to_page vmap vmemmap_base vm_get_page_prot vm_insert_page vm_mmap vm_munmap vprintk vscnprintf vsnprintf vsprintf vunmap vzalloc vzalloc_node wait_for_completion wait_for_completion_interruptible wait_for_completion_interruptible_timeout wait_for_completion_io_timeout wait_for_completion_killable wait_for_completion_timeout wait_on_page_bit __wake_up wake_up_bit __wake_up_locked wake_up_process __warn_printk wmi_evaluate_method wmi_has_guid work_busy write_cache_pages ww_mutex_lock ww_mutex_lock_interruptible ww_mutex_unlock x86_cpu_to_apicid x86_dma_fallback_dev __x86_indirect_thunk_r10 __x86_indirect_thunk_r11 __x86_indirect_thunk_r12 __x86_indirect_thunk_r13 __x86_indirect_thunk_r14 __x86_indirect_thunk_r15 __x86_indirect_thunk_r8 __x86_indirect_thunk_r9 __x86_indirect_thunk_rax __x86_indirect_thunk_rbp __x86_indirect_thunk_rbx __x86_indirect_thunk_rcx __x86_indirect_thunk_rdi __x86_indirect_thunk_rdx __x86_indirect_thunk_rsi xdp_do_flush_map xdp_do_redirect xdp_return_frame xdp_return_frame_rx_napi xdp_rxq_info_is_reg xdp_rxq_info_reg xdp_rxq_info_reg_mem_model xdp_rxq_info_unreg xdp_rxq_info_unused xfrm_replay_seqhi xz_dec_end xz_dec_init xz_dec_run yield zalloc_cpumask_var zap_vma_ptes zlib_inflate zlib_inflateEnd zlib_inflateInit2 zlib_inflate_workspacesize

2 2

来自严亮的邮件
by 严亮 04 Mar '21

04 Mar '21

yanl1229(a)163.com

1 0

[PATCH openEuler-21.03 1/2] mm: add pin memory method for checkpoint add restore
by hejingxian 02 Mar '21

02 Mar '21

From: Jingxian He <hejingxian(a)huawei.com> Date: Mon, 1 Mar 2021 17:35:32 +0800 Subject: [PATCH openEuler-21.03 1/2] mm: add pin memory method for checkpoint add restore hulk inclusion category: feature bugzilla: 48159 CVE: N/A We can use the checkpoint and restore in userspace(criu) method to dump and restore tasks when updating the kernel. Currently, criu needs dump all memory data of tasks to files. When the memory size is very large(larger than 1G), the cost time of the dumping data will be very long(more than 1 min). By pin the memory data of tasks and collect the corresponding physical pages mapping info in checkpoint process, we can remap the physical pages to restore tasks after upgrading the kernel. This pin memory method can restore the task data within one second. The pin memory area info is saved in the reserved memblock, which can keep usable in the kernel update process. The pin memory driver provides the following ioctl command for criu: 1) SET_PIN_MEM_AREA: Set pin memory area, which can be remap to the restore task. 2) CLEAR_PIN_MEM_AREA: Clear the pin memory area info, which enable user reset the pin data. 3) REMAP_PIN_MEM_AREA: Remap the pages of the pin memory to the restore task. Signed-off-by: Jingxian He <hejingxian(a)huawei.com> Reviewed-by: Wenliang He <hewenliang4(a)huawei.com> Reviewed-by: Jing Xiangfeng <jingxiangfeng(a)huawei.com> --- arch/arm64/configs/openeuler_defconfig | 2 + arch/arm64/kernel/setup.c | 9 + arch/arm64/mm/init.c | 60 +++ drivers/char/Kconfig | 6 + drivers/char/Makefile | 1 + drivers/char/pin_memory.c | 208 ++++++++ include/linux/crash_core.h | 5 + include/linux/pin_mem.h | 78 +++ kernel/crash_core.c | 11 + mm/Kconfig | 8 + mm/Makefile | 1 + mm/huge_memory.c | 61 +++ mm/memory.c | 59 ++ mm/pin_mem.c | 950 +++++++++++++++++++++++++++++++++ 14 files changed, 1459 insertions(+) create mode 100644 drivers/char/pin_memory.c create mode 100644 include/linux/pin_mem.h create mode 100644 mm/pin_mem.c diff --git a/arch/arm64/configs/openeuler_defconfig b/arch/arm64/configs/openeuler_defconfig index c5271e7..76fda68 100644 --- a/arch/arm64/configs/openeuler_defconfig +++ b/arch/arm64/configs/openeuler_defconfig @@ -1036,6 +1036,7 @@ CONFIG_FRAME_VECTOR=y # CONFIG_GUP_BENCHMARK is not set # CONFIG_READ_ONLY_THP_FOR_FS is not set CONFIG_ARCH_HAS_PTE_SPECIAL=y +CONFIG_PIN_MEMORY=y # end of Memory Management options CONFIG_NET=y @@ -3282,6 +3283,7 @@ CONFIG_TCG_TIS_ST33ZP24_SPI=y # CONFIG_RANDOM_TRUST_CPU is not set # CONFIG_RANDOM_TRUST_BOOTLOADER is not set +CONFIG_PIN_MEMORY_DEV=m # # I2C support diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c index c1f1fb9..5e282d3 100644 --- a/arch/arm64/kernel/setup.c +++ b/arch/arm64/kernel/setup.c @@ -50,6 +50,9 @@ #include <asm/efi.h> #include <asm/xen/hypervisor.h> #include <asm/mmu_context.h> +#ifdef CONFIG_PIN_MEMORY +#include <linux/pin_mem.h> +#endif static int num_standard_resources; static struct resource *standard_resources; @@ -260,6 +263,12 @@ static void __init request_standard_resources(void) quick_kexec_res.end <= res->end) request_resource(res, &quick_kexec_res); #endif +#ifdef CONFIG_PIN_MEMORY + if (pin_memory_resource.end && + pin_memory_resource.start >= res->start && + pin_memory_resource.end <= res->end) + request_resource(res, &pin_memory_resource); +#endif } } diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c index f3e5a66..8ab5aac 100644 --- a/arch/arm64/mm/init.c +++ b/arch/arm64/mm/init.c @@ -42,6 +42,9 @@ #include <linux/sizes.h> #include <asm/tlb.h> #include <asm/alternative.h> +#ifdef CONFIG_PIN_MEMORY +#include <linux/pin_mem.h> +#endif #define ARM64_ZONE_DMA_BITS 30 @@ -78,6 +81,55 @@ static void __init reserve_crashkernel(void) */ #define MAX_USABLE_RANGES 2 +#ifdef CONFIG_PIN_MEMORY +struct resource pin_memory_resource = { + .name = "Pin memory", + .start = 0, + .end = 0, + .flags = IORESOURCE_MEM, + .desc = IORES_DESC_RESERVED +}; + +static void __init reserve_pin_memory_res(void) +{ + unsigned long long mem_start, mem_len; + int ret; + + ret = parse_pin_memory(boot_command_line, memblock_phys_mem_size(), + &mem_len, &mem_start); + if (ret || !mem_len) + return; + + mem_len = PAGE_ALIGN(mem_len); + + if (!memblock_is_region_memory(mem_start, mem_len)) { + pr_warn("cannot reserve for pin memory: region is not memory!\n"); + return; + } + + if (memblock_is_region_reserved(mem_start, mem_len)) { + pr_warn("cannot reserve for pin memory: region overlaps reserved memory!\n"); + return; + } + + if (!IS_ALIGNED(mem_start, SZ_2M)) { + pr_warn("cannot reserve for pin memory: base address is not 2MB aligned\n"); + return; + } + + memblock_reserve(mem_start, mem_len); + pr_debug("pin memory resource reserved: 0x%016llx - 0x%016llx (%lld MB)\n", + mem_start, mem_start + mem_len, mem_len >> 20); + + pin_memory_resource.start = mem_start; + pin_memory_resource.end = mem_start + mem_len - 1; +} +#else +static void __init reserve_pin_memory_res(void) +{ +} +#endif /* CONFIG_PIN_MEMORY */ + #ifdef CONFIG_CRASH_DUMP static int __init early_init_dt_scan_elfcorehdr(unsigned long node, const char *uname, int depth, void *data) @@ -455,6 +507,8 @@ void __init arm64_memblock_init(void) reserve_park_mem(); #endif + reserve_pin_memory_res(); + reserve_elfcorehdr(); high_memory = __va(memblock_end_of_DRAM() - 1) + 1; @@ -583,6 +637,12 @@ void __init mem_init(void) /* this will put all unused low memory onto the freelists */ memblock_free_all(); +#ifdef CONFIG_PIN_MEMORY + /* pre alloc the pages for pin memory */ + init_reserve_page_map((unsigned long)pin_memory_resource.start, + (unsigned long)(pin_memory_resource.end - pin_memory_resource.start + 1)); +#endif + mem_init_print_info(NULL); /* diff --git a/drivers/char/Kconfig b/drivers/char/Kconfig index d229a2d..fbb94b8 100644 --- a/drivers/char/Kconfig +++ b/drivers/char/Kconfig @@ -496,3 +496,9 @@ config RANDOM_TRUST_BOOTLOADER booloader is trustworthy so it will be added to the kernel's entropy pool. Otherwise, say N here so it will be regarded as device input that only mixes the entropy pool. + +config PIN_MEMORY_DEV + bool "/dev/pinmem character device" + default m + help + pin memory driver diff --git a/drivers/char/Makefile b/drivers/char/Makefile index ffce287..71d76fd 100644 --- a/drivers/char/Makefile +++ b/drivers/char/Makefile @@ -47,3 +47,4 @@ obj-$(CONFIG_PS3_FLASH) += ps3flash.o obj-$(CONFIG_XILLYBUS) += xillybus/ obj-$(CONFIG_POWERNV_OP_PANEL) += powernv-op-panel.o obj-$(CONFIG_ADI) += adi.o +obj-$(CONFIG_PIN_MEMORY_DEV) += pin_memory.o diff --git a/drivers/char/pin_memory.c b/drivers/char/pin_memory.c new file mode 100644 index 0000000..f46e056 --- /dev/null +++ b/drivers/char/pin_memory.c @@ -0,0 +1,208 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2021. Huawei Technologies Co., Ltd. All rights reserved. + * Pin memory driver for checkpoint and restore. + */ +#include <linux/kernel.h> +#include <linux/module.h> +#include <linux/kprobes.h> +#include <linux/spinlock.h> +#include <linux/workqueue.h> +#include <linux/sched.h> +#include <linux/mm.h> +#include <linux/init.h> +#include <linux/miscdevice.h> +#include <linux/fs.h> +#include <linux/mm_types.h> +#include <linux/processor.h> +#include <uapi/asm-generic/ioctl.h> +#include <uapi/asm-generic/mman-common.h> +#include <uapi/asm/setup.h> +#include <linux/pin_mem.h> +#include <linux/sched/mm.h> + +#define MAX_PIN_MEM_AREA_NUM 16 +struct _pin_mem_area { + unsigned long virt_start; + unsigned long virt_end; +}; + +struct pin_mem_area_set { + unsigned int pid; + unsigned int area_num; + struct _pin_mem_area mem_area[MAX_PIN_MEM_AREA_NUM]; +}; + +#define PIN_MEM_MAGIC 0x59 +#define _SET_PIN_MEM_AREA 1 +#define _CLEAR_PIN_MEM_AREA 2 +#define _REMAP_PIN_MEM_AREA 3 +#define _FINISH_PIN_MEM_DUMP 4 +#define _PIN_MEM_IOC_MAX_NR 4 +#define SET_PIN_MEM_AREA _IOW(PIN_MEM_MAGIC, _SET_PIN_MEM_AREA, struct pin_mem_area_set) +#define CLEAR_PIN_MEM_AREA _IOW(PIN_MEM_MAGIC, _CLEAR_PIN_MEM_AREA, int) +#define REMAP_PIN_MEM_AREA _IOW(PIN_MEM_MAGIC, _REMAP_PIN_MEM_AREA, int) +#define FINISH_PIN_MEM_DUMP _IOW(PIN_MEM_MAGIC, _FINISH_PIN_MEM_DUMP, int) +static int set_pin_mem(struct pin_mem_area_set *pmas) +{ + int i; + int ret = 0; + struct _pin_mem_area *pma; + struct mm_struct *mm; + struct task_struct *task; + struct pid *pid_s; + + pid_s = find_get_pid(pmas->pid); + if (!pid_s) { + pr_warn("Get pid struct fail:%d.\n", pmas->pid); + return -EFAULT; + } + rcu_read_lock(); + task = pid_task(pid_s, PIDTYPE_PID); + if (!task) { + pr_warn("Get task struct fail:%d.\n", pmas->pid); + goto fail; + } + mm = get_task_mm(task); + for (i = 0; i < pmas->area_num; i++) { + pma = &(pmas->mem_area[i]); + ret = pin_mem_area(task, mm, pma->virt_start, pma->virt_end); + if (ret) { + mmput(mm); + goto fail; + } + } + mmput(mm); + rcu_read_unlock(); + put_pid(pid_s); + return ret; + +fail: + rcu_read_unlock(); + put_pid(pid_s); + return -EFAULT; +} + +static int set_pin_mem_area(unsigned long arg) +{ + struct pin_mem_area_set pmas; + void __user *buf = (void __user *)arg; + + if (!access_ok(buf, sizeof(pmas))) + return -EFAULT; + if (copy_from_user(&pmas, buf, sizeof(pmas))) + return -EINVAL; + if (pmas.area_num > MAX_PIN_MEM_AREA_NUM) { + pr_warn("Input area_num is too large.\n"); + return -EINVAL; + } + + return set_pin_mem(&pmas); +} + +static int pin_mem_remap(unsigned long arg) +{ + int pid; + struct task_struct *task; + struct mm_struct *mm; + vm_fault_t ret; + void __user *buf = (void __user *)arg; + struct pid *pid_s; + + if (!access_ok(buf, sizeof(int))) + return -EINVAL; + if (copy_from_user(&pid, buf, sizeof(int))) + return -EINVAL; + + pid_s = find_get_pid(pid); + if (!pid_s) { + pr_warn("Get pid struct fail:%d.\n", pid); + return -EINVAL; + } + rcu_read_lock(); + task = pid_task(pid_s, PIDTYPE_PID); + if (!task) { + pr_warn("Get task struct fail:%d.\n", pid); + goto fault; + } + mm = get_task_mm(task); + ret = do_mem_remap(pid, mm); + if (ret) { + pr_warn("Handle pin memory remap fail.\n"); + mmput(mm); + goto fault; + } + mmput(mm); + rcu_read_unlock(); + put_pid(pid_s); + return 0; + +fault: + rcu_read_unlock(); + put_pid(pid_s); + return -EFAULT; +} + +static long pin_memory_ioctl(struct file *file, unsigned int cmd, unsigned long arg) +{ + long ret = 0; + + if (_IOC_TYPE(cmd) != PIN_MEM_MAGIC) + return -EINVAL; + if (_IOC_NR(cmd) > _PIN_MEM_IOC_MAX_NR) + return -EINVAL; + + switch (cmd) { + case SET_PIN_MEM_AREA: + ret = set_pin_mem_area(arg); + break; + case CLEAR_PIN_MEM_AREA: + clear_pin_memory_record(); + break; + case REMAP_PIN_MEM_AREA: + ret = pin_mem_remap(arg); + break; + case FINISH_PIN_MEM_DUMP: + ret = finish_pin_mem_dump(); + break; + default: + return -EINVAL; + } + return ret; +} + +static const struct file_operations pin_memory_fops = { + .owner = THIS_MODULE, + .unlocked_ioctl = pin_memory_ioctl, + .compat_ioctl = pin_memory_ioctl, +}; + +static struct miscdevice pin_memory_miscdev = { + .minor = MISC_DYNAMIC_MINOR, + .name = "pinmem", + .fops = &pin_memory_fops, +}; + +static int pin_memory_init(void) +{ + int err = misc_register(&pin_memory_miscdev); + + if (!err) + pr_info("pin_memory init\n"); + else + pr_warn("pin_memory init failed!\n"); + return err; +} + +static void pin_memory_exit(void) +{ + misc_deregister(&pin_memory_miscdev); + pr_info("pin_memory ko exists!\n"); +} + +module_init(pin_memory_init); +module_exit(pin_memory_exit); + +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Euler"); +MODULE_DESCRIPTION("pin memory"); diff --git a/include/linux/crash_core.h b/include/linux/crash_core.h index fc0ef33..30f0df3 100644 --- a/include/linux/crash_core.h +++ b/include/linux/crash_core.h @@ -87,4 +87,9 @@ int parse_crashkernel_high(char *cmdline, unsigned long long system_ram, int parse_crashkernel_low(char *cmdline, unsigned long long system_ram, unsigned long long *crash_size, unsigned long long *crash_base); +#ifdef CONFIG_PIN_MEMORY +int __init parse_pin_memory(char *cmdline, unsigned long long system_ram, + unsigned long long *pin_size, unsigned long long *pin_base); +#endif + #endif /* LINUX_CRASH_CORE_H */ diff --git a/include/linux/pin_mem.h b/include/linux/pin_mem.h new file mode 100644 index 0000000..bc8b03e --- /dev/null +++ b/include/linux/pin_mem.h @@ -0,0 +1,78 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (C) 2021. Huawei Technologies Co., Ltd. All rights reserved. + * Provide the pin memory method for checkpoint and restore task. + */ +#ifndef _LINUX_PIN_MEMORY_H +#define _LINUX_PIN_MEMORY_H + +#ifdef CONFIG_PIN_MEMORY +#include <linux/errno.h> +#include <linux/mm_types.h> +#include <linux/err.h> +#ifdef CONFIG_ARM64 +#include <linux/ioport.h> +#endif + +#define PAGE_BUDDY_MAPCOUNT_VALUE (~PG_buddy) + +#define COLLECT_PAGES_FINISH 0 +#define COLLECT_PAGES_NEED_CONTINUE 1 +#define COLLECT_PAGES_FAIL -1 + +#define COMPOUND_PAD_MASK 0xffffffff +#define COMPOUND_PAD_START 0x88 +#define COMPOUND_PAD_DELTA 0x40 +#define LIST_POISON4 0xdead000000000400 +#define PAGE_FLAGS_CHECK_RESERVED (1UL << PG_reserved) +#define SHA256_DIGEST_SIZE 32 +#define next_pme(pme) ((unsigned long *)(pme + 1) + pme->nr_pages) +#define PIN_MEM_DUMP_MAGIC 0xfeab000000001acd +struct page_map_entry { + unsigned long virt_addr; + unsigned int nr_pages; + unsigned int is_huge_page; + unsigned long redirect_start; + unsigned long phy_addr_array[0]; +}; + +struct page_map_info { + int pid; + int pid_reserved; + unsigned int entry_num; + int disable_free_page; + struct page_map_entry *pme; +}; + +struct pin_mem_dump_info { + char sha_digest[SHA256_DIGEST_SIZE]; + unsigned long magic; + unsigned int pin_pid_num; + struct page_map_info pmi_array[0]; +}; + +struct redirect_info { + unsigned int redirect_pages; + unsigned int redirect_index[0]; +}; + +extern struct page_map_info *get_page_map_info(int pid); +extern struct page_map_info *create_page_map_info(int pid); +extern vm_fault_t do_mem_remap(int pid, struct mm_struct *mm); +extern vm_fault_t do_anon_page_remap(struct vm_area_struct *vma, unsigned long address, + pmd_t *pmd, struct page *page); +extern void clear_pin_memory_record(void); +extern int pin_mem_area(struct task_struct *task, struct mm_struct *mm, + unsigned long start_addr, unsigned long end_addr); +extern vm_fault_t do_anon_huge_page_remap(struct vm_area_struct *vma, unsigned long address, + pmd_t *pmd, struct page *page); +extern int finish_pin_mem_dump(void); + +/* reserve space for pin memory*/ +#ifdef CONFIG_ARM64 +extern struct resource pin_memory_resource; +#endif +extern void init_reserve_page_map(unsigned long map_addr, unsigned long map_size); + +#endif /* CONFIG_PIN_MEMORY */ +#endif /* _LINUX_PIN_MEMORY_H */ diff --git a/kernel/crash_core.c b/kernel/crash_core.c index bfed474..2407de3 100644 --- a/kernel/crash_core.c +++ b/kernel/crash_core.c @@ -450,6 +450,17 @@ void __init reserve_crashkernel(void) } #endif /* CONFIG_ARCH_WANT_RESERVE_CRASH_KERNEL */ +#ifdef CONFIG_PIN_MEMORY +int __init parse_pin_memory(char *cmdline, + unsigned long long system_ram, + unsigned long long *pin_size, + unsigned long long *pin_base) +{ + return __parse_crashkernel(cmdline, system_ram, pin_size, pin_base, + "pinmemory=", NULL); +} +#endif + Elf_Word *append_elf_note(Elf_Word *buf, char *name, unsigned int type, void *data, size_t data_len) { diff --git a/mm/Kconfig b/mm/Kconfig index 390165f..930dc13 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -859,4 +859,12 @@ config ARCH_HAS_HUGEPD config MAPPING_DIRTY_HELPERS bool +config PIN_MEMORY + bool "Support for pin memory" + depends on CHECKPOINT_RESTORE + help + Say y here to enable the pin memory feature for checkpoint + and restore. We can pin the memory data of tasks and collect + the corresponding physical pages mapping info in checkpoint, + and remap the physical pages to restore tasks in restore. endmenu diff --git a/mm/Makefile b/mm/Makefile index d73aed0..4963827 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -120,3 +120,4 @@ obj-$(CONFIG_MEMFD_CREATE) += memfd.o obj-$(CONFIG_MAPPING_DIRTY_HELPERS) += mapping_dirty_helpers.o obj-$(CONFIG_PTDUMP_CORE) += ptdump.o obj-$(CONFIG_PAGE_REPORTING) += page_reporting.o +obj-$(CONFIG_PIN_MEMORY) += pin_mem.o diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 0bc4a2c..8a11d30 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2996,3 +2996,64 @@ void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new) update_mmu_cache_pmd(vma, address, pvmw->pmd); } #endif + +#ifdef CONFIG_PIN_MEMORY +vm_fault_t do_anon_huge_page_remap(struct vm_area_struct *vma, unsigned long address, + pmd_t *pmd, struct page *page) +{ + gfp_t gfp; + pgtable_t pgtable; + spinlock_t *ptl; + pmd_t entry; + vm_fault_t ret = 0; + + if (unlikely(anon_vma_prepare(vma))) + return VM_FAULT_OOM; + if (unlikely(khugepaged_enter(vma, vma->vm_flags))) + return VM_FAULT_OOM; + gfp = alloc_hugepage_direct_gfpmask(vma); + prep_transhuge_page(page); + if (mem_cgroup_charge(page, vma->vm_mm, gfp)) { + put_page(page); + count_vm_event(THP_FAULT_FALLBACK); + count_vm_event(THP_FAULT_FALLBACK_CHARGE); + return VM_FAULT_FALLBACK; + } + cgroup_throttle_swaprate(page, gfp); + + pgtable = pte_alloc_one(vma->vm_mm); + if (unlikely(!pgtable)) { + ret = VM_FAULT_OOM; + goto release; + } + __SetPageUptodate(page); + ptl = pmd_lock(vma->vm_mm, pmd); + if (unlikely(!pmd_none(*pmd))) { + goto unlock_release; + } else { + ret = check_stable_address_space(vma->vm_mm); + if (ret) + goto unlock_release; + entry = mk_huge_pmd(page, vma->vm_page_prot); + entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma); + page_add_new_anon_rmap(page, vma, address, true); + lru_cache_add_inactive_or_unevictable(page, vma); + pgtable_trans_huge_deposit(vma->vm_mm, pmd, pgtable); + set_pmd_at(vma->vm_mm, address, pmd, entry); + add_mm_counter(vma->vm_mm, MM_ANONPAGES, HPAGE_PMD_NR); + mm_inc_nr_ptes(vma->vm_mm); + spin_unlock(ptl); + count_vm_event(THP_FAULT_ALLOC); + count_memcg_event_mm(vma->vm_mm, THP_FAULT_ALLOC); + } + + return 0; +unlock_release: + spin_unlock(ptl); +release: + if (pgtable) + pte_free(vma->vm_mm, pgtable); + put_page(page); + return ret; +} +#endif diff --git a/mm/memory.c b/mm/memory.c index 50632c4..7b7f1a7 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5248,3 +5248,62 @@ void ptlock_free(struct page *page) kmem_cache_free(page_ptl_cachep, page->ptl); } #endif + +#ifdef CONFIG_PIN_MEMORY +vm_fault_t do_anon_page_remap(struct vm_area_struct *vma, unsigned long address, + pmd_t *pmd, struct page *page) +{ + pte_t entry; + spinlock_t *ptl; + pte_t *pte; + vm_fault_t ret = 0; + + if (pte_alloc(vma->vm_mm, pmd)) + return VM_FAULT_OOM; + + /* See the comment in pte_alloc_one_map() */ + if (unlikely(pmd_trans_unstable(pmd))) + return 0; + + /* Allocate our own private page. */ + if (unlikely(anon_vma_prepare(vma))) + goto oom; + + if (mem_cgroup_charge(page, vma->vm_mm, GFP_KERNEL)) + goto oom_free_page; + cgroup_throttle_swaprate(page, GFP_KERNEL); + + __SetPageUptodate(page); + + entry = mk_pte(page, vma->vm_page_prot); + if (vma->vm_flags & VM_WRITE) + entry = pte_mkwrite(pte_mkdirty(entry)); + pte = pte_offset_map_lock(vma->vm_mm, pmd, address, + &ptl); + if (!pte_none(*pte)) { + ret = VM_FAULT_FALLBACK; + goto release; + } + + ret = check_stable_address_space(vma->vm_mm); + if (ret) + goto release; + inc_mm_counter_fast(vma->vm_mm, MM_ANONPAGES); + page_add_new_anon_rmap(page, vma, address, false); + lru_cache_add_inactive_or_unevictable(page, vma); + + set_pte_at(vma->vm_mm, address, pte, entry); + /* No need to invalidate - it was non-present before */ + update_mmu_cache(vma, address, pte); +unlock: + pte_unmap_unlock(pte, ptl); + return ret; +release: + put_page(page); + goto unlock; +oom_free_page: + put_page(page); +oom: + return VM_FAULT_OOM; +} +#endif diff --git a/mm/pin_mem.c b/mm/pin_mem.c new file mode 100644 index 0000000..0a143b6 --- /dev/null +++ b/mm/pin_mem.c @@ -0,0 +1,950 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2021. Huawei Technologies Co., Ltd. All rights reserved. + * Provide the pin memory method for checkpoint and restore task. + */ +#ifdef CONFIG_PIN_MEMORY +#include <linux/init.h> +#include <linux/slab.h> +#include <linux/time.h> +#include <linux/sched/cputime.h> +#include <linux/tick.h> +#include <linux/mm.h> +#include <linux/pin_mem.h> +#include <linux/idr.h> +#include <linux/page-isolation.h> +#include <linux/sched/mm.h> +#include <linux/ctype.h> +#include <linux/highmem.h> +#include <crypto/sha.h> + +#define MAX_PIN_PID_NUM 128 +static DEFINE_SPINLOCK(page_map_entry_lock); + +struct pin_mem_dump_info *pin_mem_dump_start; +unsigned int pin_pid_num; +static unsigned int *pin_pid_num_addr; +static unsigned long __page_map_entry_start; +static unsigned long page_map_entry_end; +static struct page_map_info *user_space_reserve_start; +static struct page_map_entry *page_map_entry_start; +unsigned int max_pin_pid_num __read_mostly; +unsigned long redirect_space_size; +unsigned long redirect_space_start; +#define DEFAULT_REDIRECT_SPACE_SIZE 0x100000 + +static int __init setup_max_pin_pid_num(char *str) +{ + int ret = 0; + + if (!str) + goto out; + + ret = kstrtouint(str, 10, &max_pin_pid_num); +out: + if (ret) { + pr_warn("Unable to parse max pin pid num.\n"); + } else { + if (max_pin_pid_num > MAX_PIN_PID_NUM) { + max_pin_pid_num = 0; + pr_warn("Input max_pin_pid_num is too large.\n"); + } + } + return ret; +} +early_param("max_pin_pid_num", setup_max_pin_pid_num); + +static int __init setup_redirect_space_size(char *str) +{ + if (!str) + goto out; + + redirect_space_size = memparse(str, NULL); +out: + if (!redirect_space_size) { + pr_warn("Unable to parse redirect space size, use the default value.\n"); + redirect_space_size = DEFAULT_REDIRECT_SPACE_SIZE; + } + return 0; +} +early_param("redirect_space_size", setup_redirect_space_size); + +struct page_map_info *create_page_map_info(int pid) +{ + struct page_map_info *new; + + if (!user_space_reserve_start) + return NULL; + + if (pin_pid_num >= max_pin_pid_num) { + pr_warn("Pin pid num too large than max_pin_pid_num, fail create: %d!", pid); + return NULL; + } + new = (struct page_map_info *)(user_space_reserve_start + pin_pid_num); + new->pid = pid; + new->pme = NULL; + new->entry_num = 0; + new->pid_reserved = false; + new->disable_free_page = false; + (*pin_pid_num_addr)++; + pin_pid_num++; + return new; +} +EXPORT_SYMBOL_GPL(create_page_map_info); + +struct page_map_info *get_page_map_info(int pid) +{ + int i; + + if (!user_space_reserve_start) + return NULL; + + for (i = 0; i < pin_pid_num; i++) { + if (user_space_reserve_start[i].pid == pid) + return &(user_space_reserve_start[i]); + } + return NULL; +} +EXPORT_SYMBOL_GPL(get_page_map_info); + +static struct page *find_head_page(struct page *page) +{ + struct page *p = page; + + while (!PageBuddy(p)) { + if (PageLRU(p)) + return NULL; + p--; + } + return p; +} + +static void spilt_page_area_left(struct zone *zone, struct free_area *area, struct page *page, + unsigned long size, int order) +{ + unsigned long cur_size = 1 << order; + unsigned long total_size = 0; + + while (size && cur_size > size) { + cur_size >>= 1; + order--; + area--; + if (cur_size <= size) { + list_add(&page[total_size].lru, &area->free_list[MIGRATE_MOVABLE]); + atomic_set(&(page[total_size]._mapcount), PAGE_BUDDY_MAPCOUNT_VALUE); + set_page_private(&page[total_size], order); + set_pageblock_migratetype(&page[total_size], MIGRATE_MOVABLE); + area->nr_free++; + total_size += cur_size; + size -= cur_size; + } + } +} + +static void spilt_page_area_right(struct zone *zone, struct free_area *area, struct page *page, + unsigned long size, int order) +{ + unsigned long cur_size = 1 << order; + struct page *right_page, *head_page; + + right_page = page + size; + while (size && cur_size > size) { + cur_size >>= 1; + order--; + area--; + if (cur_size <= size) { + head_page = right_page - cur_size; + list_add(&head_page->lru, &area->free_list[MIGRATE_MOVABLE]); + atomic_set(&(head_page->_mapcount), PAGE_BUDDY_MAPCOUNT_VALUE); + set_page_private(head_page, order); + set_pageblock_migratetype(head_page, MIGRATE_MOVABLE); + area->nr_free++; + size -= cur_size; + right_page = head_page; + } + } +} + +void reserve_page_from_buddy(unsigned long nr_pages, struct page *page) +{ + unsigned int current_order; + struct page *page_end; + struct free_area *area; + struct zone *zone; + struct page *head_page; + + head_page = find_head_page(page); + if (!head_page) { + pr_warn("Find page head fail."); + return; + } + current_order = head_page->private; + page_end = head_page + (1 << current_order); + zone = page_zone(head_page); + area = &(zone->free_area[current_order]); + list_del(&head_page->lru); + atomic_set(&head_page->_mapcount, -1); + set_page_private(head_page, 0); + area->nr_free--; + if (head_page != page) + spilt_page_area_left(zone, area, head_page, + (unsigned long)(page - head_page), current_order); + page = page + nr_pages; + if (page < page_end) { + spilt_page_area_right(zone, area, page, + (unsigned long)(page_end - page), current_order); + } else if (page > page_end) { + pr_warn("Find page end smaller than page."); + } +} + +static inline void reserve_user_normal_pages(struct page *page) +{ + atomic_inc(&page->_refcount); + reserve_page_from_buddy(1, page); +} + +static void init_huge_pmd_pages(struct page *head_page) +{ + int i = 0; + struct page *page = head_page; + + __set_bit(PG_head, &page->flags); + __set_bit(PG_active, &page->flags); + atomic_set(&page->_refcount, 1); + page++; + i++; + page->compound_head = (unsigned long)head_page + 1; + page->compound_dtor = HUGETLB_PAGE_DTOR + 1; + page->compound_order = HPAGE_PMD_ORDER; + page++; + i++; + page->compound_head = (unsigned long)head_page + 1; + i++; + INIT_LIST_HEAD(&(page->deferred_list)); + for (; i < HPAGE_PMD_NR; i++) { + page = head_page + i; + page->compound_head = (unsigned long)head_page + 1; + } +} + +static inline void reserve_user_huge_pmd_pages(struct page *page) +{ + atomic_inc(&page->_refcount); + reserve_page_from_buddy((1 << HPAGE_PMD_ORDER), page); + init_huge_pmd_pages(page); +} + +int reserve_user_map_pages_fail; + +void free_user_map_pages(unsigned int pid_index, unsigned int entry_index, unsigned int page_index) +{ + unsigned int i, j, index, order; + struct page_map_info *pmi; + struct page_map_entry *pme; + struct page *page; + unsigned long phy_addr; + + for (index = 0; index < pid_index; index++) { + pmi = &(user_space_reserve_start[index]); + pme = pmi->pme; + for (i = 0; i < pmi->entry_num; i++) { + for (j = 0; j < pme->nr_pages; j++) { + order = pme->is_huge_page ? HPAGE_PMD_ORDER : 0; + phy_addr = pme->phy_addr_array[j]; + if (phy_addr) { + page = phys_to_page(phy_addr); + if (!(page->flags & PAGE_FLAGS_CHECK_RESERVED)) { + __free_pages(page, order); + pme->phy_addr_array[j] = 0; + } + } + } + pme = (struct page_map_entry *)next_pme(pme); + } + } + pmi = &(user_space_reserve_start[index]); + pme = pmi->pme; + for (i = 0; i < entry_index; i++) { + for (j = 0; j < pme->nr_pages; j++) { + order = pme->is_huge_page ? HPAGE_PMD_ORDER : 0; + phy_addr = pme->phy_addr_array[j]; + if (phy_addr) { + page = phys_to_page(phy_addr); + if (!(page->flags & PAGE_FLAGS_CHECK_RESERVED)) { + __free_pages(page, order); + pme->phy_addr_array[j] = 0; + } + } + } + pme = (struct page_map_entry *)next_pme(pme); + } + for (j = 0; j < page_index; j++) { + order = pme->is_huge_page ? HPAGE_PMD_ORDER : 0; + phy_addr = pme->phy_addr_array[j]; + if (phy_addr) { + page = phys_to_page(phy_addr); + if (!(page->flags & PAGE_FLAGS_CHECK_RESERVED)) { + __free_pages(page, order); + pme->phy_addr_array[j] = 0; + } + } + } +} + +bool check_redirect_end_valid(struct redirect_info *redirect_start, + unsigned long max_redirect_page_num) +{ + unsigned long redirect_end; + + redirect_end = ((unsigned long)(redirect_start + 1) + + max_redirect_page_num * sizeof(unsigned int)); + if (redirect_end > redirect_space_start + redirect_space_size) + return false; + return false; +} + +static void reserve_user_space_map_pages(void) +{ + struct page_map_info *pmi; + struct page_map_entry *pme; + unsigned int i, j, index; + struct page *page; + unsigned long flags; + unsigned long phy_addr; + unsigned long redirect_pages = 0; + struct redirect_info *redirect_start = (struct redirect_info *)redirect_space_start; + + if (!user_space_reserve_start || !redirect_start) + return; + spin_lock_irqsave(&page_map_entry_lock, flags); + for (index = 0; index < pin_pid_num; index++) { + pmi = &(user_space_reserve_start[index]); + pme = pmi->pme; + for (i = 0; i < pmi->entry_num; i++) { + redirect_pages = 0; + if (!check_redirect_end_valid(redirect_start, pme->nr_pages)) + redirect_start = NULL; + for (j = 0; j < pme->nr_pages; j++) { + phy_addr = pme->phy_addr_array[j]; + if (!phy_addr) + continue; + page = phys_to_page(phy_addr); + if (atomic_read(&page->_refcount)) { + if ((page->flags & PAGE_FLAGS_CHECK_RESERVED) + && !pme->redirect_start) + pme->redirect_start = + (unsigned long)redirect_start; + if (redirect_start && + (page->flags & PAGE_FLAGS_CHECK_RESERVED)) { + redirect_start->redirect_index[redirect_pages] = j; + redirect_pages++; + continue; + } else { + reserve_user_map_pages_fail = 1; + pr_warn("Page %pK refcount %d large than zero, no need reserve.\n", + page, atomic_read(&page->_refcount)); + goto free_pages; + } + } + if (!pme->is_huge_page) + reserve_user_normal_pages(page); + else + reserve_user_huge_pmd_pages(page); + } + pme = (struct page_map_entry *)next_pme(pme); + if (redirect_pages && redirect_start) { + redirect_start->redirect_pages = redirect_pages; + redirect_start = (struct redirect_info *)( + (unsigned long)(redirect_start + 1) + + redirect_start->redirect_pages * sizeof(unsigned int)); + } + } + } + spin_unlock(&page_map_entry_lock); + return; +free_pages: + free_user_map_pages(index, i, j); + spin_unlock(&page_map_entry_lock); +} + + +int calculate_pin_mem_digest(struct pin_mem_dump_info *pmdi, char *digest) +{ + int i; + struct sha256_state sctx; + + if (!digest) + digest = pmdi->sha_digest; + sha256_init(&sctx); + sha256_update(&sctx, (unsigned char *)(&(pmdi->magic)), + sizeof(struct pin_mem_dump_info) - SHA256_DIGEST_SIZE); + for (i = 0; i < pmdi->pin_pid_num; i++) { + sha256_update(&sctx, (unsigned char *)(&(pmdi->pmi_array[i])), + sizeof(struct page_map_info)); + } + sha256_final(&sctx, digest); + return 0; +} + +static int check_sha_digest(struct pin_mem_dump_info *pmdi) +{ + int ret = 0; + char digest[SHA256_DIGEST_SIZE] = {0}; + + ret = calculate_pin_mem_digest(pmdi, digest); + if (ret) { + pr_warn("calculate pin mem digest fail:%d\n", ret); + return ret; + } + if (memcmp(pmdi->sha_digest, digest, SHA256_DIGEST_SIZE)) { + pr_warn("pin mem dump info sha256 digest match error!\n"); + return -EFAULT; + } + return ret; +} + +/* + * The whole page map entry collect process must be Sequentially. + * The user_space_reserve_start points to the first page map info for + * the first dump task. And the page_map_entry_start points to + * the first page map entry of the first dump vma. + */ +static void init_page_map_info(struct pin_mem_dump_info *pmdi, unsigned long map_len) +{ + if (pin_mem_dump_start || !max_pin_pid_num) { + pr_warn("pin page map already init or max_pin_pid_num not set.\n"); + return; + } + if (map_len < sizeof(struct pin_mem_dump_info) + + max_pin_pid_num * sizeof(struct page_map_info) + redirect_space_size) { + pr_warn("pin memory reserved memblock too small.\n"); + return; + } + if ((pmdi->magic != PIN_MEM_DUMP_MAGIC) || (pmdi->pin_pid_num > max_pin_pid_num) || + check_sha_digest(pmdi)) + memset(pmdi, 0, sizeof(struct pin_mem_dump_info)); + pin_mem_dump_start = pmdi; + pin_pid_num = pmdi->pin_pid_num; + pr_info("pin_pid_num: %d\n", pin_pid_num); + pin_pid_num_addr = &(pmdi->pin_pid_num); + user_space_reserve_start = + (struct page_map_info *)pmdi->pmi_array; + page_map_entry_start = + (struct page_map_entry *)(user_space_reserve_start + max_pin_pid_num); + page_map_entry_end = (unsigned long)pmdi + map_len - redirect_space_size; + redirect_space_start = page_map_entry_end; + if (pin_pid_num > 0) + reserve_user_space_map_pages(); +} + +int finish_pin_mem_dump(void) +{ + int ret; + + pin_mem_dump_start->magic = PIN_MEM_DUMP_MAGIC; + memset(pin_mem_dump_start->sha_digest, 0, SHA256_DIGEST_SIZE); + ret = calculate_pin_mem_digest(pin_mem_dump_start, NULL); + if (ret) { + pr_warn("calculate pin mem digest fail:%d\n", ret); + return ret; + } + return ret; +} + +int collect_pmd_huge_pages(struct task_struct *task, + unsigned long start_addr, unsigned long end_addr, struct page_map_entry *pme) +{ + long res; + int index = 0; + unsigned long start = start_addr; + struct page *temp_page; + + while (start < end_addr) { + temp_page = NULL; + res = get_user_pages_remote(task->mm, start, 1, + FOLL_TOUCH | FOLL_GET, &temp_page, NULL, NULL); + if (!res) { + pr_warn("Get huge page for addr(%lx) fail.", start); + return COLLECT_PAGES_FAIL; + } + if (PageHead(temp_page)) { + start += HPAGE_PMD_SIZE; + pme->phy_addr_array[index] = page_to_phys(temp_page); + index++; + } else { + pme->nr_pages = index; + atomic_dec(&((temp_page)->_refcount)); + return COLLECT_PAGES_NEED_CONTINUE; + } + } + pme->nr_pages = index; + return COLLECT_PAGES_FINISH; +} + +int collect_normal_pages(struct task_struct *task, + unsigned long start_addr, unsigned long end_addr, struct page_map_entry *pme) +{ + int res; + unsigned long next; + unsigned long i, nr_pages; + struct page *tmp_page; + unsigned long *phy_addr_array = pme->phy_addr_array; + struct page **page_array = (struct page **)pme->phy_addr_array; + + next = (start_addr & HPAGE_PMD_MASK) + HPAGE_PMD_SIZE; + next = (next > end_addr) ? end_addr : next; + pme->nr_pages = 0; + while (start_addr < next) { + nr_pages = (PAGE_ALIGN(next) - start_addr) / PAGE_SIZE; + res = get_user_pages_remote(task->mm, start_addr, 1, + FOLL_TOUCH | FOLL_GET, &tmp_page, NULL, NULL); + if (!res) { + pr_warn("Get user page of %lx fail.\n", start_addr); + return COLLECT_PAGES_FAIL; + } + if (PageHead(tmp_page)) { + atomic_dec(&(tmp_page->_refcount)); + return COLLECT_PAGES_NEED_CONTINUE; + } + atomic_dec(&(tmp_page->_refcount)); + if (PageTail(tmp_page)) { + start_addr = next; + pme->virt_addr = start_addr; + next = (next + HPAGE_PMD_SIZE) > end_addr ? + end_addr : (next + HPAGE_PMD_SIZE); + continue; + } + res = get_user_pages_remote(task->mm, start_addr, nr_pages, + FOLL_TOUCH | FOLL_GET, page_array, NULL, NULL); + if (!res) { + pr_warn("Get user pages of %lx fail.\n", start_addr); + return COLLECT_PAGES_FAIL; + } + for (i = 0; i < nr_pages; i++) + phy_addr_array[i] = page_to_phys(page_array[i]); + pme->nr_pages += nr_pages; + page_array += nr_pages; + phy_addr_array += nr_pages; + start_addr = next; + next = (next + HPAGE_PMD_SIZE) > end_addr ? end_addr : (next + HPAGE_PMD_SIZE); + } + return COLLECT_PAGES_FINISH; +} + +/* Users make sure that the pin memory belongs to anonymous vma. */ +int pin_mem_area(struct task_struct *task, struct mm_struct *mm, + unsigned long start_addr, unsigned long end_addr) +{ + int pid, ret; + int is_huge_page = false; + unsigned int page_size; + unsigned long nr_pages, flags; + struct page_map_entry *pme; + struct page_map_info *pmi; + struct vm_area_struct *vma; + unsigned long i; + struct page *tmp_page; + + if (!page_map_entry_start + || !task || !mm + || start_addr >= end_addr) + return -EFAULT; + + pid = task->pid; + spin_lock_irqsave(&page_map_entry_lock, flags); + nr_pages = ((end_addr - start_addr) / PAGE_SIZE); + if ((unsigned long)page_map_entry_start + nr_pages * sizeof(struct page *) >= + page_map_entry_end) { + pr_warn("Page map entry use up!\n"); + ret = -EFAULT; + goto finish; + } + vma = find_extend_vma(mm, start_addr); + if (!vma) { + pr_warn("Find no match vma!\n"); + ret = -EFAULT; + goto finish; + } + if (start_addr == (start_addr & HPAGE_PMD_MASK) && + transparent_hugepage_enabled(vma)) { + page_size = HPAGE_PMD_SIZE; + is_huge_page = true; + } else { + page_size = PAGE_SIZE; + } + pme = page_map_entry_start; + pme->virt_addr = start_addr; + pme->redirect_start = 0; + pme->is_huge_page = is_huge_page; + memset(pme->phy_addr_array, 0, nr_pages * sizeof(unsigned long)); + down_write(&mm->mmap_lock); + if (!is_huge_page) { + ret = collect_normal_pages(task, start_addr, end_addr, pme); + if (ret != COLLECT_PAGES_FAIL && !pme->nr_pages) { + if (ret == COLLECT_PAGES_FINISH) { + ret = 0; + up_write(&mm->mmap_lock); + goto finish; + } + pme->is_huge_page = true; + page_size = HPAGE_PMD_SIZE; + ret = collect_pmd_huge_pages(task, pme->virt_addr, end_addr, pme); + } + } else { + ret = collect_pmd_huge_pages(task, start_addr, end_addr, pme); + if (ret != COLLECT_PAGES_FAIL && !pme->nr_pages) { + if (ret == COLLECT_PAGES_FINISH) { + ret = 0; + up_write(&mm->mmap_lock); + goto finish; + } + pme->is_huge_page = false; + page_size = PAGE_SIZE; + ret = collect_normal_pages(task, pme->virt_addr, end_addr, pme); + } + } + up_write(&mm->mmap_lock); + if (ret == COLLECT_PAGES_FAIL) { + ret = -EFAULT; + goto finish; + } + + /* check for zero pages */ + for (i = 0; i < pme->nr_pages; i++) { + tmp_page = phys_to_page(pme->phy_addr_array[i]); + if (!pme->is_huge_page) { + if (page_to_pfn(tmp_page) == my_zero_pfn(pme->virt_addr + i * PAGE_SIZE)) + pme->phy_addr_array[i] = 0; + } else if (is_huge_zero_page(tmp_page)) + pme->phy_addr_array[i] = 0; + } + + page_map_entry_start = (struct page_map_entry *)(next_pme(pme)); + pmi = get_page_map_info(pid); + if (!pmi) + pmi = create_page_map_info(pid); + if (!pmi) { + pr_warn("Create page map info fail for pid: %d!\n", pid); + ret = -EFAULT; + goto finish; + } + if (!pmi->pme) + pmi->pme = pme; + pmi->entry_num++; + spin_unlock_irqrestore(&page_map_entry_lock, flags); + if (ret == COLLECT_PAGES_NEED_CONTINUE) + ret = pin_mem_area(task, mm, pme->virt_addr + pme->nr_pages * page_size, end_addr); + return ret; +finish: + spin_unlock_irqrestore(&page_map_entry_lock, flags); + return ret; +} +EXPORT_SYMBOL_GPL(pin_mem_area); + +vm_fault_t remap_normal_pages(struct mm_struct *mm, struct vm_area_struct *vma, + struct page_map_entry *pme) +{ + int ret; + unsigned int j, i; + pgd_t *pgd; + p4d_t *p4d; + pmd_t *pmd; + pud_t *pud; + struct page *page, *new; + unsigned long address; + unsigned long phy_addr; + unsigned int redirect_pages = 0; + struct redirect_info *redirect_start; + + redirect_start = (struct redirect_info *)pme->redirect_start; + for (j = 0; j < pme->nr_pages; j++) { + address = pme->virt_addr + j * PAGE_SIZE; + phy_addr = pme->phy_addr_array[j]; + if (!phy_addr) + continue; + page = phys_to_page(phy_addr); + if (page_to_pfn(page) == my_zero_pfn(address)) { + pme->phy_addr_array[j] = 0; + continue; + } + pme->phy_addr_array[j] = 0; + if (redirect_start && (redirect_pages < redirect_start->redirect_pages) && + (j == redirect_start->redirect_index[redirect_pages])) { + new = alloc_zeroed_user_highpage_movable(vma, address); + if (!new) { + pr_warn("Redirect alloc page fail\n"); + continue; + } + copy_page(page_to_virt(new), phys_to_virt(phy_addr)); + page = new; + redirect_pages++; + } + page->mapping = NULL; + pgd = pgd_offset(mm, address); + p4d = p4d_alloc(mm, pgd, address); + if (!p4d) { + ret = VM_FAULT_OOM; + goto free; + } + pud = pud_alloc(mm, p4d, address); + if (!pud) { + ret = VM_FAULT_OOM; + goto free; + } + pmd = pmd_alloc(mm, pud, address); + if (!pmd) { + ret = VM_FAULT_OOM; + goto free; + } + ret = do_anon_page_remap(vma, address, pmd, page); + if (ret) + goto free; + } + return 0; +free: + for (i = j; i < pme->nr_pages; i++) { + phy_addr = pme->phy_addr_array[i]; + if (phy_addr) { + __free_page(phys_to_page(phy_addr)); + pme->phy_addr_array[i] = 0; + } + } + return ret; +} + +static inline gfp_t get_hugepage_gfpmask(struct vm_area_struct *vma) +{ + const bool vma_madvised = !!(vma->vm_flags & VM_HUGEPAGE); + + if (test_bit(TRANSPARENT_HUGEPAGE_DEFRAG_DIRECT_FLAG, &transparent_hugepage_flags)) + return GFP_TRANSHUGE | (vma_madvised ? 0 : __GFP_NORETRY); + if (test_bit(TRANSPARENT_HUGEPAGE_DEFRAG_KSWAPD_FLAG, &transparent_hugepage_flags)) + return GFP_TRANSHUGE_LIGHT | __GFP_KSWAPD_RECLAIM; + if (test_bit(TRANSPARENT_HUGEPAGE_DEFRAG_KSWAPD_OR_MADV_FLAG, &transparent_hugepage_flags)) + return GFP_TRANSHUGE_LIGHT | (vma_madvised ? __GFP_DIRECT_RECLAIM : + __GFP_KSWAPD_RECLAIM); + if (test_bit(TRANSPARENT_HUGEPAGE_DEFRAG_REQ_MADV_FLAG, &transparent_hugepage_flags)) + return GFP_TRANSHUGE_LIGHT | (vma_madvised ? __GFP_DIRECT_RECLAIM : + 0); + return GFP_TRANSHUGE_LIGHT; +} + +vm_fault_t remap_huge_pmd_pages(struct mm_struct *mm, struct vm_area_struct *vma, + struct page_map_entry *pme) +{ + int ret; + unsigned int j, i; + pgd_t *pgd; + p4d_t *p4d; + pmd_t *pmd; + pud_t *pud; + gfp_t gfp; + struct page *page, *new; + unsigned long address; + unsigned long phy_addr; + unsigned int redirect_pages = 0; + struct redirect_info *redirect_start; + + redirect_start = (struct redirect_info *)pme->redirect_start; + for (j = 0; j < pme->nr_pages; j++) { + address = pme->virt_addr + j * HPAGE_PMD_SIZE; + phy_addr = pme->phy_addr_array[j]; + if (!phy_addr) + continue; + page = phys_to_page(phy_addr); + if (is_huge_zero_page(page)) { + pme->phy_addr_array[j] = 0; + continue; + } + pme->phy_addr_array[j] = 0; + if (redirect_start && (redirect_pages < redirect_start->redirect_pages) && + (j == redirect_start->redirect_index[redirect_pages])) { + gfp = get_hugepage_gfpmask(vma); + new = alloc_hugepage_vma(gfp, vma, address, HPAGE_PMD_ORDER); + if (!new) { + pr_warn("Redirect alloc huge page fail\n"); + continue; + } + memcpy(page_to_virt(new), phys_to_virt(phy_addr), HPAGE_PMD_SIZE); + page = new; + redirect_pages++; + } + pgd = pgd_offset(mm, address); + p4d = p4d_alloc(mm, pgd, address); + if (!p4d) { + ret = VM_FAULT_OOM; + goto free; + } + pud = pud_alloc(mm, p4d, address); + if (!pud) { + ret = VM_FAULT_OOM; + goto free; + } + pmd = pmd_alloc(mm, pud, address); + if (!pmd) { + ret = VM_FAULT_OOM; + goto free; + } + ret = do_anon_huge_page_remap(vma, address, pmd, page); + if (ret) + goto free; + } + return 0; +free: + for (i = j; i < pme->nr_pages; i++) { + phy_addr = pme->phy_addr_array[i]; + if (phy_addr) { + page = phys_to_page(phy_addr); + if (!(page->flags & PAGE_FLAGS_CHECK_RESERVED)) { + __free_pages(page, HPAGE_PMD_ORDER); + pme->phy_addr_array[i] = 0; + } + } + } + return ret; +} + +static void free_unmap_pages(struct page_map_info *pmi, + struct page_map_entry *pme, + unsigned int index) +{ + unsigned int i, j; + unsigned long phy_addr; + unsigned int order; + struct page *page; + + pme = (struct page_map_entry *)(next_pme(pme)); + for (i = index; i < pmi->entry_num; i++) { + for (j = 0; j < pme->nr_pages; j++) { + phy_addr = pme->phy_addr_array[i]; + if (phy_addr) { + page = phys_to_page(phy_addr); + order = pme->is_huge_page ? HPAGE_PMD_ORDER : 0; + if (!(page->flags & PAGE_FLAGS_CHECK_RESERVED)) { + __free_pages(page, order); + pme->phy_addr_array[i] = 0; + } + } + } + pme = (struct page_map_entry *)(next_pme(pme)); + } +} + +vm_fault_t do_mem_remap(int pid, struct mm_struct *mm) +{ + unsigned int i = 0; + vm_fault_t ret = 0; + struct vm_area_struct *vma; + struct page_map_info *pmi; + struct page_map_entry *pme; + unsigned long flags; + + if (reserve_user_map_pages_fail) + return -EFAULT; + pmi = get_page_map_info(pid); + if (!pmi) + return -EFAULT; + + spin_lock_irqsave(&page_map_entry_lock, flags); + pmi->disable_free_page = true; + spin_unlock(&page_map_entry_lock); + down_write(&mm->mmap_lock); + pme = pmi->pme; + vma = mm->mmap; + while ((i < pmi->entry_num) && (vma != NULL)) { + if (pme->virt_addr >= vma->vm_start && pme->virt_addr < vma->vm_end) { + i++; + if (!vma_is_anonymous(vma)) { + pme = (struct page_map_entry *)(next_pme(pme)); + continue; + } + if (!pme->is_huge_page) { + ret = remap_normal_pages(mm, vma, pme); + if (ret < 0) + goto free; + } else { + ret = remap_huge_pmd_pages(mm, vma, pme); + if (ret < 0) + goto free; + } + pme = (struct page_map_entry *)(next_pme(pme)); + } else { + vma = vma->vm_next; + } + } + up_write(&mm->mmap_lock); + return 0; +free: + free_unmap_pages(pmi, pme, i); + up_write(&mm->mmap_lock); + return ret; +} +EXPORT_SYMBOL_GPL(do_mem_remap); + +#if defined(CONFIG_ARM64) +void init_reserve_page_map(unsigned long map_addr, unsigned long map_size) +{ + void *addr; + + if (!map_addr || !map_size) + return; + addr = phys_to_virt(map_addr); + init_page_map_info((struct pin_mem_dump_info *)addr, map_size); +} +#else +void init_reserve_page_map(unsigned long map_addr, unsigned long map_size) +{ +} +#endif + +static void free_all_reserved_pages(void) +{ + unsigned int i, j, index, order; + struct page_map_info *pmi; + struct page_map_entry *pme; + struct page *page; + unsigned long phy_addr; + + if (!user_space_reserve_start || reserve_user_map_pages_fail) + return; + + for (index = 0; index < pin_pid_num; index++) { + pmi = &(user_space_reserve_start[index]); + if (pmi->disable_free_page) + continue; + pme = pmi->pme; + for (i = 0; i < pmi->entry_num; i++) { + for (j = 0; j < pme->nr_pages; j++) { + order = pme->is_huge_page ? HPAGE_PMD_ORDER : 0; + phy_addr = pme->phy_addr_array[j]; + if (phy_addr) { + page = phys_to_page(phy_addr); + if (!(page->flags & PAGE_FLAGS_CHECK_RESERVED)) { + __free_pages(page, order); + pme->phy_addr_array[j] = 0; + } + } + } + pme = (struct page_map_entry *)next_pme(pme); + } + } +} + +/* Clear all pin memory record. */ +void clear_pin_memory_record(void) +{ + unsigned long flags; + + spin_lock_irqsave(&page_map_entry_lock, flags); + free_all_reserved_pages(); + if (pin_pid_num_addr) { + *pin_pid_num_addr = 0; + pin_pid_num = 0; + page_map_entry_start = (struct page_map_entry *)__page_map_entry_start; + } + spin_unlock(&page_map_entry_lock); +} +EXPORT_SYMBOL_GPL(clear_pin_memory_record); + +#endif /* CONFIG_PIN_MEMORY */ -- 2.9.5

2 1

申请加入linux内核组
by 邓伟进 02 Mar '21

02 Mar '21

尊敬的linux内核组各位老师同学：你们好！我是北京拓林思软件有限公司的员工邓伟进。我对linux内核进程管理，文件系统和设备驱动有一定理解。熟悉bpf。目前正努力致力于研究学习bpf和linux内核进程管理，内存管理和文件系统。热爱操作系统和编程。严谨，并喜欢刨根究底的探究问题。这次在陈琪德老师鼓励下申请加入linux内核组，还望通过！非常感谢，我会争取能为社区做一些力所能及的工作。期待您的回复！邓伟进 2021.3.2

2 1

[PATCH OLK-5.10] arm64: enabled e820_pmem and add memmap parameter
by Zhuling 02 Mar '21

02 Mar '21

From: zhuling <zhuling8(a)huawei.com> hulk inclusion category: feature bugzilla: 48159 CVE: NA Enabled e820_pmem in arm64: Use memmap=nn[KMG]!ss[KMG] reserve memory for persistent storage when the kernel restart or update. the data in PMEM will not be lost and can be loaded faster.this is a general features. if you want use this features, you need do as follows: 1.reserve memory: add memmap to reserve memory in grub.cfg memmap=nn[KMG]!ss[KMG] exp:memmap=100K!0x1a0000000. 2.insmod nd_e820.ko: modprobe nd_e820. 3.check pmem device in /dev exp: /dev/pmem0. Signed-off-by: zhuling <zhuling8(a)huawei.com> --- arch/arm64/Kconfig | 24 ++++++++++ arch/arm64/kernel/Makefile | 1 + arch/arm64/kernel/pmem.c | 35 ++++++++++++++ arch/arm64/kernel/setup.c | 6 +++ arch/arm64/mm/init.c | 98 ++++++++++++++++++++++++++++++++++++++ drivers/nvdimm/Makefile | 1 + include/linux/mm.h | 4 ++ 7 files changed, 169 insertions(+) create mode 100644 arch/arm64/kernel/pmem.c diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index b9c56543c..f1e05d9d2 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -1141,6 +1141,30 @@ config XEN_DOM0 def_bool y depends on XEN +config ARM64_PMEM_LEGACY_DEVICE + bool + +config ARM64_PMEM_RESERVE + bool "reserve memory for persistent storage" + default y + help + Use memmap=nn[KMG]!ss[KMG](memmap=100K!0x1a0000000) reserve memory for + persistent storage + + Say y here to enable this feature + +config ARM64_PMEM_LEGACY + tristate "create persistent storage" + depends on ARM64_PMEM_RESERVE + depends on BLK_DEV + select ARM64_PMEM_LEGACY_DEVICE + select LIBNVDIMM + help + Use reserved memory for persistent storage when the kernel restart + or update. the data in PMEM will not be lost and can be loaded faster. + + Say y if unsure. + config XEN bool "Xen guest support on ARM64" depends on ARM64 && OF diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile index 2621d5c2b..c363639b8 100644 --- a/arch/arm64/kernel/Makefile +++ b/arch/arm64/kernel/Makefile @@ -63,6 +63,7 @@ obj-$(CONFIG_ARM_SDE_INTERFACE) += sdei.o obj-$(CONFIG_ARM64_PTR_AUTH) += pointer_auth.o obj-$(CONFIG_SHADOW_CALL_STACK) += scs.o obj-$(CONFIG_ARM64_MTE) += mte.o +obj-$(CONFIG_ARM64_PMEM_LEGACY_DEVICE) += pmem.o obj-y += vdso/ probes/ obj-$(CONFIG_COMPAT_VDSO) += vdso32/ diff --git a/arch/arm64/kernel/pmem.c b/arch/arm64/kernel/pmem.c new file mode 100644 index 000000000..16eaf706f --- /dev/null +++ b/arch/arm64/kernel/pmem.c @@ -0,0 +1,35 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright(c) 2021 Huawei Technologies Co., Ltd + * + * Derived from x86 and arm64 implement PMEM. + */ +#include <linux/platform_device.h> +#include <linux/init.h> +#include <linux/ioport.h> +#include <linux/module.h> + +static int found(struct resource *res, void *data) +{ + return 1; +} + +static int __init register_e820_pmem(void) +{ + struct platform_device *pdev; + int rc; + + rc = walk_iomem_res_desc(IORES_DESC_PERSISTENT_MEMORY_LEGACY, + IORESOURCE_MEM, 0, -1, NULL, found); + if (rc <= 0) + return 0; + + /* + * See drivers/nvdimm/e820.c for the implementation, this is + * simply here to trigger the module to load on demand. + */ + pdev = platform_device_alloc("e820_pmem", -1); + + return platform_device_add(pdev); +} +device_initcall(register_e820_pmem); diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c index 6aff30de8..7f506036d 100644 --- a/arch/arm64/kernel/setup.c +++ b/arch/arm64/kernel/setup.c @@ -255,6 +255,12 @@ static void __init request_standard_resources(void) request_resource(res, &crashk_res); #endif } + +#ifdef CONFIG_ARM64_PMEM_RESERVE + if (pmem_res.end && pmem_res.start) + request_resource(&iomem_resource, &pmem_res); +#endif + } static int __init reserve_memblock_reserved_regions(void) diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c index 794f992cb..e4dc19145 100644 --- a/arch/arm64/mm/init.c +++ b/arch/arm64/mm/init.c @@ -63,6 +63,18 @@ EXPORT_SYMBOL(memstart_addr); phys_addr_t arm64_dma_phys_limit __ro_after_init; phys_addr_t arm64_dma32_phys_limit __ro_after_init; +static unsigned long long pmem_size, pmem_start; + +#ifdef CONFIG_ARM64_PMEM_RESERVE +struct resource pmem_res = { + .name = "Persistent Memory (legacy)", + .start = 0, + .end = 0, + .flags = IORESOURCE_MEM, + .desc = IORES_DESC_PERSISTENT_MEMORY_LEGACY +}; +#endif + #ifndef CONFIG_KEXEC_CORE static void __init reserve_crashkernel(void) { @@ -236,6 +248,88 @@ static void __init fdt_enforce_memory_region(void) memblock_add(usable_rgns[1].base, usable_rgns[1].size); } +static int __init is_mem_valid(unsigned long long mem_size, unsigned long long mem_start) +{ + if (!memblock_is_region_memory(mem_start, mem_size)) { + pr_warn("cannot reserve mem: region is not memory!\n"); + return -EINVAL; + } + + if (memblock_is_region_reserved(mem_start, mem_size)) { + pr_warn("cannot reserve mem: region overlaps reserved memory!\n"); + return -EINVAL; + } + + if (!IS_ALIGNED(mem_start, SZ_2M)) { + pr_warn("cannot reserve mem: base address is not 2MB aligned!\n"); + return -EINVAL; + } + + return 0; +} + +static int __init parse_memmap_one(char *p) +{ + char *oldp; + phys_addr_t start_at, mem_size; + int ret; + + if (!p) + return -EINVAL; + + oldp = p; + mem_size = memparse(p, &p); + if (p == oldp) + return -EINVAL; + + if (!mem_size) + return -EINVAL; + + mem_size = PAGE_ALIGN(mem_size); + + if (*p == '!') { + start_at = memparse(p+1, &p); + + if (is_mem_valid(mem_size, start_at) != 0) + return -EINVAL; + + pr_info("pmem reserved: 0x%016llx - 0x%016llx (%lld MB)\n", + start_at, start_at + mem_size, mem_size >> 20); + pmem_start = start_at; + pmem_size = mem_size; + } else + pr_info("Unrecognized memmap option, please check the parameter.\n"); + + return *p == '\0' ? 0 : -EINVAL; +} + +static int __init parse_memmap_opt(char *str) +{ + while (str) { + char *k = strchr(str, ','); + + if (k) + *k++ = 0; + + parse_memmap_one(str); + str = k; + } + + return 0; +} +early_param("memmap", parse_memmap_opt); + +#ifdef CONFIG_ARM64_PMEM_RESERVE +static void __init reserve_pmem(void) +{ + memblock_remove(pmem_start, pmem_size); + pr_info("pmem reserved: 0x%016llx - 0x%016llx (%lld MB)\n", + pmem_start, pmem_start + pmem_size, pmem_size >> 20); + pmem_res.start = pmem_start; + pmem_res.end = pmem_start + pmem_size - 1; +} +#endif + void __init arm64_memblock_init(void) { const s64 linear_region_size = BIT(vabits_actual - 1); @@ -359,6 +453,10 @@ void __init arm64_memblock_init(void) reserve_elfcorehdr(); +#ifdef CONFIG_ARM64_PMEM_RESERVE + reserve_pmem(); +#endif + high_memory = __va(memblock_end_of_DRAM() - 1) + 1; dma_contiguous_reserve(arm64_dma32_phys_limit); diff --git a/drivers/nvdimm/Makefile b/drivers/nvdimm/Makefile index 29203f3d3..b97760e9f 100644 --- a/drivers/nvdimm/Makefile +++ b/drivers/nvdimm/Makefile @@ -4,6 +4,7 @@ obj-$(CONFIG_BLK_DEV_PMEM) += nd_pmem.o obj-$(CONFIG_ND_BTT) += nd_btt.o obj-$(CONFIG_ND_BLK) += nd_blk.o obj-$(CONFIG_X86_PMEM_LEGACY) += nd_e820.o +obj-$(CONFIG_ARM64_PMEM_LEGACY) += nd_e820.o obj-$(CONFIG_OF_PMEM) += of_pmem.o obj-$(CONFIG_VIRTIO_PMEM) += virtio_pmem.o nd_virtio.o diff --git a/include/linux/mm.h b/include/linux/mm.h index cd5c31372..a5e50495e 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -45,6 +45,10 @@ extern int sysctl_page_lock_unfairness; void init_mm_internals(void); +#ifdef CONFIG_ARM64_PMEM_RESERVE +extern struct resource pmem_res; +#endif + #ifndef CONFIG_NEED_MULTIPLE_NODES /* Don't use mapnrs, do it properly */ extern unsigned long max_mapnr; -- 2.19.1

2 1

[PATCH openEuler-21.03 2/2] pid: add pid reserve method for checkpoint and recover
by hejingxian 02 Mar '21

02 Mar '21

From: Jingxian He <hejingxian(a)huawei.com> Date: Mon, 1 Mar 2021 17:44:59 +0800 Subject: [PATCH openEuler-21.03 2/2] pid: add pid reserve method for checkpoint and recover hulk inclusion category: feature bugzilla: 48159 CVE: N/A We record the pid of dump tasks in the reserved memory, and reserve the pids before init task start. In the recover process, free the reserved pids and realloc them for use. Signed-off-by: Jingxian He <hejingxian(a)huawei.com> Reviewed-by: Wenliang He <hewenliang4(a)huawei.com> Reviewed-by: Jing Xiangfeng <jingxiangfeng(a)huawei.com> --- arch/arm64/configs/openeuler_defconfig | 1 + include/linux/pin_mem.h | 6 ++++ kernel/pid.c | 10 +++++++ mm/Kconfig | 10 +++++++ mm/pin_mem.c | 51 ++++++++++++++++++++++++++++++++++ 5 files changed, 78 insertions(+) diff --git a/arch/arm64/configs/openeuler_defconfig b/arch/arm64/configs/openeuler_defconfig index 76fda68..de6db02 100644 --- a/arch/arm64/configs/openeuler_defconfig +++ b/arch/arm64/configs/openeuler_defconfig @@ -1037,6 +1037,7 @@ CONFIG_FRAME_VECTOR=y # CONFIG_READ_ONLY_THP_FOR_FS is not set CONFIG_ARCH_HAS_PTE_SPECIAL=y CONFIG_PIN_MEMORY=y +CONFIG_PID_RESERVE=y # end of Memory Management options CONFIG_NET=y diff --git a/include/linux/pin_mem.h b/include/linux/pin_mem.h index bc8b03e..a9fe2ef 100644 --- a/include/linux/pin_mem.h +++ b/include/linux/pin_mem.h @@ -74,5 +74,11 @@ extern struct resource pin_memory_resource; #endif extern void init_reserve_page_map(unsigned long map_addr, unsigned long map_size); +#ifdef CONFIG_PID_RESERVE +extern bool is_need_reserve_pids(void); +extern void free_reserved_pid(struct idr *idr, int pid); +extern void reserve_pids(struct idr *idr, int pid_max); +#endif + #endif /* CONFIG_PIN_MEMORY */ #endif /* _LINUX_PIN_MEMORY_H */ diff --git a/kernel/pid.c b/kernel/pid.c index 4856818..32ab9ef 100644 --- a/kernel/pid.c +++ b/kernel/pid.c @@ -44,6 +44,9 @@ #include <linux/idr.h> #include <net/sock.h> #include <uapi/linux/pidfd.h> +#ifdef CONFIG_PID_RESERVE +#include <linux/pin_mem.h> +#endif struct pid init_struct_pid = { .count = REFCOUNT_INIT(1), @@ -209,6 +212,9 @@ struct pid *alloc_pid(struct pid_namespace *ns, pid_t *set_tid, spin_lock_irq(&pidmap_lock); if (tid) { +#ifdef CONFIG_PID_RESERVE + free_reserved_pid(&tmp->idr, tid); +#endif nr = idr_alloc(&tmp->idr, NULL, tid, tid + 1, GFP_ATOMIC); /* @@ -621,6 +627,10 @@ void __init pid_idr_init(void) init_pid_ns.pid_cachep = KMEM_CACHE(pid, SLAB_HWCACHE_ALIGN | SLAB_PANIC | SLAB_ACCOUNT); +#ifdef CONFIG_PID_RESERVE + if (is_need_reserve_pids()) + reserve_pids(&init_pid_ns.idr, pid_max); +#endif } static struct file *__pidfd_fget(struct task_struct *task, int fd) diff --git a/mm/Kconfig b/mm/Kconfig index 930dc13..e27d2c6 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -868,3 +868,13 @@ config PIN_MEMORY the corresponding physical pages mapping info in checkpoint, and remap the physical pages to restore tasks in restore. endmenu + +config PID_RESERVE + bool "Support for reserve pid" + depends on PIN_MEMORY + help + Say y here to enable the pid reserved feature for checkpoint. + and restore. + We record the pid of dump task in the reserve memory, + and reserve the pids before init task start. In restore process, + free the reserved pids and realloc them for use. diff --git a/mm/pin_mem.c b/mm/pin_mem.c index 0a143b6..a040853 100644 --- a/mm/pin_mem.c +++ b/mm/pin_mem.c @@ -947,4 +947,55 @@ void clear_pin_memory_record(void) } EXPORT_SYMBOL_GPL(clear_pin_memory_record); +#ifdef CONFIG_PID_RESERVE +struct idr *reserve_idr; + +/* test if there exist pin memory tasks */ +bool is_need_reserve_pids(void) +{ + return (pin_pid_num > 0); +} + +void free_reserved_pid(struct idr *idr, int pid) +{ + unsigned int index; + struct page_map_info *pmi; + + if (!max_pin_pid_num || idr != reserve_idr) + return; + + for (index = 0; index < pin_pid_num; index++) { + pmi = &(user_space_reserve_start[index]); + if (pmi->pid == pid && pmi->pid_reserved) { + idr_remove(idr, pid); + return; + } + } +} + +/* reserve pids for check point tasks which pinned memory */ +void reserve_pids(struct idr *idr, int pid_max) +{ + int alloc_pid; + unsigned int index; + struct page_map_info *pmi; + + if (!max_pin_pid_num) + return; + reserve_idr = idr; + for (index = 0; index < pin_pid_num; index++) { + pmi = &(user_space_reserve_start[index]); + pmi->pid_reserved = true; + alloc_pid = idr_alloc(idr, NULL, pmi->pid, pid_max, GFP_ATOMIC); + if (alloc_pid != pmi->pid) { + if (alloc_pid > 0) + idr_remove(idr, alloc_pid); + pr_warn("Reserve pid (%d) fail, real pid is %d.\n", alloc_pid, pmi->pid); + pmi->pid_reserved = false; + continue; + } + } +} +#endif /* CONFIG_PID_RESERVE */ + #endif /* CONFIG_PIN_MEMORY */ -- 2.9.5

1 0

[PATCH openEuler-21.03 1/2] mm: add pin memory method for checkpoint add restore
by hejingxian＠huawei.com 02 Mar '21

02 Mar '21

From: Jingxian He <hejingxian(a)huawei.com> hulk inclusion category: feature bugzilla: 48159 CVE: N/A We can use the checkpoint and restore in userspace(criu) method to dump and restore tasks when updating the kernel. Currently, criu needs dump all memory data of tasks to files. When the memory size is very large(larger than 1G), the cost time of the dumping data will be very long(more than 1 min). By pin the memory data of tasks and collect the corresponding physical pages mapping info in checkpoint process, we can remap the physical pages to restore tasks after upgrading the kernel. This pin memory method can restore the task data within one second. The pin memory area info is saved in the reserved memblock, which can keep usable in the kernel update process. The pin memory driver provides the following ioctl command for criu: 1) SET_PIN_MEM_AREA: Set pin memory area, which can be remap to the restore task. 2) CLEAR_PIN_MEM_AREA: Clear the pin memory area info, which enable user reset the pin data. 3) REMAP_PIN_MEM_AREA: Remap the pages of the pin memory to the restore task. Signed-off-by: Jingxian He <hejingxian(a)huawei.com> Reviewed-by: Wenliang He <hewenliang4(a)huawei.com> Reviewed-by: Jing Xiangfeng <jingxiangfeng(a)huawei.com> --- arch/arm64/configs/openeuler_defconfig | 2 + arch/arm64/kernel/setup.c | 9 + arch/arm64/mm/init.c | 60 +++ drivers/char/Kconfig | 6 + drivers/char/Makefile | 1 + drivers/char/pin_memory.c | 208 ++++++++ include/linux/crash_core.h | 5 + include/linux/pin_mem.h | 78 +++ kernel/crash_core.c | 11 + mm/Kconfig | 8 + mm/Makefile | 1 + mm/huge_memory.c | 61 +++ mm/memory.c | 59 ++ mm/pin_mem.c | 950 +++++++++++++++++++++++++++++++++ 14 files changed, 1459 insertions(+) create mode 100644 drivers/char/pin_memory.c create mode 100644 include/linux/pin_mem.h create mode 100644 mm/pin_mem.c diff --git a/arch/arm64/configs/openeuler_defconfig b/arch/arm64/configs/openeuler_defconfig index c5271e7..76fda68 100644 --- a/arch/arm64/configs/openeuler_defconfig +++ b/arch/arm64/configs/openeuler_defconfig @@ -1036,6 +1036,7 @@ CONFIG_FRAME_VECTOR=y # CONFIG_GUP_BENCHMARK is not set # CONFIG_READ_ONLY_THP_FOR_FS is not set CONFIG_ARCH_HAS_PTE_SPECIAL=y +CONFIG_PIN_MEMORY=y # end of Memory Management options CONFIG_NET=y @@ -3282,6 +3283,7 @@ CONFIG_TCG_TIS_ST33ZP24_SPI=y # CONFIG_RANDOM_TRUST_CPU is not set # CONFIG_RANDOM_TRUST_BOOTLOADER is not set +CONFIG_PIN_MEMORY_DEV=m # # I2C support diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c index c1f1fb9..5e282d3 100644 --- a/arch/arm64/kernel/setup.c +++ b/arch/arm64/kernel/setup.c @@ -50,6 +50,9 @@ #include <asm/efi.h> #include <asm/xen/hypervisor.h> #include <asm/mmu_context.h> +#ifdef CONFIG_PIN_MEMORY +#include <linux/pin_mem.h> +#endif static int num_standard_resources; static struct resource *standard_resources; @@ -260,6 +263,12 @@ static void __init request_standard_resources(void) quick_kexec_res.end <= res->end) request_resource(res, &quick_kexec_res); #endif +#ifdef CONFIG_PIN_MEMORY + if (pin_memory_resource.end && + pin_memory_resource.start >= res->start && + pin_memory_resource.end <= res->end) + request_resource(res, &pin_memory_resource); +#endif } } diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c index f3e5a66..8ab5aac 100644 --- a/arch/arm64/mm/init.c +++ b/arch/arm64/mm/init.c @@ -42,6 +42,9 @@ #include <linux/sizes.h> #include <asm/tlb.h> #include <asm/alternative.h> +#ifdef CONFIG_PIN_MEMORY +#include <linux/pin_mem.h> +#endif #define ARM64_ZONE_DMA_BITS 30 @@ -78,6 +81,55 @@ static void __init reserve_crashkernel(void) */ #define MAX_USABLE_RANGES 2 +#ifdef CONFIG_PIN_MEMORY +struct resource pin_memory_resource = { + .name = "Pin memory", + .start = 0, + .end = 0, + .flags = IORESOURCE_MEM, + .desc = IORES_DESC_RESERVED +}; + +static void __init reserve_pin_memory_res(void) +{ + unsigned long long mem_start, mem_len; + int ret; + + ret = parse_pin_memory(boot_command_line, memblock_phys_mem_size(), + &mem_len, &mem_start); + if (ret || !mem_len) + return; + + mem_len = PAGE_ALIGN(mem_len); + + if (!memblock_is_region_memory(mem_start, mem_len)) { + pr_warn("cannot reserve for pin memory: region is not memory!\n"); + return; + } + + if (memblock_is_region_reserved(mem_start, mem_len)) { + pr_warn("cannot reserve for pin memory: region overlaps reserved memory!\n"); + return; + } + + if (!IS_ALIGNED(mem_start, SZ_2M)) { + pr_warn("cannot reserve for pin memory: base address is not 2MB aligned\n"); + return; + } + + memblock_reserve(mem_start, mem_len); + pr_debug("pin memory resource reserved: 0x%016llx - 0x%016llx (%lld MB)\n", + mem_start, mem_start + mem_len, mem_len >> 20); + + pin_memory_resource.start = mem_start; + pin_memory_resource.end = mem_start + mem_len - 1; +} +#else +static void __init reserve_pin_memory_res(void) +{ +} +#endif /* CONFIG_PIN_MEMORY */ + #ifdef CONFIG_CRASH_DUMP static int __init early_init_dt_scan_elfcorehdr(unsigned long node, const char *uname, int depth, void *data) @@ -455,6 +507,8 @@ void __init arm64_memblock_init(void) reserve_park_mem(); #endif + reserve_pin_memory_res(); + reserve_elfcorehdr(); high_memory = __va(memblock_end_of_DRAM() - 1) + 1; @@ -583,6 +637,12 @@ void __init mem_init(void) /* this will put all unused low memory onto the freelists */ memblock_free_all(); +#ifdef CONFIG_PIN_MEMORY + /* pre alloc the pages for pin memory */ + init_reserve_page_map((unsigned long)pin_memory_resource.start, + (unsigned long)(pin_memory_resource.end - pin_memory_resource.start + 1)); +#endif + mem_init_print_info(NULL); /* diff --git a/drivers/char/Kconfig b/drivers/char/Kconfig index d229a2d..fbb94b8 100644 --- a/drivers/char/Kconfig +++ b/drivers/char/Kconfig @@ -496,3 +496,9 @@ config RANDOM_TRUST_BOOTLOADER booloader is trustworthy so it will be added to the kernel's entropy pool. Otherwise, say N here so it will be regarded as device input that only mixes the entropy pool. + +config PIN_MEMORY_DEV + bool "/dev/pinmem character device" + default m + help + pin memory driver diff --git a/drivers/char/Makefile b/drivers/char/Makefile index ffce287..71d76fd 100644 --- a/drivers/char/Makefile +++ b/drivers/char/Makefile @@ -47,3 +47,4 @@ obj-$(CONFIG_PS3_FLASH) += ps3flash.o obj-$(CONFIG_XILLYBUS) += xillybus/ obj-$(CONFIG_POWERNV_OP_PANEL) += powernv-op-panel.o obj-$(CONFIG_ADI) += adi.o +obj-$(CONFIG_PIN_MEMORY_DEV) += pin_memory.o diff --git a/drivers/char/pin_memory.c b/drivers/char/pin_memory.c new file mode 100644 index 0000000..f46e056 --- /dev/null +++ b/drivers/char/pin_memory.c @@ -0,0 +1,208 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright @ Huawei Technologies Co., Ltd. 2020-2020. ALL rights reserved. + * Description: Euler pin memory driver + */ +#include <linux/kernel.h> +#include <linux/module.h> +#include <linux/kprobes.h> +#include <linux/spinlock.h> +#include <linux/workqueue.h> +#include <linux/sched.h> +#include <linux/mm.h> +#include <linux/init.h> +#include <linux/miscdevice.h> +#include <linux/fs.h> +#include <linux/mm_types.h> +#include <linux/processor.h> +#include <uapi/asm-generic/ioctl.h> +#include <uapi/asm-generic/mman-common.h> +#include <uapi/asm/setup.h> +#include <linux/pin_mem.h> +#include <linux/sched/mm.h> + +#define MAX_PIN_MEM_AREA_NUM 16 +struct _pin_mem_area { + unsigned long virt_start; + unsigned long virt_end; +}; + +struct pin_mem_area_set { + unsigned int pid; + unsigned int area_num; + struct _pin_mem_area mem_area[MAX_PIN_MEM_AREA_NUM]; +}; + +#define PIN_MEM_MAGIC 0x59 +#define _SET_PIN_MEM_AREA 1 +#define _CLEAR_PIN_MEM_AREA 2 +#define _REMAP_PIN_MEM_AREA 3 +#define _FINISH_PIN_MEM_DUMP 4 +#define _PIN_MEM_IOC_MAX_NR 4 +#define SET_PIN_MEM_AREA _IOW(PIN_MEM_MAGIC, _SET_PIN_MEM_AREA, struct pin_mem_area_set) +#define CLEAR_PIN_MEM_AREA _IOW(PIN_MEM_MAGIC, _CLEAR_PIN_MEM_AREA, int) +#define REMAP_PIN_MEM_AREA _IOW(PIN_MEM_MAGIC, _REMAP_PIN_MEM_AREA, int) +#define FINISH_PIN_MEM_DUMP _IOW(PIN_MEM_MAGIC, _FINISH_PIN_MEM_DUMP, int) +static int set_pin_mem(struct pin_mem_area_set *pmas) +{ + int i; + int ret = 0; + struct _pin_mem_area *pma; + struct mm_struct *mm; + struct task_struct *task; + struct pid *pid_s; + + pid_s = find_get_pid(pmas->pid); + if (!pid_s) { + pr_warn("Get pid struct fail:%d.\n", pmas->pid); + return -EFAULT; + } + rcu_read_lock(); + task = pid_task(pid_s, PIDTYPE_PID); + if (!task) { + pr_warn("Get task struct fail:%d.\n", pmas->pid); + goto fail; + } + mm = get_task_mm(task); + for (i = 0; i < pmas->area_num; i++) { + pma = &(pmas->mem_area[i]); + ret = pin_mem_area(task, mm, pma->virt_start, pma->virt_end); + if (ret) { + mmput(mm); + goto fail; + } + } + mmput(mm); + rcu_read_unlock(); + put_pid(pid_s); + return ret; + +fail: + rcu_read_unlock(); + put_pid(pid_s); + return -EFAULT; +} + +static int set_pin_mem_area(unsigned long arg) +{ + struct pin_mem_area_set pmas; + void __user *buf = (void __user *)arg; + + if (!access_ok(buf, sizeof(pmas))) + return -EFAULT; + if (copy_from_user(&pmas, buf, sizeof(pmas))) + return -EINVAL; + if (pmas.area_num > MAX_PIN_MEM_AREA_NUM) { + pr_warn("Input area_num is too large.\n"); + return -EINVAL; + } + + return set_pin_mem(&pmas); +} + +static int pin_mem_remap(unsigned long arg) +{ + int pid; + struct task_struct *task; + struct mm_struct *mm; + vm_fault_t ret; + void __user *buf = (void __user *)arg; + struct pid *pid_s; + + if (!access_ok(buf, sizeof(int))) + return -EINVAL; + if (copy_from_user(&pid, buf, sizeof(int))) + return -EINVAL; + + pid_s = find_get_pid(pid); + if (!pid_s) { + pr_warn("Get pid struct fail:%d.\n", pid); + return -EINVAL; + } + rcu_read_lock(); + task = pid_task(pid_s, PIDTYPE_PID); + if (!task) { + pr_warn("Get task struct fail:%d.\n", pid); + goto fault; + } + mm = get_task_mm(task); + ret = do_mem_remap(pid, mm); + if (ret) { + pr_warn("Handle pin memory remap fail.\n"); + mmput(mm); + goto fault; + } + mmput(mm); + rcu_read_unlock(); + put_pid(pid_s); + return 0; + +fault: + rcu_read_unlock(); + put_pid(pid_s); + return -EFAULT; +} + +static long pin_memory_ioctl(struct file *file, unsigned int cmd, unsigned long arg) +{ + long ret = 0; + + if (_IOC_TYPE(cmd) != PIN_MEM_MAGIC) + return -EINVAL; + if (_IOC_NR(cmd) > _PIN_MEM_IOC_MAX_NR) + return -EINVAL; + + switch (cmd) { + case SET_PIN_MEM_AREA: + ret = set_pin_mem_area(arg); + break; + case CLEAR_PIN_MEM_AREA: + clear_pin_memory_record(); + break; + case REMAP_PIN_MEM_AREA: + ret = pin_mem_remap(arg); + break; + case FINISH_PIN_MEM_DUMP: + ret = finish_pin_mem_dump(); + break; + default: + return -EINVAL; + } + return ret; +} + +static const struct file_operations pin_memory_fops = { + .owner = THIS_MODULE, + .unlocked_ioctl = pin_memory_ioctl, + .compat_ioctl = pin_memory_ioctl, +}; + +static struct miscdevice pin_memory_miscdev = { + .minor = MISC_DYNAMIC_MINOR, + .name = "pinmem", + .fops = &pin_memory_fops, +}; + +static int pin_memory_init(void) +{ + int err = misc_register(&pin_memory_miscdev); + + if (!err) + pr_info("pin_memory init\n"); + else + pr_warn("pin_memory init failed!\n"); + return err; +} + +static void pin_memory_exit(void) +{ + misc_deregister(&pin_memory_miscdev); + pr_info("pin_memory ko exists!\n"); +} + +module_init(pin_memory_init); +module_exit(pin_memory_exit); + +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Euler"); +MODULE_DESCRIPTION("pin memory"); diff --git a/include/linux/crash_core.h b/include/linux/crash_core.h index fc0ef33..30f0df3 100644 --- a/include/linux/crash_core.h +++ b/include/linux/crash_core.h @@ -87,4 +87,9 @@ int parse_crashkernel_high(char *cmdline, unsigned long long system_ram, int parse_crashkernel_low(char *cmdline, unsigned long long system_ram, unsigned long long *crash_size, unsigned long long *crash_base); +#ifdef CONFIG_PIN_MEMORY +int __init parse_pin_memory(char *cmdline, unsigned long long system_ram, + unsigned long long *pin_size, unsigned long long *pin_base); +#endif + #endif /* LINUX_CRASH_CORE_H */ diff --git a/include/linux/pin_mem.h b/include/linux/pin_mem.h new file mode 100644 index 0000000..bc8b03e --- /dev/null +++ b/include/linux/pin_mem.h @@ -0,0 +1,78 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (C) 2020. Huawei Technologies Co., Ltd. All rights reserved. + * Provide the pin memory method for check point and restore task. + */ +#ifndef _LINUX_PIN_MEMORY_H +#define _LINUX_PIN_MEMORY_H + +#ifdef CONFIG_PIN_MEMORY +#include <linux/errno.h> +#include <linux/mm_types.h> +#include <linux/err.h> +#ifdef CONFIG_ARM64 +#include <linux/ioport.h> +#endif + +#define PAGE_BUDDY_MAPCOUNT_VALUE (~PG_buddy) + +#define COLLECT_PAGES_FINISH 0 +#define COLLECT_PAGES_NEED_CONTINUE 1 +#define COLLECT_PAGES_FAIL -1 + +#define COMPOUND_PAD_MASK 0xffffffff +#define COMPOUND_PAD_START 0x88 +#define COMPOUND_PAD_DELTA 0x40 +#define LIST_POISON4 0xdead000000000400 +#define PAGE_FLAGS_CHECK_RESERVED (1UL << PG_reserved) +#define SHA256_DIGEST_SIZE 32 +#define next_pme(pme) ((unsigned long *)(pme + 1) + pme->nr_pages) +#define PIN_MEM_DUMP_MAGIC 0xfeab000000001acd +struct page_map_entry { + unsigned long virt_addr; + unsigned int nr_pages; + unsigned int is_huge_page; + unsigned long redirect_start; + unsigned long phy_addr_array[0]; +}; + +struct page_map_info { + int pid; + int pid_reserved; + unsigned int entry_num; + int disable_free_page; + struct page_map_entry *pme; +}; + +struct pin_mem_dump_info { + char sha_digest[SHA256_DIGEST_SIZE]; + unsigned long magic; + unsigned int pin_pid_num; + struct page_map_info pmi_array[0]; +}; + +struct redirect_info { + unsigned int redirect_pages; + unsigned int redirect_index[0]; +}; + +extern struct page_map_info *get_page_map_info(int pid); +extern struct page_map_info *create_page_map_info(int pid); +extern vm_fault_t do_mem_remap(int pid, struct mm_struct *mm); +extern vm_fault_t do_anon_page_remap(struct vm_area_struct *vma, unsigned long address, + pmd_t *pmd, struct page *page); +extern void clear_pin_memory_record(void); +extern int pin_mem_area(struct task_struct *task, struct mm_struct *mm, + unsigned long start_addr, unsigned long end_addr); +extern vm_fault_t do_anon_huge_page_remap(struct vm_area_struct *vma, unsigned long address, + pmd_t *pmd, struct page *page); +extern int finish_pin_mem_dump(void); + +/* reserve space for pin memory*/ +#ifdef CONFIG_ARM64 +extern struct resource pin_memory_resource; +#endif +extern void init_reserve_page_map(unsigned long map_addr, unsigned long map_size); + +#endif /* CONFIG_PIN_MEMORY */ +#endif /* _LINUX_PIN_MEMORY_H */ diff --git a/kernel/crash_core.c b/kernel/crash_core.c index bfed474..2407de3 100644 --- a/kernel/crash_core.c +++ b/kernel/crash_core.c @@ -450,6 +450,17 @@ void __init reserve_crashkernel(void) } #endif /* CONFIG_ARCH_WANT_RESERVE_CRASH_KERNEL */ +#ifdef CONFIG_PIN_MEMORY +int __init parse_pin_memory(char *cmdline, + unsigned long long system_ram, + unsigned long long *pin_size, + unsigned long long *pin_base) +{ + return __parse_crashkernel(cmdline, system_ram, pin_size, pin_base, + "pinmemory=", NULL); +} +#endif + Elf_Word *append_elf_note(Elf_Word *buf, char *name, unsigned int type, void *data, size_t data_len) { diff --git a/mm/Kconfig b/mm/Kconfig index 390165f..930dc13 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -859,4 +859,12 @@ config ARCH_HAS_HUGEPD config MAPPING_DIRTY_HELPERS bool +config PIN_MEMORY + bool "Support for pin memory" + depends on CHECKPOINT_RESTORE + help + Say y here to enable the pin memory feature for checkpoint + and restore. We can pin the memory data of tasks and collect + the corresponding physical pages mapping info in checkpoint, + and remap the physical pages to restore tasks in restore. endmenu diff --git a/mm/Makefile b/mm/Makefile index d73aed0..4963827 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -120,3 +120,4 @@ obj-$(CONFIG_MEMFD_CREATE) += memfd.o obj-$(CONFIG_MAPPING_DIRTY_HELPERS) += mapping_dirty_helpers.o obj-$(CONFIG_PTDUMP_CORE) += ptdump.o obj-$(CONFIG_PAGE_REPORTING) += page_reporting.o +obj-$(CONFIG_PIN_MEMORY) += pin_mem.o diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 0bc4a2c..8a11d30 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2996,3 +2996,64 @@ void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new) update_mmu_cache_pmd(vma, address, pvmw->pmd); } #endif + +#ifdef CONFIG_PIN_MEMORY +vm_fault_t do_anon_huge_page_remap(struct vm_area_struct *vma, unsigned long address, + pmd_t *pmd, struct page *page) +{ + gfp_t gfp; + pgtable_t pgtable; + spinlock_t *ptl; + pmd_t entry; + vm_fault_t ret = 0; + + if (unlikely(anon_vma_prepare(vma))) + return VM_FAULT_OOM; + if (unlikely(khugepaged_enter(vma, vma->vm_flags))) + return VM_FAULT_OOM; + gfp = alloc_hugepage_direct_gfpmask(vma); + prep_transhuge_page(page); + if (mem_cgroup_charge(page, vma->vm_mm, gfp)) { + put_page(page); + count_vm_event(THP_FAULT_FALLBACK); + count_vm_event(THP_FAULT_FALLBACK_CHARGE); + return VM_FAULT_FALLBACK; + } + cgroup_throttle_swaprate(page, gfp); + + pgtable = pte_alloc_one(vma->vm_mm); + if (unlikely(!pgtable)) { + ret = VM_FAULT_OOM; + goto release; + } + __SetPageUptodate(page); + ptl = pmd_lock(vma->vm_mm, pmd); + if (unlikely(!pmd_none(*pmd))) { + goto unlock_release; + } else { + ret = check_stable_address_space(vma->vm_mm); + if (ret) + goto unlock_release; + entry = mk_huge_pmd(page, vma->vm_page_prot); + entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma); + page_add_new_anon_rmap(page, vma, address, true); + lru_cache_add_inactive_or_unevictable(page, vma); + pgtable_trans_huge_deposit(vma->vm_mm, pmd, pgtable); + set_pmd_at(vma->vm_mm, address, pmd, entry); + add_mm_counter(vma->vm_mm, MM_ANONPAGES, HPAGE_PMD_NR); + mm_inc_nr_ptes(vma->vm_mm); + spin_unlock(ptl); + count_vm_event(THP_FAULT_ALLOC); + count_memcg_event_mm(vma->vm_mm, THP_FAULT_ALLOC); + } + + return 0; +unlock_release: + spin_unlock(ptl); +release: + if (pgtable) + pte_free(vma->vm_mm, pgtable); + put_page(page); + return ret; +} +#endif diff --git a/mm/memory.c b/mm/memory.c index 50632c4..7b7f1a7 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5248,3 +5248,62 @@ void ptlock_free(struct page *page) kmem_cache_free(page_ptl_cachep, page->ptl); } #endif + +#ifdef CONFIG_PIN_MEMORY +vm_fault_t do_anon_page_remap(struct vm_area_struct *vma, unsigned long address, + pmd_t *pmd, struct page *page) +{ + pte_t entry; + spinlock_t *ptl; + pte_t *pte; + vm_fault_t ret = 0; + + if (pte_alloc(vma->vm_mm, pmd)) + return VM_FAULT_OOM; + + /* See the comment in pte_alloc_one_map() */ + if (unlikely(pmd_trans_unstable(pmd))) + return 0; + + /* Allocate our own private page. */ + if (unlikely(anon_vma_prepare(vma))) + goto oom; + + if (mem_cgroup_charge(page, vma->vm_mm, GFP_KERNEL)) + goto oom_free_page; + cgroup_throttle_swaprate(page, GFP_KERNEL); + + __SetPageUptodate(page); + + entry = mk_pte(page, vma->vm_page_prot); + if (vma->vm_flags & VM_WRITE) + entry = pte_mkwrite(pte_mkdirty(entry)); + pte = pte_offset_map_lock(vma->vm_mm, pmd, address, + &ptl); + if (!pte_none(*pte)) { + ret = VM_FAULT_FALLBACK; + goto release; + } + + ret = check_stable_address_space(vma->vm_mm); + if (ret) + goto release; + inc_mm_counter_fast(vma->vm_mm, MM_ANONPAGES); + page_add_new_anon_rmap(page, vma, address, false); + lru_cache_add_inactive_or_unevictable(page, vma); + + set_pte_at(vma->vm_mm, address, pte, entry); + /* No need to invalidate - it was non-present before */ + update_mmu_cache(vma, address, pte); +unlock: + pte_unmap_unlock(pte, ptl); + return ret; +release: + put_page(page); + goto unlock; +oom_free_page: + put_page(page); +oom: + return VM_FAULT_OOM; +} +#endif diff --git a/mm/pin_mem.c b/mm/pin_mem.c new file mode 100644 index 0000000..0a143b6 --- /dev/null +++ b/mm/pin_mem.c @@ -0,0 +1,950 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2020. Huawei Technologies Co., Ltd. All rights reserved. + * Provide the pin memory method for check point and restore task. + */ +#ifdef CONFIG_PIN_MEMORY +#include <linux/init.h> +#include <linux/slab.h> +#include <linux/time.h> +#include <linux/sched/cputime.h> +#include <linux/tick.h> +#include <linux/mm.h> +#include <linux/pin_mem.h> +#include <linux/idr.h> +#include <linux/page-isolation.h> +#include <linux/sched/mm.h> +#include <linux/ctype.h> +#include <linux/highmem.h> +#include <crypto/sha.h> + +#define MAX_PIN_PID_NUM 128 +static DEFINE_SPINLOCK(page_map_entry_lock); + +struct pin_mem_dump_info *pin_mem_dump_start; +unsigned int pin_pid_num; +static unsigned int *pin_pid_num_addr; +static unsigned long __page_map_entry_start; +static unsigned long page_map_entry_end; +static struct page_map_info *user_space_reserve_start; +static struct page_map_entry *page_map_entry_start; +unsigned int max_pin_pid_num __read_mostly; +unsigned long redirect_space_size; +unsigned long redirect_space_start; +#define DEFAULT_REDIRECT_SPACE_SIZE 0x100000 + +static int __init setup_max_pin_pid_num(char *str) +{ + int ret = 0; + + if (!str) + goto out; + + ret = kstrtouint(str, 10, &max_pin_pid_num); +out: + if (ret) { + pr_warn("Unable to parse max pin pid num.\n"); + } else { + if (max_pin_pid_num > MAX_PIN_PID_NUM) { + max_pin_pid_num = 0; + pr_warn("Input max_pin_pid_num is too large.\n"); + } + } + return ret; +} +early_param("max_pin_pid_num", setup_max_pin_pid_num); + +static int __init setup_redirect_space_size(char *str) +{ + if (!str) + goto out; + + redirect_space_size = memparse(str, NULL); +out: + if (!redirect_space_size) { + pr_warn("Unable to parse redirect space size, use the default value.\n"); + redirect_space_size = DEFAULT_REDIRECT_SPACE_SIZE; + } + return 0; +} +early_param("redirect_space_size", setup_redirect_space_size); + +struct page_map_info *create_page_map_info(int pid) +{ + struct page_map_info *new; + + if (!user_space_reserve_start) + return NULL; + + if (pin_pid_num >= max_pin_pid_num) { + pr_warn("Pin pid num too large than max_pin_pid_num, fail create: %d!", pid); + return NULL; + } + new = (struct page_map_info *)(user_space_reserve_start + pin_pid_num); + new->pid = pid; + new->pme = NULL; + new->entry_num = 0; + new->pid_reserved = false; + new->disable_free_page = false; + (*pin_pid_num_addr)++; + pin_pid_num++; + return new; +} +EXPORT_SYMBOL_GPL(create_page_map_info); + +struct page_map_info *get_page_map_info(int pid) +{ + int i; + + if (!user_space_reserve_start) + return NULL; + + for (i = 0; i < pin_pid_num; i++) { + if (user_space_reserve_start[i].pid == pid) + return &(user_space_reserve_start[i]); + } + return NULL; +} +EXPORT_SYMBOL_GPL(get_page_map_info); + +static struct page *find_head_page(struct page *page) +{ + struct page *p = page; + + while (!PageBuddy(p)) { + if (PageLRU(p)) + return NULL; + p--; + } + return p; +} + +static void spilt_page_area_left(struct zone *zone, struct free_area *area, struct page *page, + unsigned long size, int order) +{ + unsigned long cur_size = 1 << order; + unsigned long total_size = 0; + + while (size && cur_size > size) { + cur_size >>= 1; + order--; + area--; + if (cur_size <= size) { + list_add(&page[total_size].lru, &area->free_list[MIGRATE_MOVABLE]); + atomic_set(&(page[total_size]._mapcount), PAGE_BUDDY_MAPCOUNT_VALUE); + set_page_private(&page[total_size], order); + set_pageblock_migratetype(&page[total_size], MIGRATE_MOVABLE); + area->nr_free++; + total_size += cur_size; + size -= cur_size; + } + } +} + +static void spilt_page_area_right(struct zone *zone, struct free_area *area, struct page *page, + unsigned long size, int order) +{ + unsigned long cur_size = 1 << order; + struct page *right_page, *head_page; + + right_page = page + size; + while (size && cur_size > size) { + cur_size >>= 1; + order--; + area--; + if (cur_size <= size) { + head_page = right_page - cur_size; + list_add(&head_page->lru, &area->free_list[MIGRATE_MOVABLE]); + atomic_set(&(head_page->_mapcount), PAGE_BUDDY_MAPCOUNT_VALUE); + set_page_private(head_page, order); + set_pageblock_migratetype(head_page, MIGRATE_MOVABLE); + area->nr_free++; + size -= cur_size; + right_page = head_page; + } + } +} + +void reserve_page_from_buddy(unsigned long nr_pages, struct page *page) +{ + unsigned int current_order; + struct page *page_end; + struct free_area *area; + struct zone *zone; + struct page *head_page; + + head_page = find_head_page(page); + if (!head_page) { + pr_warn("Find page head fail."); + return; + } + current_order = head_page->private; + page_end = head_page + (1 << current_order); + zone = page_zone(head_page); + area = &(zone->free_area[current_order]); + list_del(&head_page->lru); + atomic_set(&head_page->_mapcount, -1); + set_page_private(head_page, 0); + area->nr_free--; + if (head_page != page) + spilt_page_area_left(zone, area, head_page, + (unsigned long)(page - head_page), current_order); + page = page + nr_pages; + if (page < page_end) { + spilt_page_area_right(zone, area, page, + (unsigned long)(page_end - page), current_order); + } else if (page > page_end) { + pr_warn("Find page end smaller than page."); + } +} + +static inline void reserve_user_normal_pages(struct page *page) +{ + atomic_inc(&page->_refcount); + reserve_page_from_buddy(1, page); +} + +static void init_huge_pmd_pages(struct page *head_page) +{ + int i = 0; + struct page *page = head_page; + + __set_bit(PG_head, &page->flags); + __set_bit(PG_active, &page->flags); + atomic_set(&page->_refcount, 1); + page++; + i++; + page->compound_head = (unsigned long)head_page + 1; + page->compound_dtor = HUGETLB_PAGE_DTOR + 1; + page->compound_order = HPAGE_PMD_ORDER; + page++; + i++; + page->compound_head = (unsigned long)head_page + 1; + i++; + INIT_LIST_HEAD(&(page->deferred_list)); + for (; i < HPAGE_PMD_NR; i++) { + page = head_page + i; + page->compound_head = (unsigned long)head_page + 1; + } +} + +static inline void reserve_user_huge_pmd_pages(struct page *page) +{ + atomic_inc(&page->_refcount); + reserve_page_from_buddy((1 << HPAGE_PMD_ORDER), page); + init_huge_pmd_pages(page); +} + +int reserve_user_map_pages_fail; + +void free_user_map_pages(unsigned int pid_index, unsigned int entry_index, unsigned int page_index) +{ + unsigned int i, j, index, order; + struct page_map_info *pmi; + struct page_map_entry *pme; + struct page *page; + unsigned long phy_addr; + + for (index = 0; index < pid_index; index++) { + pmi = &(user_space_reserve_start[index]); + pme = pmi->pme; + for (i = 0; i < pmi->entry_num; i++) { + for (j = 0; j < pme->nr_pages; j++) { + order = pme->is_huge_page ? HPAGE_PMD_ORDER : 0; + phy_addr = pme->phy_addr_array[j]; + if (phy_addr) { + page = phys_to_page(phy_addr); + if (!(page->flags & PAGE_FLAGS_CHECK_RESERVED)) { + __free_pages(page, order); + pme->phy_addr_array[j] = 0; + } + } + } + pme = (struct page_map_entry *)next_pme(pme); + } + } + pmi = &(user_space_reserve_start[index]); + pme = pmi->pme; + for (i = 0; i < entry_index; i++) { + for (j = 0; j < pme->nr_pages; j++) { + order = pme->is_huge_page ? HPAGE_PMD_ORDER : 0; + phy_addr = pme->phy_addr_array[j]; + if (phy_addr) { + page = phys_to_page(phy_addr); + if (!(page->flags & PAGE_FLAGS_CHECK_RESERVED)) { + __free_pages(page, order); + pme->phy_addr_array[j] = 0; + } + } + } + pme = (struct page_map_entry *)next_pme(pme); + } + for (j = 0; j < page_index; j++) { + order = pme->is_huge_page ? HPAGE_PMD_ORDER : 0; + phy_addr = pme->phy_addr_array[j]; + if (phy_addr) { + page = phys_to_page(phy_addr); + if (!(page->flags & PAGE_FLAGS_CHECK_RESERVED)) { + __free_pages(page, order); + pme->phy_addr_array[j] = 0; + } + } + } +} + +bool check_redirect_end_valid(struct redirect_info *redirect_start, + unsigned long max_redirect_page_num) +{ + unsigned long redirect_end; + + redirect_end = ((unsigned long)(redirect_start + 1) + + max_redirect_page_num * sizeof(unsigned int)); + if (redirect_end > redirect_space_start + redirect_space_size) + return false; + return false; +} + +static void reserve_user_space_map_pages(void) +{ + struct page_map_info *pmi; + struct page_map_entry *pme; + unsigned int i, j, index; + struct page *page; + unsigned long flags; + unsigned long phy_addr; + unsigned long redirect_pages = 0; + struct redirect_info *redirect_start = (struct redirect_info *)redirect_space_start; + + if (!user_space_reserve_start || !redirect_start) + return; + spin_lock_irqsave(&page_map_entry_lock, flags); + for (index = 0; index < pin_pid_num; index++) { + pmi = &(user_space_reserve_start[index]); + pme = pmi->pme; + for (i = 0; i < pmi->entry_num; i++) { + redirect_pages = 0; + if (!check_redirect_end_valid(redirect_start, pme->nr_pages)) + redirect_start = NULL; + for (j = 0; j < pme->nr_pages; j++) { + phy_addr = pme->phy_addr_array[j]; + if (!phy_addr) + continue; + page = phys_to_page(phy_addr); + if (atomic_read(&page->_refcount)) { + if ((page->flags & PAGE_FLAGS_CHECK_RESERVED) + && !pme->redirect_start) + pme->redirect_start = + (unsigned long)redirect_start; + if (redirect_start && + (page->flags & PAGE_FLAGS_CHECK_RESERVED)) { + redirect_start->redirect_index[redirect_pages] = j; + redirect_pages++; + continue; + } else { + reserve_user_map_pages_fail = 1; + pr_warn("Page %pK refcount %d large than zero, no need reserve.\n", + page, atomic_read(&page->_refcount)); + goto free_pages; + } + } + if (!pme->is_huge_page) + reserve_user_normal_pages(page); + else + reserve_user_huge_pmd_pages(page); + } + pme = (struct page_map_entry *)next_pme(pme); + if (redirect_pages && redirect_start) { + redirect_start->redirect_pages = redirect_pages; + redirect_start = (struct redirect_info *)( + (unsigned long)(redirect_start + 1) + + redirect_start->redirect_pages * sizeof(unsigned int)); + } + } + } + spin_unlock(&page_map_entry_lock); + return; +free_pages: + free_user_map_pages(index, i, j); + spin_unlock(&page_map_entry_lock); +} + + +int calculate_pin_mem_digest(struct pin_mem_dump_info *pmdi, char *digest) +{ + int i; + struct sha256_state sctx; + + if (!digest) + digest = pmdi->sha_digest; + sha256_init(&sctx); + sha256_update(&sctx, (unsigned char *)(&(pmdi->magic)), + sizeof(struct pin_mem_dump_info) - SHA256_DIGEST_SIZE); + for (i = 0; i < pmdi->pin_pid_num; i++) { + sha256_update(&sctx, (unsigned char *)(&(pmdi->pmi_array[i])), + sizeof(struct page_map_info)); + } + sha256_final(&sctx, digest); + return 0; +} + +static int check_sha_digest(struct pin_mem_dump_info *pmdi) +{ + int ret = 0; + char digest[SHA256_DIGEST_SIZE] = {0}; + + ret = calculate_pin_mem_digest(pmdi, digest); + if (ret) { + pr_warn("calculate pin mem digest fail:%d\n", ret); + return ret; + } + if (memcmp(pmdi->sha_digest, digest, SHA256_DIGEST_SIZE)) { + pr_warn("pin mem dump info sha256 digest match error!\n"); + return -EFAULT; + } + return ret; +} + +/* + * The whole page map entry collect process must be Sequentially. + * The user_space_reserve_start points to the first page map info for + * the first dump task. And the page_map_entry_start points to + * the first page map entry of the first dump vma. + */ +static void init_page_map_info(struct pin_mem_dump_info *pmdi, unsigned long map_len) +{ + if (pin_mem_dump_start || !max_pin_pid_num) { + pr_warn("pin page map already init or max_pin_pid_num not set.\n"); + return; + } + if (map_len < sizeof(struct pin_mem_dump_info) + + max_pin_pid_num * sizeof(struct page_map_info) + redirect_space_size) { + pr_warn("pin memory reserved memblock too small.\n"); + return; + } + if ((pmdi->magic != PIN_MEM_DUMP_MAGIC) || (pmdi->pin_pid_num > max_pin_pid_num) || + check_sha_digest(pmdi)) + memset(pmdi, 0, sizeof(struct pin_mem_dump_info)); + pin_mem_dump_start = pmdi; + pin_pid_num = pmdi->pin_pid_num; + pr_info("pin_pid_num: %d\n", pin_pid_num); + pin_pid_num_addr = &(pmdi->pin_pid_num); + user_space_reserve_start = + (struct page_map_info *)pmdi->pmi_array; + page_map_entry_start = + (struct page_map_entry *)(user_space_reserve_start + max_pin_pid_num); + page_map_entry_end = (unsigned long)pmdi + map_len - redirect_space_size; + redirect_space_start = page_map_entry_end; + if (pin_pid_num > 0) + reserve_user_space_map_pages(); +} + +int finish_pin_mem_dump(void) +{ + int ret; + + pin_mem_dump_start->magic = PIN_MEM_DUMP_MAGIC; + memset(pin_mem_dump_start->sha_digest, 0, SHA256_DIGEST_SIZE); + ret = calculate_pin_mem_digest(pin_mem_dump_start, NULL); + if (ret) { + pr_warn("calculate pin mem digest fail:%d\n", ret); + return ret; + } + return ret; +} + +int collect_pmd_huge_pages(struct task_struct *task, + unsigned long start_addr, unsigned long end_addr, struct page_map_entry *pme) +{ + long res; + int index = 0; + unsigned long start = start_addr; + struct page *temp_page; + + while (start < end_addr) { + temp_page = NULL; + res = get_user_pages_remote(task->mm, start, 1, + FOLL_TOUCH | FOLL_GET, &temp_page, NULL, NULL); + if (!res) { + pr_warn("Get huge page for addr(%lx) fail.", start); + return COLLECT_PAGES_FAIL; + } + if (PageHead(temp_page)) { + start += HPAGE_PMD_SIZE; + pme->phy_addr_array[index] = page_to_phys(temp_page); + index++; + } else { + pme->nr_pages = index; + atomic_dec(&((temp_page)->_refcount)); + return COLLECT_PAGES_NEED_CONTINUE; + } + } + pme->nr_pages = index; + return COLLECT_PAGES_FINISH; +} + +int collect_normal_pages(struct task_struct *task, + unsigned long start_addr, unsigned long end_addr, struct page_map_entry *pme) +{ + int res; + unsigned long next; + unsigned long i, nr_pages; + struct page *tmp_page; + unsigned long *phy_addr_array = pme->phy_addr_array; + struct page **page_array = (struct page **)pme->phy_addr_array; + + next = (start_addr & HPAGE_PMD_MASK) + HPAGE_PMD_SIZE; + next = (next > end_addr) ? end_addr : next; + pme->nr_pages = 0; + while (start_addr < next) { + nr_pages = (PAGE_ALIGN(next) - start_addr) / PAGE_SIZE; + res = get_user_pages_remote(task->mm, start_addr, 1, + FOLL_TOUCH | FOLL_GET, &tmp_page, NULL, NULL); + if (!res) { + pr_warn("Get user page of %lx fail.\n", start_addr); + return COLLECT_PAGES_FAIL; + } + if (PageHead(tmp_page)) { + atomic_dec(&(tmp_page->_refcount)); + return COLLECT_PAGES_NEED_CONTINUE; + } + atomic_dec(&(tmp_page->_refcount)); + if (PageTail(tmp_page)) { + start_addr = next; + pme->virt_addr = start_addr; + next = (next + HPAGE_PMD_SIZE) > end_addr ? + end_addr : (next + HPAGE_PMD_SIZE); + continue; + } + res = get_user_pages_remote(task->mm, start_addr, nr_pages, + FOLL_TOUCH | FOLL_GET, page_array, NULL, NULL); + if (!res) { + pr_warn("Get user pages of %lx fail.\n", start_addr); + return COLLECT_PAGES_FAIL; + } + for (i = 0; i < nr_pages; i++) + phy_addr_array[i] = page_to_phys(page_array[i]); + pme->nr_pages += nr_pages; + page_array += nr_pages; + phy_addr_array += nr_pages; + start_addr = next; + next = (next + HPAGE_PMD_SIZE) > end_addr ? end_addr : (next + HPAGE_PMD_SIZE); + } + return COLLECT_PAGES_FINISH; +} + +/* Users make sure that the pin memory belongs to anonymous vma. */ +int pin_mem_area(struct task_struct *task, struct mm_struct *mm, + unsigned long start_addr, unsigned long end_addr) +{ + int pid, ret; + int is_huge_page = false; + unsigned int page_size; + unsigned long nr_pages, flags; + struct page_map_entry *pme; + struct page_map_info *pmi; + struct vm_area_struct *vma; + unsigned long i; + struct page *tmp_page; + + if (!page_map_entry_start + || !task || !mm + || start_addr >= end_addr) + return -EFAULT; + + pid = task->pid; + spin_lock_irqsave(&page_map_entry_lock, flags); + nr_pages = ((end_addr - start_addr) / PAGE_SIZE); + if ((unsigned long)page_map_entry_start + nr_pages * sizeof(struct page *) >= + page_map_entry_end) { + pr_warn("Page map entry use up!\n"); + ret = -EFAULT; + goto finish; + } + vma = find_extend_vma(mm, start_addr); + if (!vma) { + pr_warn("Find no match vma!\n"); + ret = -EFAULT; + goto finish; + } + if (start_addr == (start_addr & HPAGE_PMD_MASK) && + transparent_hugepage_enabled(vma)) { + page_size = HPAGE_PMD_SIZE; + is_huge_page = true; + } else { + page_size = PAGE_SIZE; + } + pme = page_map_entry_start; + pme->virt_addr = start_addr; + pme->redirect_start = 0; + pme->is_huge_page = is_huge_page; + memset(pme->phy_addr_array, 0, nr_pages * sizeof(unsigned long)); + down_write(&mm->mmap_lock); + if (!is_huge_page) { + ret = collect_normal_pages(task, start_addr, end_addr, pme); + if (ret != COLLECT_PAGES_FAIL && !pme->nr_pages) { + if (ret == COLLECT_PAGES_FINISH) { + ret = 0; + up_write(&mm->mmap_lock); + goto finish; + } + pme->is_huge_page = true; + page_size = HPAGE_PMD_SIZE; + ret = collect_pmd_huge_pages(task, pme->virt_addr, end_addr, pme); + } + } else { + ret = collect_pmd_huge_pages(task, start_addr, end_addr, pme); + if (ret != COLLECT_PAGES_FAIL && !pme->nr_pages) { + if (ret == COLLECT_PAGES_FINISH) { + ret = 0; + up_write(&mm->mmap_lock); + goto finish; + } + pme->is_huge_page = false; + page_size = PAGE_SIZE; + ret = collect_normal_pages(task, pme->virt_addr, end_addr, pme); + } + } + up_write(&mm->mmap_lock); + if (ret == COLLECT_PAGES_FAIL) { + ret = -EFAULT; + goto finish; + } + + /* check for zero pages */ + for (i = 0; i < pme->nr_pages; i++) { + tmp_page = phys_to_page(pme->phy_addr_array[i]); + if (!pme->is_huge_page) { + if (page_to_pfn(tmp_page) == my_zero_pfn(pme->virt_addr + i * PAGE_SIZE)) + pme->phy_addr_array[i] = 0; + } else if (is_huge_zero_page(tmp_page)) + pme->phy_addr_array[i] = 0; + } + + page_map_entry_start = (struct page_map_entry *)(next_pme(pme)); + pmi = get_page_map_info(pid); + if (!pmi) + pmi = create_page_map_info(pid); + if (!pmi) { + pr_warn("Create page map info fail for pid: %d!\n", pid); + ret = -EFAULT; + goto finish; + } + if (!pmi->pme) + pmi->pme = pme; + pmi->entry_num++; + spin_unlock_irqrestore(&page_map_entry_lock, flags); + if (ret == COLLECT_PAGES_NEED_CONTINUE) + ret = pin_mem_area(task, mm, pme->virt_addr + pme->nr_pages * page_size, end_addr); + return ret; +finish: + spin_unlock_irqrestore(&page_map_entry_lock, flags); + return ret; +} +EXPORT_SYMBOL_GPL(pin_mem_area); + +vm_fault_t remap_normal_pages(struct mm_struct *mm, struct vm_area_struct *vma, + struct page_map_entry *pme) +{ + int ret; + unsigned int j, i; + pgd_t *pgd; + p4d_t *p4d; + pmd_t *pmd; + pud_t *pud; + struct page *page, *new; + unsigned long address; + unsigned long phy_addr; + unsigned int redirect_pages = 0; + struct redirect_info *redirect_start; + + redirect_start = (struct redirect_info *)pme->redirect_start; + for (j = 0; j < pme->nr_pages; j++) { + address = pme->virt_addr + j * PAGE_SIZE; + phy_addr = pme->phy_addr_array[j]; + if (!phy_addr) + continue; + page = phys_to_page(phy_addr); + if (page_to_pfn(page) == my_zero_pfn(address)) { + pme->phy_addr_array[j] = 0; + continue; + } + pme->phy_addr_array[j] = 0; + if (redirect_start && (redirect_pages < redirect_start->redirect_pages) && + (j == redirect_start->redirect_index[redirect_pages])) { + new = alloc_zeroed_user_highpage_movable(vma, address); + if (!new) { + pr_warn("Redirect alloc page fail\n"); + continue; + } + copy_page(page_to_virt(new), phys_to_virt(phy_addr)); + page = new; + redirect_pages++; + } + page->mapping = NULL; + pgd = pgd_offset(mm, address); + p4d = p4d_alloc(mm, pgd, address); + if (!p4d) { + ret = VM_FAULT_OOM; + goto free; + } + pud = pud_alloc(mm, p4d, address); + if (!pud) { + ret = VM_FAULT_OOM; + goto free; + } + pmd = pmd_alloc(mm, pud, address); + if (!pmd) { + ret = VM_FAULT_OOM; + goto free; + } + ret = do_anon_page_remap(vma, address, pmd, page); + if (ret) + goto free; + } + return 0; +free: + for (i = j; i < pme->nr_pages; i++) { + phy_addr = pme->phy_addr_array[i]; + if (phy_addr) { + __free_page(phys_to_page(phy_addr)); + pme->phy_addr_array[i] = 0; + } + } + return ret; +} + +static inline gfp_t get_hugepage_gfpmask(struct vm_area_struct *vma) +{ + const bool vma_madvised = !!(vma->vm_flags & VM_HUGEPAGE); + + if (test_bit(TRANSPARENT_HUGEPAGE_DEFRAG_DIRECT_FLAG, &transparent_hugepage_flags)) + return GFP_TRANSHUGE | (vma_madvised ? 0 : __GFP_NORETRY); + if (test_bit(TRANSPARENT_HUGEPAGE_DEFRAG_KSWAPD_FLAG, &transparent_hugepage_flags)) + return GFP_TRANSHUGE_LIGHT | __GFP_KSWAPD_RECLAIM; + if (test_bit(TRANSPARENT_HUGEPAGE_DEFRAG_KSWAPD_OR_MADV_FLAG, &transparent_hugepage_flags)) + return GFP_TRANSHUGE_LIGHT | (vma_madvised ? __GFP_DIRECT_RECLAIM : + __GFP_KSWAPD_RECLAIM); + if (test_bit(TRANSPARENT_HUGEPAGE_DEFRAG_REQ_MADV_FLAG, &transparent_hugepage_flags)) + return GFP_TRANSHUGE_LIGHT | (vma_madvised ? __GFP_DIRECT_RECLAIM : + 0); + return GFP_TRANSHUGE_LIGHT; +} + +vm_fault_t remap_huge_pmd_pages(struct mm_struct *mm, struct vm_area_struct *vma, + struct page_map_entry *pme) +{ + int ret; + unsigned int j, i; + pgd_t *pgd; + p4d_t *p4d; + pmd_t *pmd; + pud_t *pud; + gfp_t gfp; + struct page *page, *new; + unsigned long address; + unsigned long phy_addr; + unsigned int redirect_pages = 0; + struct redirect_info *redirect_start; + + redirect_start = (struct redirect_info *)pme->redirect_start; + for (j = 0; j < pme->nr_pages; j++) { + address = pme->virt_addr + j * HPAGE_PMD_SIZE; + phy_addr = pme->phy_addr_array[j]; + if (!phy_addr) + continue; + page = phys_to_page(phy_addr); + if (is_huge_zero_page(page)) { + pme->phy_addr_array[j] = 0; + continue; + } + pme->phy_addr_array[j] = 0; + if (redirect_start && (redirect_pages < redirect_start->redirect_pages) && + (j == redirect_start->redirect_index[redirect_pages])) { + gfp = get_hugepage_gfpmask(vma); + new = alloc_hugepage_vma(gfp, vma, address, HPAGE_PMD_ORDER); + if (!new) { + pr_warn("Redirect alloc huge page fail\n"); + continue; + } + memcpy(page_to_virt(new), phys_to_virt(phy_addr), HPAGE_PMD_SIZE); + page = new; + redirect_pages++; + } + pgd = pgd_offset(mm, address); + p4d = p4d_alloc(mm, pgd, address); + if (!p4d) { + ret = VM_FAULT_OOM; + goto free; + } + pud = pud_alloc(mm, p4d, address); + if (!pud) { + ret = VM_FAULT_OOM; + goto free; + } + pmd = pmd_alloc(mm, pud, address); + if (!pmd) { + ret = VM_FAULT_OOM; + goto free; + } + ret = do_anon_huge_page_remap(vma, address, pmd, page); + if (ret) + goto free; + } + return 0; +free: + for (i = j; i < pme->nr_pages; i++) { + phy_addr = pme->phy_addr_array[i]; + if (phy_addr) { + page = phys_to_page(phy_addr); + if (!(page->flags & PAGE_FLAGS_CHECK_RESERVED)) { + __free_pages(page, HPAGE_PMD_ORDER); + pme->phy_addr_array[i] = 0; + } + } + } + return ret; +} + +static void free_unmap_pages(struct page_map_info *pmi, + struct page_map_entry *pme, + unsigned int index) +{ + unsigned int i, j; + unsigned long phy_addr; + unsigned int order; + struct page *page; + + pme = (struct page_map_entry *)(next_pme(pme)); + for (i = index; i < pmi->entry_num; i++) { + for (j = 0; j < pme->nr_pages; j++) { + phy_addr = pme->phy_addr_array[i]; + if (phy_addr) { + page = phys_to_page(phy_addr); + order = pme->is_huge_page ? HPAGE_PMD_ORDER : 0; + if (!(page->flags & PAGE_FLAGS_CHECK_RESERVED)) { + __free_pages(page, order); + pme->phy_addr_array[i] = 0; + } + } + } + pme = (struct page_map_entry *)(next_pme(pme)); + } +} + +vm_fault_t do_mem_remap(int pid, struct mm_struct *mm) +{ + unsigned int i = 0; + vm_fault_t ret = 0; + struct vm_area_struct *vma; + struct page_map_info *pmi; + struct page_map_entry *pme; + unsigned long flags; + + if (reserve_user_map_pages_fail) + return -EFAULT; + pmi = get_page_map_info(pid); + if (!pmi) + return -EFAULT; + + spin_lock_irqsave(&page_map_entry_lock, flags); + pmi->disable_free_page = true; + spin_unlock(&page_map_entry_lock); + down_write(&mm->mmap_lock); + pme = pmi->pme; + vma = mm->mmap; + while ((i < pmi->entry_num) && (vma != NULL)) { + if (pme->virt_addr >= vma->vm_start && pme->virt_addr < vma->vm_end) { + i++; + if (!vma_is_anonymous(vma)) { + pme = (struct page_map_entry *)(next_pme(pme)); + continue; + } + if (!pme->is_huge_page) { + ret = remap_normal_pages(mm, vma, pme); + if (ret < 0) + goto free; + } else { + ret = remap_huge_pmd_pages(mm, vma, pme); + if (ret < 0) + goto free; + } + pme = (struct page_map_entry *)(next_pme(pme)); + } else { + vma = vma->vm_next; + } + } + up_write(&mm->mmap_lock); + return 0; +free: + free_unmap_pages(pmi, pme, i); + up_write(&mm->mmap_lock); + return ret; +} +EXPORT_SYMBOL_GPL(do_mem_remap); + +#if defined(CONFIG_ARM64) +void init_reserve_page_map(unsigned long map_addr, unsigned long map_size) +{ + void *addr; + + if (!map_addr || !map_size) + return; + addr = phys_to_virt(map_addr); + init_page_map_info((struct pin_mem_dump_info *)addr, map_size); +} +#else +void init_reserve_page_map(unsigned long map_addr, unsigned long map_size) +{ +} +#endif + +static void free_all_reserved_pages(void) +{ + unsigned int i, j, index, order; + struct page_map_info *pmi; + struct page_map_entry *pme; + struct page *page; + unsigned long phy_addr; + + if (!user_space_reserve_start || reserve_user_map_pages_fail) + return; + + for (index = 0; index < pin_pid_num; index++) { + pmi = &(user_space_reserve_start[index]); + if (pmi->disable_free_page) + continue; + pme = pmi->pme; + for (i = 0; i < pmi->entry_num; i++) { + for (j = 0; j < pme->nr_pages; j++) { + order = pme->is_huge_page ? HPAGE_PMD_ORDER : 0; + phy_addr = pme->phy_addr_array[j]; + if (phy_addr) { + page = phys_to_page(phy_addr); + if (!(page->flags & PAGE_FLAGS_CHECK_RESERVED)) { + __free_pages(page, order); + pme->phy_addr_array[j] = 0; + } + } + } + pme = (struct page_map_entry *)next_pme(pme); + } + } +} + +/* Clear all pin memory record. */ +void clear_pin_memory_record(void) +{ + unsigned long flags; + + spin_lock_irqsave(&page_map_entry_lock, flags); + free_all_reserved_pages(); + if (pin_pid_num_addr) { + *pin_pid_num_addr = 0; + pin_pid_num = 0; + page_map_entry_start = (struct page_map_entry *)__page_map_entry_start; + } + spin_unlock(&page_map_entry_lock); +} +EXPORT_SYMBOL_GPL(clear_pin_memory_record); + +#endif /* CONFIG_PIN_MEMORY */ -- 2.9.5

1 1

[PATCH OLK-5.10 v1] arm64: Declare var of local_cpu_stop only on PARK
by sangyan＠huawei.com 01 Mar '21

01 Mar '21

From: Sang Yan <sangyan(a)huawei.com> hulk inclusion category: feature bugzilla: 48159 CVE: N/A Fix compile warning: unused variable 'ops' 'cpu' while CONFIG_ARM64_CPU_PARK=n. Put declaration of 'ops' and 'cpu' under CONFIG_ARM64_CPU_PARK in local_cpu_stop. Signed-off-by: Sang Yan <sangyan(a)huawei.com> --- arch/arm64/kernel/smp.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c index 644bbd7..d7b750a 100644 --- a/arch/arm64/kernel/smp.c +++ b/arch/arm64/kernel/smp.c @@ -1024,8 +1024,10 @@ void arch_irq_work_raise(void) static void local_cpu_stop(void) { +#ifdef CONFIG_ARM64_CPU_PARK int cpu; const struct cpu_operations *ops = NULL; +#endif set_cpu_online(smp_processor_id(), false); -- 2.9.5

1 0

[PATCH OLK-5.10 v2 1/2] arm64: smp: Add support for cpu park
by sangyan＠huawei.com 26 Feb '21

26 Feb '21

From: Sang Yan <sangyan(a)huawei.com> hulk inclusion category: feature bugzilla: 48159 CVE: N/A Introducing a feature of CPU PARK in order to save time of cpus down and up during kexec, which may cost 250ms of per cpu's down and 30ms of up. As a result, for 128 cores, it costs more than 30 seconds to down and up cpus during kexec. Think about 256 cores and more. CPU PARK is a state that cpu power-on and staying in spin loop, polling for exit chances, such as writing exit address. Reserving a block of memory, to fill with cpu park text section, exit address and park-magic-flag of each cpu. In implementation, reserved one page for one cpu core. Cpus going to park state instead of down in machine_shutdown(). Cpus going out of park state in smp_init instead of brought up. One of cpu park sections in pre-reserved memory blocks,: +--------------+ + exit address + +--------------+ + park magic + +--------------+ + park codes + + . + + . + + . + +--------------+ Signed-off-by: Sang Yan <sangyan(a)huawei.com> Reviewed-by: Jing Xiangfeng <jingxiangfeng(a)huawei.com> --- arch/arm64/Kconfig | 12 ++ arch/arm64/include/asm/kexec.h | 6 + arch/arm64/include/asm/smp.h | 15 +++ arch/arm64/kernel/Makefile | 1 + arch/arm64/kernel/cpu-park.S | 59 ++++++++++ arch/arm64/kernel/machine_kexec.c | 2 +- arch/arm64/kernel/process.c | 4 + arch/arm64/kernel/smp.c | 230 ++++++++++++++++++++++++++++++++++++++ arch/arm64/mm/init.c | 55 +++++++++ 9 files changed, 383 insertions(+), 1 deletion(-) create mode 100644 arch/arm64/kernel/cpu-park.S diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index b9c5654..0885668 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -345,6 +345,18 @@ config KASAN_SHADOW_OFFSET default 0xeffffff900000000 if ARM64_VA_BITS_36 && KASAN_SW_TAGS default 0xffffffffffffffff +config ARM64_CPU_PARK + bool "Support CPU PARK on kexec" + depends on SMP + depends on KEXEC_CORE + help + This enables support for CPU PARK feature in + order to save time of cpu down to up. + CPU park is a state through kexec, spin loop + instead of cpu die before jumping to new kernel, + jumping out from loop to new kernel entry in + smp_init. + source "arch/arm64/Kconfig.platforms" menu "Kernel Features" diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h index 79909ae..a133889 100644 --- a/arch/arm64/include/asm/kexec.h +++ b/arch/arm64/include/asm/kexec.h @@ -36,6 +36,11 @@ #define CRASH_ADDR_HIGH_MAX MEMBLOCK_ALLOC_ACCESSIBLE +#ifdef CONFIG_ARM64_CPU_PARK +/* CPU park state flag: "park" */ +#define PARK_MAGIC 0x7061726b +#endif + #ifndef __ASSEMBLY__ /** @@ -104,6 +109,7 @@ static inline void crash_post_resume(void) {} #ifdef CONFIG_KEXEC_CORE extern void __init reserve_crashkernel(void); #endif +void machine_kexec_mask_interrupts(void); #ifdef CONFIG_KEXEC_FILE #define ARCH_HAS_KIMAGE_ARCH diff --git a/arch/arm64/include/asm/smp.h b/arch/arm64/include/asm/smp.h index 2e7f529..8c5d2d6 100644 --- a/arch/arm64/include/asm/smp.h +++ b/arch/arm64/include/asm/smp.h @@ -145,6 +145,21 @@ bool cpus_are_stuck_in_kernel(void); extern void crash_smp_send_stop(void); extern bool smp_crash_stop_failed(void); +#ifdef CONFIG_ARM64_CPU_PARK +#define PARK_SECTION_SIZE 1024 +struct cpu_park_info { + /* Physical address of reserved park memory. */ + unsigned long start; + /* park reserve mem len should be PARK_SECTION_SIZE * NR_CPUS */ + unsigned long len; + /* Virtual address of reserved park memory. */ + unsigned long start_v; +}; +extern struct cpu_park_info park_info; +extern void enter_cpu_park(unsigned long text, unsigned long exit); +extern void do_cpu_park(unsigned long exit); +extern int kexec_smp_send_park(void); +#endif #endif /* ifndef __ASSEMBLY__ */ diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile index 2621d5c..60478d2 100644 --- a/arch/arm64/kernel/Makefile +++ b/arch/arm64/kernel/Makefile @@ -54,6 +54,7 @@ obj-$(CONFIG_RANDOMIZE_BASE) += kaslr.o obj-$(CONFIG_HIBERNATION) += hibernate.o hibernate-asm.o obj-$(CONFIG_KEXEC_CORE) += machine_kexec.o relocate_kernel.o \ cpu-reset.o +obj-$(CONFIG_ARM64_CPU_PARK) += cpu-park.o obj-$(CONFIG_KEXEC_FILE) += machine_kexec_file.o kexec_image.o obj-$(CONFIG_ARM64_RELOC_TEST) += arm64-reloc-test.o arm64-reloc-test-y := reloc_test_core.o reloc_test_syms.o diff --git a/arch/arm64/kernel/cpu-park.S b/arch/arm64/kernel/cpu-park.S new file mode 100644 index 0000000..10c685c --- /dev/null +++ b/arch/arm64/kernel/cpu-park.S @@ -0,0 +1,59 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * CPU park routines + * + * Copyright (C) 2020 Huawei Technologies., Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#include <linux/linkage.h> +#include <asm/assembler.h> +#include <asm/kexec.h> +#include <asm/sysreg.h> +#include <asm/virt.h> + +.text +.pushsection .idmap.text, "awx" + +/* cpu park helper in idmap section */ +SYM_CODE_START(enter_cpu_park) + /* Clear sctlr_el1 flags. */ + mrs x12, sctlr_el1 + mov_q x13, SCTLR_ELx_FLAGS + bic x12, x12, x13 + pre_disable_mmu_workaround + msr sctlr_el1, x12 /* disable mmu */ + isb + + mov x18, x0 + mov x0, x1 /* secondary_entry addr */ + br x18 /* call do_cpu_park of each cpu */ +SYM_CODE_END(enter_cpu_park) + +.popsection + +SYM_CODE_START(do_cpu_park) + ldr x18, =PARK_MAGIC /* magic number "park" */ + add x1, x0, #8 + str x18, [x1] /* set on-park flag */ + dc civac, x1 /* flush cache of "park" */ + dsb nsh + isb + +.Lloop: + wfe + isb + ldr x19, [x0] + cmp x19, #0 /* test secondary_entry */ + b.eq .Lloop + + ic iallu /* invalidate the local I-cache */ + dsb nsh + isb + + br x19 /* jump to secondary_entry */ +SYM_CODE_END(do_cpu_park) + diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c index a0b144c..f47ce96 100644 --- a/arch/arm64/kernel/machine_kexec.c +++ b/arch/arm64/kernel/machine_kexec.c @@ -213,7 +213,7 @@ void machine_kexec(struct kimage *kimage) BUG(); /* Should never get here. */ } -static void machine_kexec_mask_interrupts(void) +void machine_kexec_mask_interrupts(void) { unsigned int i; struct irq_desc *desc; diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c index 73e3b32..10cffee 100644 --- a/arch/arm64/kernel/process.c +++ b/arch/arm64/kernel/process.c @@ -146,6 +146,10 @@ void arch_cpu_idle_dead(void) */ void machine_shutdown(void) { +#ifdef CONFIG_ARM64_CPU_PARK + if (kexec_smp_send_park() == 0) + return; +#endif smp_shutdown_nonboot_cpus(reboot_cpu); } diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c index 18e9727..dea67d0 100644 --- a/arch/arm64/kernel/smp.c +++ b/arch/arm64/kernel/smp.c @@ -32,6 +32,7 @@ #include <linux/irq_work.h> #include <linux/kernel_stat.h> #include <linux/kexec.h> + #include <linux/kvm_host.h> #include <asm/alternative.h> @@ -93,6 +94,167 @@ static inline int op_cpu_kill(unsigned int cpu) } #endif +#ifdef CONFIG_ARM64_CPU_PARK +struct cpu_park_section { + unsigned long exit; /* exit address of park look */ + unsigned long magic; /* maigc represent park state */ + char text[0]; /* text section of park */ +}; + +static int mmap_cpu_park_mem(void) +{ + if (!park_info.start) + return -ENOMEM; + + if (park_info.start_v) + return 0; + + park_info.start_v = (unsigned long)__ioremap(park_info.start, + park_info.len, + PAGE_KERNEL_EXEC); + if (!park_info.start_v) { + pr_warn("map park memory failed."); + return -ENOMEM; + } + + return 0; +} + +static inline unsigned long cpu_park_section_v(unsigned int cpu) +{ + return park_info.start_v + PARK_SECTION_SIZE * (cpu - 1); +} + +static inline unsigned long cpu_park_section_p(unsigned int cpu) +{ + return park_info.start + PARK_SECTION_SIZE * (cpu - 1); +} + +/* + * Write the secondary_entry to exit section of park state. + * Then the secondary cpu will jump straight into the kernel + * by the secondary_entry. + */ +static int write_park_exit(unsigned int cpu) +{ + struct cpu_park_section *park_section; + unsigned long *park_exit; + unsigned long *park_text; + + if (mmap_cpu_park_mem() != 0) + return -EPERM; + + park_section = (struct cpu_park_section *)cpu_park_section_v(cpu); + park_exit = &park_section->exit; + park_text = (unsigned long *)park_section->text; + pr_debug("park_text 0x%lx : 0x%lx, do_cpu_park text 0x%lx : 0x%lx", + (unsigned long)park_text, *park_text, + (unsigned long)do_cpu_park, + *(unsigned long *)do_cpu_park); + + /* + * Test first 8 bytes to determine + * whether needs to write cpu park exit. + */ + if (*park_text == *(unsigned long *)do_cpu_park) { + writeq_relaxed(__pa_symbol(secondary_entry), park_exit); + __flush_dcache_area((__force void *)park_exit, + sizeof(unsigned long)); + flush_icache_range((unsigned long)park_exit, + (unsigned long)(park_exit + 1)); + sev(); + dsb(sy); + isb(); + + pr_debug("Write cpu %u secondary entry 0x%lx to 0x%lx.", + cpu, *park_exit, (unsigned long)park_exit); + pr_info("Boot cpu %u from PARK state.", cpu); + return 0; + } + + return -EPERM; +} + +/* Install cpu park sections for the specific cpu. */ +static int install_cpu_park(unsigned int cpu) +{ + struct cpu_park_section *park_section; + unsigned long *park_exit; + unsigned long *park_magic; + unsigned long park_text_len; + + park_section = (struct cpu_park_section *)cpu_park_section_v(cpu); + pr_debug("Install cpu park on cpu %u park exit 0x%lx park text 0x%lx", + cpu, (unsigned long)park_section, + (unsigned long)(park_section->text)); + + park_exit = &park_section->exit; + park_magic = &park_section->magic; + park_text_len = PARK_SECTION_SIZE - sizeof(struct cpu_park_section); + + *park_exit = 0UL; + *park_magic = 0UL; + memcpy((void *)park_section->text, do_cpu_park, park_text_len); + __flush_dcache_area((void *)park_section, PARK_SECTION_SIZE); + + return 0; +} + +static int uninstall_cpu_park(unsigned int cpu) +{ + unsigned long park_section; + + if (mmap_cpu_park_mem() != 0) + return -EPERM; + + park_section = cpu_park_section_v(cpu); + memset((void *)park_section, 0, PARK_SECTION_SIZE); + __flush_dcache_area((void *)park_section, PARK_SECTION_SIZE); + + return 0; +} + +static int cpu_wait_park(unsigned int cpu) +{ + long timeout; + struct cpu_park_section *park_section; + + volatile unsigned long *park_magic; + + park_section = (struct cpu_park_section *)cpu_park_section_v(cpu); + park_magic = &park_section->magic; + + timeout = USEC_PER_SEC; + while (*park_magic != PARK_MAGIC && timeout--) + udelay(1); + + if (timeout > 0) + pr_debug("cpu %u park done.", cpu); + else + pr_err("cpu %u park failed.", cpu); + + return *park_magic == PARK_MAGIC; +} + +static void cpu_park(unsigned int cpu) +{ + unsigned long park_section_p; + unsigned long park_exit_phy; + unsigned long do_park; + typeof(enter_cpu_park) *park; + + park_section_p = cpu_park_section_p(cpu); + park_exit_phy = park_section_p; + pr_debug("Go to park cpu %u exit address 0x%lx", cpu, park_exit_phy); + + do_park = park_section_p + sizeof(struct cpu_park_section); + park = (void *)__pa_symbol(enter_cpu_park); + + cpu_install_idmap(); + park(do_park, park_exit_phy); + unreachable(); +} +#endif /* * Boot a secondary CPU, and assign it the specified idle task. @@ -102,6 +264,10 @@ static int boot_secondary(unsigned int cpu, struct task_struct *idle) { const struct cpu_operations *ops = get_cpu_ops(cpu); +#ifdef CONFIG_ARM64_CPU_PARK + if (write_park_exit(cpu) == 0) + return 0; +#endif if (ops->cpu_boot) return ops->cpu_boot(cpu); @@ -131,6 +297,9 @@ int __cpu_up(unsigned int cpu, struct task_struct *idle) return ret; } +#ifdef CONFIG_ARM64_CPU_PARK + uninstall_cpu_park(cpu); +#endif /* * CPU was successfully started, wait for it to come online or * time out. @@ -844,10 +1013,32 @@ void arch_irq_work_raise(void) static void local_cpu_stop(void) { +#ifdef CONFIG_ARM64_CPU_PARK + int cpu; + const struct cpu_operations *ops = NULL; +#endif + set_cpu_online(smp_processor_id(), false); local_daif_mask(); sdei_mask_local_cpu(); + +#ifdef CONFIG_ARM64_CPU_PARK + /* + * Go to cpu park state. + * Otherwise go to cpu die. + */ + cpu = smp_processor_id(); + if (kexec_in_progress && park_info.start_v) { + machine_kexec_mask_interrupts(); + cpu_park(cpu); + + ops = get_cpu_ops(cpu); + if (ops && ops->cpu_die) + ops->cpu_die(cpu); + } +#endif + cpu_park_loop(); } @@ -1053,6 +1244,45 @@ void smp_send_stop(void) sdei_mask_local_cpu(); } +#ifdef CONFIG_ARM64_CPU_PARK +int kexec_smp_send_park(void) +{ + unsigned long cpu; + + if (WARN_ON(!kexec_in_progress)) { + pr_crit("%s called not in kexec progress.", __func__); + return -EPERM; + } + + if (mmap_cpu_park_mem() != 0) { + pr_info("no cpuparkmem, goto normal way."); + return -EPERM; + } + + local_irq_disable(); + + if (num_online_cpus() > 1) { + cpumask_t mask; + + cpumask_copy(&mask, cpu_online_mask); + cpumask_clear_cpu(smp_processor_id(), &mask); + + for_each_cpu(cpu, &mask) + install_cpu_park(cpu); + smp_cross_call(&mask, IPI_CPU_STOP); + + /* Wait for other CPUs to park */ + for_each_cpu(cpu, &mask) + cpu_wait_park(cpu); + pr_info("smp park other cpus done\n"); + } + + sdei_mask_local_cpu(); + + return 0; +} +#endif + #ifdef CONFIG_KEXEC_CORE void crash_smp_send_stop(void) { diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c index 794f992..d01259c 100644 --- a/arch/arm64/mm/init.c +++ b/arch/arm64/mm/init.c @@ -236,6 +236,57 @@ static void __init fdt_enforce_memory_region(void) memblock_add(usable_rgns[1].base, usable_rgns[1].size); } +#ifdef CONFIG_ARM64_CPU_PARK +struct cpu_park_info park_info = { + .start = 0, + .len = PARK_SECTION_SIZE * NR_CPUS, + .start_v = 0, +}; + +static int __init parse_park_mem(char *p) +{ + if (!p) + return 0; + + park_info.start = PAGE_ALIGN(memparse(p, NULL)); + if (park_info.start == 0) + pr_info("cpu park mem params[%s]", p); + + return 0; +} +early_param("cpuparkmem", parse_park_mem); + +static int __init reserve_park_mem(void) +{ + if (park_info.start == 0 || park_info.len == 0) + return 0; + + park_info.start = PAGE_ALIGN(park_info.start); + park_info.len = PAGE_ALIGN(park_info.len); + + if (!memblock_is_region_memory(park_info.start, park_info.len)) { + pr_warn("cannot reserve park mem: region is not memory!"); + goto out; + } + + if (memblock_is_region_reserved(park_info.start, park_info.len)) { + pr_warn("cannot reserve park mem: region overlaps reserved memory!"); + goto out; + } + + memblock_remove(park_info.start, park_info.len); + pr_info("cpu park mem reserved: 0x%016lx - 0x%016lx (%ld MB)", + park_info.start, park_info.start + park_info.len, + park_info.len >> 20); + + return 0; +out: + park_info.start = 0; + park_info.len = 0; + return -EINVAL; +} +#endif + void __init arm64_memblock_init(void) { const s64 linear_region_size = BIT(vabits_actual - 1); @@ -357,6 +408,10 @@ void __init arm64_memblock_init(void) reserve_crashkernel(); +#ifdef CONFIG_ARM64_CPU_PARK + reserve_park_mem(); +#endif + reserve_elfcorehdr(); high_memory = __va(memblock_end_of_DRAM() - 1) + 1; -- 2.9.5

1 1

[PATCH openEuler-1.0-LTS 00/65] MPAM bugfix @ 20210224
by Cheng Jian 24 Feb '21

24 Feb '21

MPAM bugfix @ 20210224 James Morse (10): arm64/mpam: Add mpam driver discovery phase and kbuild boiler plate cacheinfo: Provide a helper to find a cacheinfo leaf arm64/mpam: Probe supported partid/pmg ranges from devices arm64/mpam: Supplement MPAM MSC register layout definitions arm64/mpam: Probe the features resctrl supports arm64/mpam: Reset controls when CPUs come online arm64/mpam: Summarize feature support during mpam_enable() arm64/mpam: resctrl: Re-synchronise resctrl's view of online CPUs drivers: base: cacheinfo: Add helper to search cacheinfo by of_node arm64/mpam: Enabling registering and logging error interrupts Wang ShaoBo (55): arm64/mpam: Preparing for MPAM refactoring arm64/mpam: Add helper for getting mpam sysprops arm64/mpam: Allocate mpam component configuration arrays arm64/mpam: Pick MPAM resources and events for resctrl_res exported arm64/mpam: Init resctrl resources' info from resctrl_res selected arm64/mpam: resctrl: Handle cpuhp and resctrl_dom allocation arm64/mpam: Implement helpers for handling configuration and monitoring arm64/mpam: Migrate old MSCs' discovery process to new branch arm64/mpam: Add helper for getting MSCs' configuration arm64/mpam: Probe partid,pmg and feature capabilities' ranges from classes arm64/mpam: resctrl: Rebuild configuration and monitoring pipeline arm64/mpam: resctrl: Append schemata CDP definitions arm64/mpam: resctrl: Supplement cdpl2,cdpl3 for mount options arm64/mpam: resctrl: Add helpers for init and destroy schemata list arm64/mpam: resctrl: Use resctrl_group_init_alloc() to init schema list arm64/mpam: resctrl: Write and read schemata by schema_list arm64/mpam: Support cdp in mpam_sched_in() arm64/mpam: resctrl: Update resources reset process arm64/mpam: resctrl: Update closid alloc and free process with bitmap arm64/mpam: resctrl: Move ctrlmon sysfile write/read function to mpam_ctrlmon.c arm64/mpam: Support cdp on allocating monitors arm64/mpam: resctrl: Support cdp on monitoring data arm64/mpam: Clean up header files and rearrange declarations arm64/mpam: resctrl: Remove ctrlmon sysfile arm64/mpam: resctrl: Remove unnecessary CONFIG_ARM64 arm64/mpam: Implement intpartid narrowing process arm64/mpam: Using software-defined id for rdtgroup instead of 32-bit integer arm64/mpam: resctrl: collect child mon group's monitor data arm64/mpam: resctrl: Support cpus' monitoring for mon group arm64/mpam: resctrl: Support priority and hardlimit(Memory bandwidth) configuration arm64/mpam: Store intpri and dspri for mpam device reset arm64/mpam: Squash default priority from mpam device to class arm64/mpam: Restore extend ctrls' max width for checking schemata input arm64/mpam: Re-plan intpartid narrowing process arm64/mpam: Add hook-events id for ctrl features arm64/mpam: Integrate monitor data for Memory Bandwidth if cdp enabled arm64/mpam: Fix MPAM_ESR intPARTID_range error arm64/mpam: Separate internal and downstream priority event arm64/mpam: Remap reqpartid,pmg to rmid and intpartid to closid arm64/mpam: Add wait queue for monitor alloc and free arm64/mpam: Add resctrl_ctrl_feature structure to manage ctrl features arm64/mpam: resctrl: Export resource's properties to info directory arm64/mpam: Split header files into suitable location arm64/mpam: resctrl: Add rmid file in resctrl sysfs arm64/mpam: Filter schema control type with ctrl features arm64/mpam: Simplify mpamid cdp mapping process arm64/mpam: Set per-cpu's closid to none zero for cdp ACPI/MPAM: Use acpi_map_pxm_to_node() to get node id for memory node arm64/mpam: Supplement additional useful ctrl features for mount options arm64/mpam: resctrl: Add proper error handling to resctrl_mount() arm64/mpam: resctrl: Use resctrl_group_init_alloc() for default group arm64/mpam: resctrl: Allow setting register MPAMCFG_MBW_MIN to 0 arm64/mpam: resctrl: Refresh cpu mask for handling cpuhp arm64/mpam: Sort domains when cpu online arm64/mpam: Fix compile warning arch/arm64/include/asm/mpam.h | 324 +--- arch/arm64/include/asm/mpam_resource.h | 129 -- arch/arm64/include/asm/mpam_sched.h | 8 - arch/arm64/include/asm/resctrl.h | 514 +++++- arch/arm64/kernel/Makefile | 2 +- arch/arm64/kernel/mpam.c | 1499 ---------------- arch/arm64/kernel/mpam/Makefile | 3 + arch/arm64/kernel/mpam/mpam_ctrlmon.c | 961 ++++++++++ arch/arm64/kernel/mpam/mpam_device.c | 1706 ++++++++++++++++++ arch/arm64/kernel/mpam/mpam_device.h | 140 ++ arch/arm64/kernel/mpam/mpam_internal.h | 345 ++++ arch/arm64/kernel/mpam/mpam_mon.c | 334 ++++ arch/arm64/kernel/mpam/mpam_resctrl.c | 2240 ++++++++++++++++++++++++ arch/arm64/kernel/mpam/mpam_resource.h | 228 +++ arch/arm64/kernel/mpam/mpam_setup.c | 608 +++++++ arch/arm64/kernel/mpam_ctrlmon.c | 623 ------- arch/arm64/kernel/mpam_mon.c | 124 -- drivers/acpi/arm64/mpam.c | 87 +- drivers/base/cacheinfo.c | 38 + fs/resctrlfs.c | 396 +++-- include/linux/arm_mpam.h | 118 ++ include/linux/cacheinfo.h | 36 + include/linux/resctrlfs.h | 30 - 23 files changed, 7521 insertions(+), 2972 deletions(-) delete mode 100644 arch/arm64/include/asm/mpam_resource.h delete mode 100644 arch/arm64/kernel/mpam.c create mode 100644 arch/arm64/kernel/mpam/Makefile create mode 100644 arch/arm64/kernel/mpam/mpam_ctrlmon.c create mode 100644 arch/arm64/kernel/mpam/mpam_device.c create mode 100644 arch/arm64/kernel/mpam/mpam_device.h create mode 100644 arch/arm64/kernel/mpam/mpam_internal.h create mode 100644 arch/arm64/kernel/mpam/mpam_mon.c create mode 100644 arch/arm64/kernel/mpam/mpam_resctrl.c create mode 100644 arch/arm64/kernel/mpam/mpam_resource.h create mode 100644 arch/arm64/kernel/mpam/mpam_setup.c delete mode 100644 arch/arm64/kernel/mpam_ctrlmon.c delete mode 100644 arch/arm64/kernel/mpam_mon.c create mode 100644 include/linux/arm_mpam.h -- 2.25.1

1 65